External environment recognition device

Abstract

Provided is an external environment recognition device that can expand the range of situations in which parallax error of a stereo camera can be corrected (adjusted) during vehicle travel. For a landmark identified, through fusion of a stereo camera 1 (including a parallax error-based distance measurement error) and a millimeter-wave radar 2, as corresponding to the same landmark, a parallax error correction amount for the stereo camera 1 is calculated using a high-precision distance detected by the millimeter-wave radar 2, and is integrated with a parallax error correction amount obtained using the stereo camera 1 alone. As a result, it is possible to achieve the effect of expanding the range of situations to which parallax error correction (adjustment) of the stereo camera 1 can be applied during vehicle travel.

Description

External recognition device

[0001] The present invention relates to an external recognition device that can correct (adjust) the parallax shift of a stereo camera mounted on a vehicle, for example, by using the detection distance of a high-precision millimeter-wave radar.

[0002] In a conventional stereo camera, correction of parallax shift due to lateral optical axis misalignment is performed during driving (one of the automatic correction functions). As the parallax shift correction specifications in a conventional stereo camera, there are parallax shift correction using a white line (specifically, calculating the parallax error and parallax shift correction amount from the relationship between the intersection point (infinity point) of the left and right white lines and the parallax: see Patent Document 1), parallax shift correction using a brake light (specifically, calculating the parallax error and parallax shift correction amount from the relationship between the interval of the brake lights of the preceding vehicle (hereinafter also referred to as the following vehicle) and the parallax: see Patent Document 2), parallax shift correction using stationary object feature points (specifically, calculating the parallax error and parallax shift correction amount from the tracking information of the stationary object feature points: see Patent Document 3), etc. In a conventional stereo camera, the integrated parallax shift correction amount is calculated by weighted average weighted by the presence or absence of each parallax shift correction in each scene, and the parallax shift is corrected during vehicle driving.

[0003] Also, as a parallax shift correction specification in a stereo camera during vehicle driving, there is a known technique for calculating the parallax shift correction amount from the average of the parallax conversion values of the difference between the detection distances of the camera and the radar converted into parallax, using the detection distance of a millimeter-wave radar (hereinafter sometimes simply referred to as a radar) mounted on the vehicle (see particularly FIG. 5 of Patent Document 4).

[0004] Japanese Patent Application Laid-Open No. 2001-160137, Japanese Patent Application Laid-Open No. 2017-009388, Japanese Patent Application Laid-Open No. 2019-190847, Japanese Patent Application Laid-Open No. 2007-024590

[0005] However, as described in Patent Documents 1 to 3 above, parallax shift correction using white lines, brake lights, and stationary object feature points cannot be applied in scenes where white lines are not visible, brake lights are not illuminated, or stationary object feature points are absent, respectively. In other words, parallax shift correction using the intersection point of left and right white lines (point at infinity), the spacing of brake lights of the pursued vehicle, and stationary object feature points, which were implemented with conventional stereo cameras, are applicable in limited scenes, and therefore there is a risk that parallax shift correction will not be performed for extended periods.

[0006] Furthermore, parallax shift correction as described in Patent Document 4 may not be applicable in scenes where the error changes non-linearly with distance. Also, since integration with the parallax shift correction amount of a stereo camera alone, as described in Patent Documents 1 to 3, is not considered, the scenes in which it can be applied may be limited.

[0007] The object of the present invention is to provide an external environment recognition device that can expand the scene in which parallax misalignment of a stereo camera during vehicle operation can be corrected (adjusted).

[0008] To solve the above problems, the external environment recognition device of the present invention comprises a first target detection sensor that acquires first distance data to a target around a vehicle, and a second target detection sensor that acquires second distance data to the target around the vehicle using a detection principle different from that of the first target detection sensor, and has a correction function that corrects the detection distance deviation amount specific to the first target detection sensor based on the first distance data obtained by the first target detection sensor and the second distance data obtained by the second target detection sensor, wherein the external environment recognition device stores a plurality of detection distance deviation adjustment patterns or a plurality of detection distance deviation correction patterns for correcting the detection distance deviation amount of the first target detection sensor. At least one of the plurality of detection distance deviation adjustment patterns or the plurality of detection distance deviation correction patterns is a composite detection distance deviation adjustment pattern or composite detection distance deviation correction pattern that corrects the amount of detection distance deviation of the first target detection sensor by referring to combination data of at least three or more first distance data and second distance data, and the external environment recognition device corrects the amount of detection distance deviation of the first target detection sensor by comparing the composite detection distance deviation adjustment pattern or the composite detection distance deviation correction pattern with at least one of the plurality of detection distance deviation adjustment patterns or the plurality of detection distance deviation correction patterns other than the composite detection distance deviation adjustment pattern or the composite detection distance deviation correction pattern.

[0009] According to the present invention, for example, for targets determined to be identical by fusion of a stereo camera (including distance measurement errors due to parallax shift) and a millimeter-wave radar, the parallax shift correction amount of the stereo camera can be calculated using the detection range of the high-precision millimeter-wave radar, and by integrating this with the parallax shift correction amount of the stereo camera alone, the range of scenes in which the parallax shift of the stereo camera during vehicle operation can be corrected (adjusted) can be expanded.

[0010] Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments.

[0011] A functional block diagram showing an example of the configuration of an external environment recognition device according to an embodiment of the present invention. A flowchart showing an example of the parallax shift correction process flow according to an embodiment of the present invention. A diagram showing the relationship between stereo camera parallax and radar detection distance according to an embodiment of the present invention. A diagram showing an example of weighting for each parallax shift correction according to an embodiment of the present invention. A flowchart showing another example of the parallax shift correction process flow according to an embodiment of the present invention.

[0012] Hereinafter, an embodiment of the external environment recognition device of the present invention will be described with reference to the drawings.

[0013] In the following description of the embodiments, "(parallax shift) correction" may be read as "(parallax shift) adjustment." That is, the parallax shift correction unit, parallax shift correction amount, parallax shift correction process, detection distance shift correction pattern, etc., may be read as the parallax shift adjustment unit, parallax shift adjustment amount, parallax shift adjustment process, detection distance shift adjustment pattern, etc., respectively.

[0014] <External Environment Recognition Device 100> Figure 1 is a functional block diagram showing an example of the configuration of an external environment recognition device 100 according to an embodiment of the present invention.

[0015] The external environment recognition device 100 of this embodiment is mounted on a vehicle such as an automobile (the vehicle itself) and used to detect targets around the vehicle. The external environment recognition device 100 of this embodiment is equipped with a plurality of target detection sensors with different detection principles (detection methods) and integrates (fusions) the target information detected by each target detection sensor. The target information detected by the target detection sensors includes, for example, the distance to the target, the angle to the target, i.e., the direction of the target (i.e., the relative position of the target), the relative velocity of the target in that direction, and the reflection intensity of electromagnetic waves from the target (in the case of radar). The external environment recognition device 100 of this embodiment is equipped with a stereo camera 1, which is a first target detection sensor, and a millimeter-wave radar 2, which is a second target detection sensor, and the stereo camera 1 and the millimeter-wave radar 2 are communicated together via a communication line or connection line such as a CAN (Controller Area Network).

[0016] The stereo camera 1 includes multiple cameras (10a, 10b) as imaging units, an electronic control unit (ECU) 11, and a storage device (not shown).

[0017] The multiple cameras in the stereo camera 1 are arranged at a predetermined optical axis interval (baseline length) such that their optical axes are parallel to each other, and each is composed of an image sensor such as a CCD or CMOS, an optical lens, etc. The stereo camera 1 in this embodiment has a pair of cameras (imaging units) on the left and right sides, a left camera 10a and a right camera 10b, which are arranged side by side in the left-right direction (lateral direction). The stereo camera 1 is an in-vehicle stereo camera, and the left camera 10a and the right camera 10b are arranged to image the area around the vehicle (for example, the front).

[0018] The ECU 11 is a unit that, for example, reads data and computer programs stored in a memory device by a Central Processing Unit (CPU) and executes various processes. In this embodiment, the ECU 11 reads a stereo camera parallax correction processing program from the memory device, executes stereo camera parallax correction processing, and calculates the parallax correction information of the stereo camera (details will be explained later). Furthermore, based on the target information after parallax correction detected by the stereo camera 1, the ECU 11 can also provide driving assistance such as automated driving (AD) and lane change assist (LCA).

[0019] The storage device of the stereo camera 1 is composed of, for example, volatile memory such as RAM (Random Access Memory) and non-volatile memory such as flash memory. The storage device stores, for example, data and computer programs for executing calculations and control by the CPU of the ECU 11, and stores data and information calculated by the CPU. In this embodiment, the storage device stores the stereo camera parallax shift correction processing program, stereo camera parallax shift correction information, and the like.

[0020] The radar 2 includes a transmitting / receiving unit 20, an electronic control unit (ECU) 21, and a storage device (not shown).

[0021] The transmitting / receiving unit 20 transmits electromagnetic waves as transmitted waves toward a target in the vicinity of the vehicle (for example, in front) and receives reflected waves of the transmitted waves reflected by the target. The transmitting / receiving unit 20 is a millimeter-wave radar sensor that uses electromagnetic waves in a frequency band ranging from approximately 30 GHz to approximately 300 GHz. The transmitting / receiving unit 20 is composed of, for example, a transmitting (TX) antenna, a receiving (RX) antenna, radio frequency (RF) components, a clock, an A / D converter, etc. The transmitting / receiving unit 20 is, for example, a frequency-modulated continuous-wave radar. The transmitting / receiving unit 20 may also be a pulse radar.

[0022] The ECU 21 is a unit that, for example, reads data and computer programs stored in a memory device by the central processing unit (CPU) and performs various processes. In this embodiment, the ECU 21 transmits detected target information to the stereo camera 1 at predetermined intervals or timings.

[0023] The memory device of radar 2 consists of, for example, volatile memory such as RAM (Random Access Memory) and non-volatile memory such as flash memory. The memory device stores, for example, data and computer programs for executing calculations and control by the CPU of ECU 21, and stores data and information calculated by the CPU.

[0024] <ECU21> The ECU21 includes a radar detection unit 22 as a calculation unit.

[0025] The radar detection unit 22 controls the transmission and reception of electromagnetic waves by the transmitting and receiving unit 20, detects target information based on the transmission and reception results of electromagnetic waves and reflected waves from the transmitting and receiving unit 20, temporarily stores the target information in a storage device, and outputs (transmits) it to the ECU 11 at a predetermined period or timing.

[0026] <ECU11> The ECU11 includes a camera recognition unit 12, a target fusion unit 13, and a parallax shift correction unit 14 as calculation units.

[0027] The camera recognition unit 12 controls imaging by the left camera 10a and the right camera 10b, and generates a parallax image (an image with parallax as pixel value) based on multiple images (left image and right image) captured by the left camera 10a and the right camera 10b, respectively, using a parallax shift correction amount described later. Based on the generated parallax image, it detects target information, temporarily stores the target information in a storage device, and outputs (transmits) it to the target fusion unit 13 at a predetermined period or timing. The camera recognition unit 12 can also identify the type of detected target (vehicle, motorcycle, bicycle, pedestrian, etc.) based on multiple images (left image and right image) captured by the left camera 10a and the right camera 10b, respectively, using a learning function or the like.

[0028] The target fusion unit 13 acquires target information from the radar detection unit 22 (hereinafter sometimes referred to as radar target information) and target information from the camera recognition unit 12 (hereinafter sometimes referred to as camera target information) and performs fusion processing (integration processing). In this embodiment, the target fusion unit 13 compares the radar target information and the camera target information to determine whether or not the same target has been detected (in other words, whether or not the same target has been detected), and temporarily stores information of targets determined to be the same, that is, combination data of radar target information and camera target information, in detail, combination data of parallax (w) corresponding to the detection distance of the stereo camera 1 and the detection distance (Z) of the radar 2 in a storage device, and outputs (transmits) it to the parallax shift correction unit 14 at a predetermined period or timing.

[0029] The parallax shift correction unit 14 has a correction function that corrects the amount of detection distance shift specific to the stereo camera 1 based on radar target information (in particular, the detection distance obtained by radar 2) and camera target information (in particular, the detection distance obtained by stereo camera 1). The amount of detection distance shift specific to the stereo camera 1 is due to the parallax shift between the left camera 10a and the right camera 10b of the stereo camera 1.

[0030] In this embodiment, the parallax shift correction unit 14 calculates the parallax shift correction amount for the stereo camera 1 using the detection distance of the millimeter-wave radar 2, which has high accuracy (especially in terms of longitudinal distance accuracy), for targets that the target fusion unit 13 determines to be the same as the stereo camera 1 (including distance measurement error due to parallax shift) and the millimeter-wave radar 2 while the vehicle is in motion, and integrates this with the parallax shift correction amount for the stereo camera 1 alone.

[0031] Specifically, in this embodiment, multiple detection distance shift correction patterns for correcting the detection distance shift amount of the stereo camera 1 are pre-stored in the storage device. In this embodiment, as shown in Figure 4, four detection distance shift correction patterns are pre-stored in the storage device: parallax shift correction using "radar linkage", parallax shift correction using "white lines" (see Patent Document 1), parallax shift correction using "brake lights" (see Patent Document 2), and parallax shift correction using "stationary object feature points" (see Patent Document 3). In this embodiment, at least one of the multiple stored detection distance shift correction patterns (in this embodiment, parallax shift correction using "radar linkage" in Figure 4) is a composite detection distance shift correction pattern that corrects the detection distance shift amount of the stereo camera 1 by referring to combination data of detection distances (distance data) of at least three or more stereo cameras 1 and detection distances (distance data) of radar 2. On the other hand, in this embodiment, at least one of the multiple stored detection distance shift correction patterns, other than the composite detection distance shift correction pattern (in this embodiment, parallax shift correction using "white lines," "brake lights," and "stationary object feature points" other than "radar linkage" in Figure 4), is a detection distance shift correction pattern that corrects the amount of detection distance shift of the stereo camera 1 by referring only to the detection distance (distance data) of the stereo camera 1.

[0032] In this embodiment, the parallax shift correction unit 14 acquires combination data of parallax (w) corresponding to the detection distance of the stereo camera 1 and the detection distance (Z) of the radar 2 at predetermined intervals or timings while the vehicle is in motion. If the acquired combination data of parallax (w) and detection distance (Z) is greater than or equal to a predetermined number (at least three) and greater than or equal to a predetermined distance range, the parallax shift correction unit 14 performs regression analysis on targets (black circles in Figure 3) that are determined to be identical by the fusion of the stereo camera 1 and the millimeter-wave radar 2. That is, if the acquired combination data of parallax (w) and detection distance (Z) is greater than or equal to a predetermined number (at least three) and greater than or equal to a predetermined distance range, the parallax shift correction unit 14 performs regression analysis on at least three or more combination data of parallax (w) and detection distance (Z) to calculate a regression equation in which the parallax (w) corresponding to the detection distance of the stereo camera 1 is the dependent variable and the reciprocal of the detection distance (Z) of the radar 2 (1 / Z) is the independent variable (see the solid line in Figure 3). Furthermore, the parallax shift correction unit 14 sets the parallax shift correction amount (hereinafter also referred to as the radar-linked parallax shift correction amount) if the intercept of the regression equation, i.e., the constant term C corresponding to the parallax shift at the point of infinity, is greater than or equal to a predetermined value (see the intercept in Figure 3).

[0033] In this embodiment, the parallax shift correction unit 14 calculates an integrated parallax shift correction amount by weighting (averaging) the calculated radar-coordinated parallax shift correction amount (corresponding to parallax shift correction using "radar coordination" in Figure 4) and the parallax shift correction amount of the stereo camera 1 alone (corresponding to parallax shift correction using "white lines," "brake lights," and "stationary object feature points" in Figure 4) according to the scene.

[0034] In this example, the integrated parallax correction amount is calculated by weighting (averaging) the parallax correction amounts according to the scene as follows (see also Figure 4). Scene 1: Nighttime, tracking vehicle present, brake lights on Integrated parallax correction amount = Radar-linked parallax correction amount × 0.5 + Parallax correction amount using "brake lights" × 0.5 Scene 2: Daytime, tracking vehicle present, white lines present, stationary object feature points present Integrated parallax correction amount = Radar-linked parallax correction amount × 0.5 + Parallax correction amount using "white lines" × 0.4 + Parallax correction amount using "stationary object feature points" × 0.1 Scene 3: Daytime, snowy road (no white lines or stationary object feature points), tracking vehicle present Integrated parallax correction amount = Radar-linked parallax correction amount × 1.0

[0035] In this embodiment, a disparity image is generated using the calculated integrated disparity shift correction amount, and target information is detected based on the generated disparity image, thereby correcting the detection distance shift amount specific to the stereo camera 1.

[0036] In other words, the parallax shift correction unit 14 of this embodiment corrects the amount of detected distance shift specific to the stereo camera 1 by comparing (weighting) information between a composite detection distance shift correction pattern (corresponding to parallax shift correction using "radar linkage" in Figure 4) and at least one of the multiple detection distance shift correction patterns other than the composite detection distance shift correction pattern (corresponding to parallax shift correction using "white lines," "brake lights," and "stationary object feature points" in Figure 4).

[0037] <Operation of the external environment recognition device 100> The operation of the external environment recognition device 100 in this embodiment will be described below with reference to Figures 2 to 4.

[0038] Figure 2 is a flowchart showing an example of the parallax shift correction process according to an embodiment of the present invention. In this embodiment, the parallax shift of the stereo camera 1 is corrected using the detection range of a high-precision (particularly high-precision in the front-to-back distance) millimeter-wave radar 2. Steps S31 to S34 in Figure 2 are performed by the target fusion unit 13 of the ECU 11, and steps S35 to S39 in Figure 2 are performed by the parallax shift correction unit 14 of the ECU 11.

[0039] First, in step S31, the ECU 11 acquires target information (camera target information) from the camera recognition unit 12. In step S32, the ECU 11 acquires target information (radar target information) from the radar detection unit 22. Note that steps S31 and S32 may be performed in reverse order or simultaneously.

[0040] In step S33, the ECU 11 performs fusion processing (integration processing) of camera target information and radar target information.

[0041] In step S34, the ECU 11 compares the camera target information and the radar target information to determine whether or not the same target has been detected (in other words, whether or not the same target has been detected). If the same target is detected, the ECU 11 temporarily stores the information of the target determined to be the same, that is, the combined data of the radar target information and the camera target information, specifically the combined data of the parallax (w) corresponding to the detection distance of the stereo camera 1 and the detection distance (Z) of the radar 2, in the storage device. If the same target is detected, the process proceeds to step S35; if the same target is not detected, the process returns to step S31.

[0042] In step S35, the ECU 11 acquires combined data of the parallax (w) corresponding to the detection distance of the stereo camera 1 and the detection distance (Z) of the radar 2.

[0043] In step S36, the ECU 11 determines whether the acquired combination data of parallax (w) and detection distance (Z) is greater than or equal to a predetermined number (at least three) and greater than or equal to a predetermined distance range. If the acquired combination data of parallax (w) and detection distance (Z) is greater than or equal to a predetermined number (at least three) and greater than or equal to a predetermined distance range, the process proceeds to step S37. If the acquired combination data of parallax (w) and detection distance (Z) is less than or equal to a predetermined number (at least three) or less than or equal to a predetermined distance range, the process returns to step S31.

[0044] In step S37, the ECU 11 performs a regression analysis to calculate a regression equation with the parallax (w) corresponding to the detection distance of the stereo camera 1 as the objective variable and the reciprocal (1 / Z) of the detection distance (Z) of the radar 2 as the explanatory variable for at least three or more combinations of parallax (w) and detection distance (Z) data (see the solid line in FIG. 3).

[0045] In step S38, the ECU 11 determines whether or not the intercept of the regression equation, that is, the constant term C corresponding to the parallax shift at the infinite point, is equal to or greater than a predetermined value. If the constant term C is equal to or greater than the predetermined value, the process proceeds to step S39. If the constant term C is less than the predetermined value, the process returns to step S31.

[0046] In step S39, the ECU 11 sets the constant term C as the radar-linked parallax shift correction amount (see the intercept in FIG. 3).

[0047] In step S40, the ECU 11 performs weighted averaging (weighting) of the calculated radar-linked parallax shift correction amount (corresponding to the parallax shift correction using "radar-linked" in FIG. 4) and the parallax shift correction amount of the stereo camera 1 alone (corresponding to the parallax shift correction using "white line", "brake light", and "static object feature points" in FIG. 4) according to the scene, and calculates an integrated parallax shift correction amount.

[0048] In step S36 of FIG. 2, it is determined whether or not the combination data of parallax (w) and detection distance (Z) is equal to or greater than a predetermined number (at least three) and within a predetermined distance range. However, as shown in step S36A of FIG. 5, in addition to these determination conditions, it may also be determined whether or not the number of combination data at a predetermined distance or more is equal to or greater than a predetermined number. As a result, since more combination data in the far distance at a predetermined distance or more is taken into consideration, it becomes possible to calculate the constant term C corresponding to the parallax shift at the infinite point with higher accuracy.

[0049] <Summary> As described above, the external environment recognition device 100 of this embodiment includes a first target detection sensor (stereo camera 1 in this embodiment) that acquires first distance data to targets around the vehicle, and a second target detection sensor (radar 2 in this embodiment) that acquires second distance data to targets around the vehicle using a detection principle different from that of the first target detection sensor. The external environment recognition device 100 has a correction function (parallax shift correction unit 14) that corrects the detection distance shift amount specific to the first target detection sensor based on the first distance data obtained by the first target detection sensor and the second distance data obtained by the second target detection sensor. The external environment recognition device 100 includes a plurality of detection distance shift adjustment patterns or a plurality of detection distance shift correction patterns (in Figure 4, "radar linkage", "white line", "brake light", "static") for correcting the detection distance shift amount of the first target detection sensor. The system stores parallax shift correction using "stationary object feature points" and at least one of the plurality of detection distance shift adjustment patterns or the plurality of detection distance shift correction patterns (corresponding to parallax shift correction using "radar linkage" in Figure 4) is a composite detection distance shift adjustment pattern or composite detection distance shift correction pattern that corrects the amount of detection distance shift of the first target detection sensor by referring to combination data of at least three or more of the first distance data and the second distance data. The external recognition device 100 corrects the amount of detection distance shift of the first target detection sensor by matching (weighting) the composite detection distance shift adjustment pattern or composite detection distance shift correction pattern with at least one of the plurality of detection distance shift adjustment patterns or the plurality of detection distance shift correction patterns other than the composite detection distance shift adjustment pattern or composite detection distance shift correction pattern.

[0050] According to this embodiment, for targets determined to be identical by the fusion of the stereo camera 1 (including distance measurement errors due to parallax shift) and the millimeter-wave radar 2, the parallax shift correction amount of the stereo camera 1 is calculated using the detection range of the high-precision millimeter-wave radar 2, and this is integrated with the parallax shift correction amount of the stereo camera 1 alone. This has the effect of expanding the application scenes of parallax shift correction (adjustment) of the stereo camera 1 while the vehicle is in motion.

[0051] Furthermore, the external environment recognition device 100 of this embodiment includes a first target detection sensor (stereo camera 1 in this embodiment) that acquires first distance data to targets around the vehicle, and a second target detection sensor (radar 2 in this embodiment) that acquires second distance data to targets around the vehicle using a detection principle different from that of the first target detection sensor, and a correction function that corrects the detection distance deviation amount specific to the first target detection sensor based on the first distance data obtained by the first target detection sensor and the second distance data obtained by the second target detection sensor ( In an external environment recognition device 100 having a parallax shift correction unit 14), the external environment recognition device 100 performs regression analysis on at least three or more combination data of the first distance data and the second distance data, calculating a regression equation in which the parallax (w) corresponding to the first distance data is the dependent variable and the reciprocal of the second distance data (1 / Z) is the independent variable, and corrects the detected distance shift amount of the first target detection sensor using the intercept (constant term C) of the regression equation as the parallax shift correction amount (has a composite detection distance shift adjustment pattern or a composite detection distance shift correction pattern).

[0052] In this embodiment, the parallax shift correction amount at infinity is calculated from the constant term C of a regression analysis that calculates a regression equation with the reciprocal of the detection distance of radar 2 (1 / Z) as the explanatory variable and the parallax (w) corresponding to the detection distance of stereo camera 1 as the dependent variable. This provides the effect of being able to accurately calculate the parallax shift correction amount at infinity even when the regression equation is nonlinear.

[0053] Furthermore, in this embodiment, the external environment recognition device 100 takes into account more combination data from locations beyond a predetermined distance.

[0054] According to this embodiment, when calculating the parallax shift correction amount at infinity from the constant term C of a regression analysis in which the reciprocal of the detection distance of radar 2 (1 / Z) is the explanatory variable and the parallax (w) corresponding to the detection distance of stereo camera 1 is the dependent variable, the effect is obtained that the parallax shift correction amount at infinity can be calculated with greater accuracy even when the regression equation is nonlinear.

[0055] In summary, the external recognition device 100 of this embodiment focuses on targets (such as a tracked vehicle) determined to be the same by the fusion of the stereo camera 1 and the millimeter-wave radar 2, calculates the parallax shift correction amount from the detection distance of the high-precision radar 2, and corrects it by integrating it with the parallax shift correction amount of the stereo camera 1 alone.

[0056] Conventional stereo cameras perform parallax correction due to left-right optical axis misalignment while driving (one of the automatic adjustment functions). This includes parallax correction using the intersection of left and right white lines (point of infinity), the spacing of the brake lights of the pursued vehicle, and feature points of stationary objects.

[0057] The parallax shift correction methods described above are not applicable in scenes where white lines are not visible, brake lights are not illuminated, or there are no characteristic features of stationary objects. The objective of this invention is to expand the range of scenes in which parallax shift of a stereo camera during vehicle movement can be corrected.

[0058] Therefore, in this embodiment, if the target detected by stereo camera 1 (including distance measurement error due to parallax shift) and the target detected by radar 2 are determined to be the same through fusion, the parallax shift correction amount for stereo camera 1 is calculated from the front-to-back distance of the target detected by radar 2. The radar-coordinated parallax shift correction amount calculated above is combined with the parallax shift correction amount for stereo camera 1 alone and corrected.

[0059] The detection range of the high-precision millimeter-wave radar 2 is used to calculate the parallax shift correction amount for the stereo camera 1, and this is integrated with the parallax shift correction amount for the stereo camera 1 alone. This expands the range of applications for parallax shift correction (adjustment) while the vehicle is in motion.

[0060] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described.

[0061] Furthermore, each of the above configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above configurations, functions, etc., may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0062] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In reality, it is safe to assume that almost all components are interconnected.

[0063] 1...Stereo camera (first target detection sensor) 2...Millimeter-wave radar (second target detection sensor) 10a...Left camera (left imaging unit) 10b...Right camera (right imaging unit) 11...Electronic control unit (ECU) 12...Camera recognition unit 13...Target fusion unit 14...Parallax shift correction unit 20...Transmit / receive unit 21...Electronic control unit (ECU) 22...Radar detection unit 100...External environment recognition device

Claims

1. An external environment recognition device comprising: a first target detection sensor that acquires first distance data to a target in the vicinity of a vehicle; and a second target detection sensor that acquires second distance data to the target in the vicinity of the vehicle using a detection principle different from that of the first target detection sensor, wherein the external environment recognition device has a correction function that corrects the detection distance deviation amount specific to the first target detection sensor based on the first distance data obtained by the first target detection sensor and the second distance data obtained by the second target detection sensor, wherein the external environment recognition device stores a plurality of detection distance deviation adjustment patterns or a plurality of detection distance deviation correction patterns for correcting the detection distance deviation amount of the first target detection sensor, and at least one of the plurality of detection distance deviation adjustment patterns or a plurality of detection distance deviation correction patterns is a composite detection distance deviation adjustment pattern or composite detection distance deviation correction pattern that corrects the detection distance deviation amount of the first target detection sensor by referring to at least three or more combination data of the first distance data and the second distance data, The external environment recognition device corrects the amount of detected distance deviation of the first target detection sensor by comparing the composite detection distance deviation adjustment pattern or the composite detection distance deviation correction pattern with at least one of the plurality of detection distance deviation adjustment patterns or the plurality of detection distance deviation correction patterns other than the composite detection distance deviation adjustment pattern or the composite detection distance deviation correction pattern.

2. An external environment recognition device comprising: a first target detection sensor that acquires first distance data to a target around a vehicle; and a second target detection sensor that acquires second distance data to the target around the vehicle using a detection principle different from that of the first target detection sensor, wherein the external environment recognition device has a correction function that corrects the detection distance deviation amount specific to the first target detection sensor based on the first distance data obtained by the first target detection sensor and the second distance data obtained by the second target detection sensor, wherein the external environment recognition device performs regression analysis on at least three or more combination data of the first distance data and the second distance data to calculate a regression equation in which the parallax corresponding to the first distance data is the dependent variable and the reciprocal of the second distance data is the independent variable, and corrects the detection distance deviation amount of the first target detection sensor using the intercept of the regression equation as the parallax deviation correction amount.

3. An external environment recognition device according to claim 2, wherein the combination data takes into account more combination data from locations at a distance greater than a predetermined distance.

4. An external environment recognition device according to claim 1, wherein at least one of the plurality of detection distance deviation adjustment patterns or plurality of detection distance deviation correction patterns is a composite detection distance deviation adjustment pattern or composite detection distance deviation correction pattern which performs regression analysis on combination data of at least three or more first distance data and second distance data to calculate a regression equation in which the parallax corresponding to the first distance data is the dependent variable and the reciprocal of the second distance data is the independent variable, and corrects the detection distance deviation amount of the first target detection sensor using the intercept of the regression equation as the parallax deviation correction amount.

5. An external environment recognition device according to claim 4, wherein the combination data gives more consideration to combination data from locations at a distance greater than a predetermined distance.

6. An external environment recognition device according to claim 1, wherein at least one of the plurality of detection distance deviation adjustment patterns or detection distance deviation correction patterns, other than the composite detection distance deviation adjustment pattern or the composite detection distance deviation correction pattern, is a detection distance deviation adjustment pattern or detection distance deviation correction pattern that corrects the amount of detection distance deviation of the first target detection sensor by referring to the first distance data.

Images