External recognition device and external recognition system

Abstract

To provide an external environment recognition device with which it is possible to correctly detect a three-dimensional object around the host vehicle, even when images imaged by a collimating-misaligned camera are used.SOLUTION: The external environment recognition device comprises: a sensor unit for detecting the surrounding of a vehicle; a three-dimensional object detection unit for detecting a three-dimensional object candidate on the basis of the external environment information acquired from the sensor unit; a reliability calculation unit for assessing, using a reliability calculation model, the possibility of the three-dimensional object candidate detected on the basis of the external environment information newly acquired from the sensor being an erroneously detected three-dimensional object; a three-dimensional object tracking unit for tracking a three-dimensional object candidate along with the progression of the host vehicle and continuing detection; an erroneously detected three-dimensional object assessment unit for assessing whether or not the three-dimensional object being tracked is erroneously detected, on the basis of the detection result of tracking; and an erroneously detected three-dimensional object storage unit which, when it is assessed by the erroneously detected three-dimensional object assessment unit that the three-dimensional object candidate is erroneously detected, stores at least a portion of external environment information regarding the three-dimensional object candidate having been tracked until the three-dimensional object candidate is assessed as being erroneously detected.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an external environment recognition device and an external environment recognition system that detect three-dimensional objects around a vehicle based on images captured by an in-vehicle stereo camera. [Background technology]

[0002] An in-vehicle stereo camera is an imaging device that obtains 3D information about the surroundings of the vehicle by calculating the parallax between the left and right cameras. By accurately detecting three-dimensional objects around the vehicle from the 3D information, the vehicle control device can issue appropriate warnings to the driver according to the environment around the vehicle and execute appropriate avoidance control.

[0003] Here, an object detection device disclosed in Patent Document 1 is known as a conventional technique for detecting three-dimensional objects around a vehicle. For example, the abstract of the document states, with regard to an “object detection device capable of quickly determining target detection results and improving accuracy,” that “the target recognition unit (21) derives recognition information representing a state including at least one of the position and speed of a target that can be recognized from sensor information. The prediction unit (22) predicts a state including at least one of the position and speed of the target at the next second observation timing based on the recognition information derived at the first observation timing. The score derivation unit derives a score according to the degree of difference between the state of the target observed at the second observation timing and the state of the next target predicted from the first observation timing. The reliability derivation unit (23) derives reliability by statistically processing the scores related to the target derived by the score derivation unit for each of multiple observation timings from the past to the present. The determination unit (24) determines that the target related to the reliability actually exists if the reliability satisfies a predetermined criterion.”

[0004] That is, the object detection device of Patent Document 1 estimates the state of the target (three-dimensional object) at the current time (second observation timing) from its position and speed at the previous time (first observation timing), calculates the score of the target as the difference from the actually observed state, and derives the reliability of the target based on this score. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-78554 A Summary of the Invention [Problem to be solved by the invention]

[0006] Due to the characteristics of stereo cameras, flat surfaces such as road paint, shadows, and dirt on the road surface may be mistakenly detected as three-dimensional objects. Possible causes of false detection include calibration errors and deviations from parallelism due to aging. When deviations from parallelism occur, the falsely detected three-dimensional object is detected from the same position over several frames.

[0007] The object detection device in Patent Document 1 calculates a score for a three-dimensional object (target) by comparing a prediction from the previous time (first observation timing) with an observation from the current time (second observation timing). However, when a stereo camera with a parallelism deviation is used and the score is calculated using the method in Patent Document 1, there is no inconsistency in the comparison between the prediction and observation of the three-dimensional object, and it is not possible to eliminate falsely detected three-dimensional objects.

[0008] Therefore, an object of the present invention is to provide an external environment recognition device and an external environment recognition system that can correctly detect three-dimensional objects around a vehicle by removing erroneously detected three-dimensional objects as noise from among the detected three-dimensional objects, even when a stereo camera with parallel deviation is used. [Means for solving the problem]

[0009] an external environment recognition device comprising: a sensor that detects the periphery of a vehicle; a three-dimensional object detection unit that detects a three-dimensional object candidate based on external environment information acquired from the sensor; a reliability calculation unit that uses a reliability calculation model to determine the possibility that a three-dimensional object candidate detected based on the newly acquired external environment information from the sensor is a false detection three-dimensional object; a three-dimensional object tracking unit that tracks and continues to detect the three-dimensional object candidate as the vehicle progresses; an false detection three-dimensional object determination unit that determines whether the tracked three-dimensional object candidate is a false detection based on the tracking detection result; an false detection three-dimensional object memory unit that, when the false detection three-dimensional object determination unit determines that the three-dimensional object candidate is a false detection, stores at least a part of the external environment information related to the three-dimensional object candidate when it was tracked until the three-dimensional object candidate was determined to be a false detection; and a reliability calculation model learning unit that learns a reliability calculation model from the information stored in the false detection three-dimensional object memory unit. Effect of the Invention

[0010] According to the external environment recognition device and the external environment recognition system of the present invention, even when a stereo camera with parallel deviation is used, it is possible to correctly detect three-dimensional objects around the vehicle by removing erroneously detected three-dimensional objects as noise from among the detected three-dimensional objects. [Brief description of the drawings]

[0011] [Figure 1] Hardware configuration diagram of the external environment recognition device according to the first embodiment [Diagram 2] Functional block diagram of the external environment recognition device according to the first embodiment [Diagram 3] Processing flow chart of the external environment recognition device according to the first embodiment [Figure 4] An example of how to detect a three-dimensional object [Diagram 5] Examples of false positives [Figure 6] Explanation of misalignment [Figure 7] Relationship between parallax value and object height [Figure 8] Functional block diagram of the appearance recognition system according to the second embodiment [Figure 9] Usage environment of the appearance recognition system of the second embodiment [Figure 10] Functional block diagram of the data storage unit according to the second embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of an external environment recognition device and an external environment recognition system according to the present invention will be described with reference to the drawings. EXAMPLES

[0013] First, an external environment recognition device 100 according to a first embodiment of the present invention will be described with reference to FIGS.

[0014] <Hardware configuration diagram of the external recognition device> 1 is a hardware configuration diagram of an external environment recognition device 100 according to the present embodiment. The external environment recognition device 100 is an in-vehicle device that transmits the vehicle external environment recognized based on an image captured by an in-vehicle stereo camera to a vehicle control device (Electronic Control Unit, ECU) that controls the vehicle's steering system, drive system, and braking system via a Controller Area Network (CAN). The vehicle external environment recognized by the external environment recognition device 100 includes, for example, the position of white lines on the road, the position and speed of pedestrians, the position and speed of other vehicles, other three-dimensional objects, signals, signs, and lighting lamps.

[0015] 1, the external environment recognition device 100 includes a left camera 11 and a right camera 12 that capture images in front of the vehicle, a camera interface 101 that controls both cameras and acquires captured images, a calculation processing unit 102 such as a CPU, a storage unit 103 such as a semiconductor memory, and a CAN interface 104 that handles input and output with the CAN. The camera interface 101 to the CAN interface 104 are computer units interconnected via an internal bus, and the calculation processing unit 102 executes a predetermined program to execute various processes described below. However, the present invention will be described below without describing such well-known techniques.

[0016] <Functional block diagram of the external environment recognition device 100> 2 is a functional block diagram of the external environment recognition device 100 of this embodiment. As shown here, the external environment recognition device 100 includes a sensor unit 1, a three-dimensional object detection unit 2, a reliability calculation unit 3, a three-dimensional object tracking unit 4, a false-detection three-dimensional object determination unit 5, a false-detection three-dimensional object storage unit 6, and a reliability calculation model learning unit 7. In addition, among the components shown in FIG. 2, the components other than the left camera 11 and the right camera 12 are specifically functions realized by the arithmetic processing unit 102 and the storage unit 103 in FIG. 1 working together.

[0017] Below, an overview of each part will be given with reference to the flowchart in Figure 3. The details of each part will be described later.

[0018] First, in step S1, the sensor unit 1 uses a stereo camera consisting of the left camera 11 and the right camera 12 to obtain a distance image based on the parallax between a left image PL and a right image PR captured simultaneously.

[0019] Next, in step S2, the three-dimensional object detection unit 2 detects three-dimensional object candidates based on the distance image acquired from the sensor unit 1, and outputs information on the detected three-dimensional object candidates to the reliability calculation unit 3 and the three-dimensional object tracking unit 4.

[0020] Here, in order to detect a three-dimensional object candidate, the three-dimensional object detection unit 2 first estimates the road surface area by observing the continuity in the depth direction from the distance image output by the sensor unit 1. The estimated road surface area is excluded from the distance image, and the remaining areas are grouped by close distance, so that the position, height, and width of the three-dimensional object candidate can be calculated. In addition, the present invention is not limited to this, and any three-dimensional object calculation method can be used. Then, the three-dimensional object detection unit 2 outputs the calculated position, height, and width of the three-dimensional object candidate and a template image cut out from the three-dimensional object candidate to the reliability calculation unit 3 and the three-dimensional object tracking unit 4.

[0021] The three-dimensional object candidates detected by the three-dimensional object detection unit 2 in this step include false positives where a flat surface that is not actually a three-dimensional object is detected as a three-dimensional object. For example, if a parallel deviation occurs between the left and right cameras due to a calibration error or aging deterioration of the stereo camera, the distance to a crosswalk, white line, or other road paint may be inaccurate, and therefore the object may be falsely detected as a three-dimensional object protruding from the road surface. A similar phenomenon may occur with shadows, cracks, dirt, etc. on the road surface, and it is necessary to appropriately eliminate such false positives in later processing.

[0022] After completing step S2, the external environment recognition device 100 executes in parallel the process of step S3 equivalent to the conventional technology and the processes of steps S4 to S8 specific to the present invention.

[0023] In step S3, the reliability calculation unit 3 calculates the reliability of the three-dimensional object candidate detected by the three-dimensional object detection unit 2, which indicates the possibility that the object candidate is a three-dimensional object. If the object candidate is a true three-dimensional object, the reliability value is high, whereas if the object candidate is a false positive and is not actually a three-dimensional object, the reliability value is low. This reliability is calculated by inputting a template image of the three-dimensional object candidate into the reliability calculation model. The reliability calculation model uses a decision tree that determines whether the input image is a three-dimensional object or a false positive. A model that performs a convolution operation such as deep learning on the template image and finally solves the classification problem of whether it is a three-dimensional object or a false positive may be used. Two types of models may be used, such as dividing the area into two areas based on the distance from the vehicle, distant and nearby, and using a distant / nearby model for each area. Multiple types of models may be used, such as dividing the area by an arbitrary index and using a model dedicated to each area.

[0024] Depending on the reliability calculated by the reliability calculation unit 3, it is determined whether the three-dimensional object candidate is a three-dimensional object or a false positive, and information about the three-dimensional object is output to the CAN only for the three-dimensional object candidate that is determined to be a three-dimensional object. As a result, the ECU can execute control such as braking and steering to avoid the real three-dimensional object.

[0025] Meanwhile, in step S4, the three-dimensional object tracking unit 4 tracks the three-dimensional object candidate detected by the three-dimensional object detection unit 2. The three-dimensional object tracking unit 4 stores information on three-dimensional object candidates detected in the past, and performs tracking by comparing the three-dimensional object candidate detected in the current frame with information on the past three-dimensional object candidates.

[0026] In step S5, the erroneous detection three-dimensional object determination unit 5 performs an erroneous detection determination based on the change in shape of the three-dimensional object being tracked from the tracking information of the three-dimensional object output by the three-dimensional object tracking unit 4. The cause of the erroneous detection is a parallax error caused by a deviation in the parallelism of the left and right cameras, and the effect of the parallax error changes depending on the distance between the vehicle and the three-dimensional object candidate. Therefore, when a three-dimensional object candidate that is erroneously detected due to a deviation in the parallelism is observed in a time series, the shape of the three-dimensional object candidate changes. For example, when the height of a three-dimensional object candidate that is erroneously detected is observed in a time series, the effect of the parallax error is large at a distance, and points on the road surface are observed to be greatly raised, but the closer the vehicle approaches the three-dimensional object candidate, the smaller the effect of the parallax error becomes, and the lower the height becomes. On the other hand, the height of a three-dimensional object that is not erroneously detected does not change even when observed in a time series, so that a three-dimensional object that is erroneously detected can be determined by observing the height change in a time series.

[0027] In step S6, the false-detection three-dimensional object memory unit 6 stores information on the three-dimensional object candidate that the false-detection three-dimensional object determination unit 5 has determined to be a false detection. The stored three-dimensional object information is output to the reliability calculation model learning unit 7 and used to update the parameters of the reliability calculation model so that the reliability calculation model can determine a false detection. The timing of output to the reliability calculation model learning unit 7 is when a certain number of pieces of information on false-detection three-dimensional objects are accumulated in the false-detection three-dimensional object memory unit 6, and the information may also be output at any other timing.

[0028] In step S7, the reliability calculation model learning unit 7 updates the parameters of the reliability calculation model. Using the template image of the falsely detected three-dimensional object output by the falsely detected three-dimensional object storage unit 6, the model parameters are updated so that the model classifies the falsely detected three-dimensional object as a false detection. Then, by outputting the updated model to the reliability calculation unit 3, the reliability calculation model used by the reliability calculation unit 3 is updated to the latest information. As a result, even for false detections that could not be determined as false detection three-dimensional objects using the conventional reliability calculation model, the reliability can be appropriately lowered and false detections can be determined at the time of the next and subsequent reliability calculations.

[0029] In this way, by executing the process of step S3 and the processes of steps S4 to S8 in parallel, it is possible to improve the accuracy of the reliability calculation model while performing three-dimensional object detection for vehicle control. The processes in each of the above-mentioned parts will be described in more detail below with reference to FIG. 2.

[0030] <Sensor part 1> The sensor unit 1 is a stereo camera that obtains a distance image based on parallax and outputs it to the three-dimensional object detection unit 2, and has a left camera 11, a right camera 12, a matching unit 13, and a parallax calculation unit .

[0031] The left camera 11 and the right camera 12 are imaging devices arranged at a predetermined distance from each other on the upper part of the inner surface of the windshield so as to capture images of the area in front of the vehicle. The left camera 11 captures a left image PL, and the right camera 12 captures a right image PR.

[0032] When the left and right cameras capture images of the same object, the matching unit 13 identifies the image positions of the same object on the left and right images by matching.

[0033] The parallax calculation unit 14 measures the distance by identifying the difference in the positions of the same object captured on the left and right images, and generates a parallax image. Specifically, the parallax image is generated by triangulation by identifying a triangle whose base is the distance between the left and right cameras and whose apex is the position of the same object on the image.

[0034] <Three-dimensional object detection unit 2> The three-dimensional object detection unit 2 receives the disparity image generated by the sensor unit 1 as input, detects three-dimensional object candidates from the disparity image, and outputs information on the detected three-dimensional object candidates to the reliability calculation unit 3 and the three-dimensional object tracking unit 4. The three-dimensional object detection unit 2 has a three-dimensional object extraction unit 21 and a three-dimensional object information extraction unit 22.

[0035] First, the three-dimensional object extraction unit 21 extracts a three-dimensional object from the parallax image. A method for detecting a three-dimensional object candidate from a parallax image is shown in FIG. 4. First, the three-dimensional object detection unit 2 estimates the road surface area from the parallax image in order to detect a three-dimensional object candidate. When a parallax image is generated from the camera image in FIG. 4(a), the distance to each pixel is shown by a shade as shown in FIG. 4(b). In the road surface area in FIG. 4(b), the parallax value stored in each pixel changes smoothly when the vertical position of the image changes. Therefore, the road surface area is uniformly light in color. On the other hand, it can be seen that three-dimensional objects such as vehicles are at the same distance from the vehicle and have the same parallax value. When a v-disparity map with the horizontal axis d (parallax value) and the vertical axis v (image vertical position) shown in FIG. 4(c) is created from this parallax image, it can be seen that three-dimensional objects such as vehicles are in a vertical straight line, while the road surface area is in a straight line sloping downward to the right. By obtaining the parameters of this downward-sloping straight line, the parallax value of the road surface at each vertical position of the image can be calculated, and the positions where values ​​close to the road surface parallax value are stored can be separated as road surface areas, and the rest can be separated as solid object areas. After separating the road surface area from the solid object area, the position, height, and width of the solid object candidate can be calculated by grouping the solid object areas that are close to each other. Note that this is not limited to this, and any solid object calculation method can be used.

[0036] Next, the three-dimensional object information extraction unit 22 extracts a camera image in the same position as the three-dimensional object candidate calculated by the three-dimensional object extraction unit 21, based on its position, height, and width, as a template. This template of the three-dimensional object candidate and information on its position, height, and width are output to the reliability calculation unit 3 and the three-dimensional object tracking unit 4.

[0037] <3D Object Tracking Unit 4> The three-dimensional object tracking unit 4 tracks the three-dimensional object candidates detected by the three-dimensional object detection unit 2 in chronological order, stores the tracking information, and outputs it to the false detection three-dimensional object determination unit 5.It has a tracking three-dimensional object information storage unit 41, a three-dimensional object position prediction unit 42, and a three-dimensional object matching unit 43.

[0038] The tracking three-dimensional object information storage unit 41 stores observation information of three-dimensional object candidates observed in the past. The stored observation information includes the position, height, width, and template image of the three-dimensional object candidate output by the three-dimensional object detection unit 2. In addition, the same ID is given to the same three-dimensional object candidate, and information regarding the observation time is also stored at the same time.

[0039] In addition, if the three-dimensional object matching unit 43 determines that the three-dimensional object candidate observed in the current frame and the previously observed three-dimensional object candidate stored in the tracking three-dimensional object information storage unit 41 are the same three-dimensional object, the three-dimensional object information of the current frame is registered with the same ID.

[0040] The three-dimensional object position prediction unit 42 predicts the position observed in the current frame for the three-dimensional object candidate stored in the tracked three-dimensional object information storage unit 41. The prediction is calculated based on the vehicle behavior obtained from a sensor attached to the vehicle. In addition, if the three-dimensional object candidate is a moving object, a deviation will occur if the predicted position is calculated only from the vehicle behavior, so the moving speed of the three-dimensional object candidate may be calculated from the past vehicle behavior and the position of the three-dimensional object candidate, and the value may be taken into consideration in calculating the predicted position.

[0041] The three-dimensional object matching unit 43 matches the same three-dimensional object candidate based on the three-dimensional object candidate position in the current frame predicted by the three-dimensional object position prediction unit 42 and the three-dimensional object candidate position detected by the three-dimensional object detection unit 2 in the current frame, and registers them with the same ID in the tracked three-dimensional object information storage unit 41. If the three-dimensional object detection unit 2 detects a three-dimensional object candidate in the vicinity of the position predicted by the three-dimensional object position prediction unit 42, they are registered as the same three-dimensional object candidate in the tracked three-dimensional object information storage unit 41. In addition, by considering the height and width in addition to the position of the three-dimensional object candidate, registration in the tracked three-dimensional object information storage unit 41 may be performed only when three-dimensional objects of the same size appear. In addition, a template image may be used to consider the similarity of the pattern.

[0042] <False positive three-dimensional object judgment unit 5> The erroneous detection three-dimensional object determination unit 5 determines whether the tracked three-dimensional object is an erroneous detection based on the tracked three-dimensional object information stored by the three-dimensional object tracking unit 4. If it is determined to be an erroneous detection, it outputs the three-dimensional object information to the erroneous detection three-dimensional object memory unit 6, and has a three-dimensional object height analysis unit 51 and an erroneous detection three-dimensional object information transmission unit 52.

[0043] The three-dimensional object height analysis unit 51 uses height information of the three-dimensional object candidate being tracked by the three-dimensional object tracking unit 4 to perform erroneous detection determination.

[0044] Figure 5 shows an example of a false positive three-dimensional object. One of the places where false positives are likely to occur is around road paint such as crosswalks and white lines. Similar phenomena also occur in shadows, cracks, and dirt on the road surface. The cause of these is a deviation in the parallelism of the stereo camera due to calibration errors and aging. Figure 6(a) shows the left image PL and right image PR under normal conditions with no parallelism deviation, and Figure 6(b) shows the left image PL and right image PR under abnormal conditions with parallelism deviation. In this case, a parallax deviation dD [pix] occurs, especially in areas where a diagonal straight line is captured in the image. If the inclination of the straight back is θ [rad] and the parallelism deviation is dj [pix], the parallax deviation is calculated using Equation 1. In addition, the height deviation dY of the target object when a parallax deviation dD occurs is calculated using Equation 2 for the height Y of the camera. If this error is large, the system will erroneously detect that there is an obstacle in the parallax obtained from the road surface.

[0045]

number

[0046]

number

[0047] At this time, it can be seen from Equation 2 that the height deviation dY of the target object changes according to the value of the parallax D [pix]. Figure 7 shows an example of the relationship between the parallax value and the height of a three-dimensional object. That is, when the parallax value is small (when the distance between the vehicle and the candidate three-dimensional object is far), the value of the height deviation dY is also large, and paint on the road surface such as a white line is erroneously detected as a tall three-dimensional object. On the other hand, as the parallax value increases (as the distance between the vehicle and the candidate three-dimensional object decreases), the value of the height deviation dY also decreases, and the same three-dimensional object is recognized as being shorter. Therefore, when observing the height of a candidate three-dimensional object approaching the vehicle over time, a falsely detected three-dimensional object such as road paint shows a change in height that decreases. Since the height of a real three-dimensional object does not change even if the vehicle approaches, it is possible to distinguish between a three-dimensional object and a falsely detected three-dimensional object by observing the difference in height change.

[0048] The erroneous detection three-dimensional object information transmission unit 52 transmits information on the three-dimensional object candidate that has been determined to be erroneously detected by the three-dimensional object height analysis unit 51 to the erroneous detection three-dimensional object storage unit 6. In addition, when the vehicle position (GNSS information, etc.) at the moment when the three-dimensional object candidate is determined to be erroneously detected can be acquired, this information may also be output. In addition, when environmental information including the time and weather conditions at the moment when the three-dimensional object candidate is determined to be erroneously detected can be acquired, this information may also be output.

[0049] <False positive three-dimensional object memory unit 6> The falsely detected three-dimensional object memory unit 6 receives as input information on three-dimensional object candidates that the falsely detected three-dimensional object determination unit 5 has determined to be falsely detected, and outputs the stored information to the reliability calculation model learning unit at the timing for updating the reliability calculation model.The falsely detected three-dimensional object memory unit 6 has a falsely detected three-dimensional object information storage unit 61 and a re-learning timing determination unit 62.

[0050] The false-detection three-dimensional object information storage unit 61 stores information on three-dimensional object candidates that the false-detection three-dimensional object determination unit 5 has determined to be falsely detected. The information to be stored is all texture information in a time series from when the three-dimensional object candidate is first detected until it is determined to be falsely detected. In addition, time series information sampled at any interval, such as when the three-dimensional object is 2 m away, 15 m away, or 1 m away, depending on the distance from the vehicle, may be stored. This may be classified and stored according to the time until collision with the vehicle, regardless of distance alone. In addition, only texture information at the time of first detection may be stored, or only information at any distance from the vehicle may be stored.

[0051] In addition, when there are multiple reliability calculation models, the information required for each model may be classified and stored. For example, when models for long distance and short distance are used separately, a threshold value from the vehicle may be set, and long distance information farther than the threshold and short distance information closer to the threshold may be stored separately. In addition, when information such as the vehicle's position, time, and weather conditions is included in the information output from the erroneous detection three-dimensional object information transmitting unit 52, the information may be classified and stored based on the information.

[0052] The re-learning timing determination unit 62 outputs the stored information to the reliability calculation model learning unit at the timing to update the reliability calculation model. The timing to update the model is when an arbitrary number of pieces of data are accumulated in the erroneously detected three-dimensional object information storage unit 61. Also, instead of determining the timing based on an arbitrary number of pieces of data, the timing may be determined by setting a threshold value based on the elapsed time since the previous model update. Also, the timing may be determined based on the behavior of the vehicle when it is stopped, parked, etc.

[0053] In addition, if the information output from the erroneously detected three-dimensional object information transmission unit 52 includes information such as the vehicle's position, time, and weather conditions, the timing may be determined based on that information. For example, if a lot of rainy weather data is output and then sunny weather data is output, it can be predicted that the environment around the vehicle has changed from rainy to sunny. Therefore, since it can be determined that there is little possibility of observing rainy weather data in the future, re-learning of the model used in rainy environments may be started using the rainy weather data accumulated up to now.

[0054] <Reliability calculation model learning unit 7> The reliability calculation model learning unit 7 updates parameters of the reliability calculation model, and includes a model storage unit 71, a model learning unit 72, and a model verification unit 73. Then, the model used in the reliability calculation unit 3 is made to learn new information on erroneously detected three-dimensional objects, thereby improving the accuracy of reliability calculation.

[0055] The model storage unit 71 stores the same model as that used by the reliability calculation unit 3. When multiple models are used by the reliability calculation unit 3, information on all of the models is stored. Also, depending on the frequency of false detection, models with and without parameter updates may be set, and only information on models with parameter updates may be stored.

[0056] The model learning unit 72 updates parameters of the models stored in the model storage unit 71 at the timing when the erroneously detected three-dimensional object storage unit 6 transmits data for additional learning. When there are multiple models, one of them is selected and additional learning is performed. In addition, multiple models may be additionally learned simultaneously in parallel.

[0057] The model verification unit 73 receives model information for which additional learning by the model learning unit 72 has been completed, and verifies whether or not erroneous learning has been performed as a result of the additional learning. For this purpose, the model verification unit 73 receives a template image of a candidate solid object for verification from the reliability calculation unit 3, calculates the reliability of the candidate solid object, and checks whether the calculated reliability is consistent with the behavior of the vehicle. For example, if a high reliability is calculated for a solid object in front of the vehicle (confidently estimating that the solid object in front of the vehicle is a solid object), but the vehicle does not slow down / stop (the driver or the model currently operating before the parameter update judges that the object is not a solid object), it is determined that there is a high possibility that the reliability has become unable to be calculated accurately due to additional learning. Verification is performed for a certain period or a certain number of times, and only when it is determined that erroneous additional learning has not been performed, the verified model is output to the reliability calculation unit 3. If it is determined that erroneous additional learning has been performed, measures such as redoing the learning, or deleting the information of the erroneously detected solid object received this time from the erroneously detected solid object memory unit 6 and not using it for future additional learning are taken.

[0058] In addition, when multiple models are used, the consistency between the multiple models may be verified. For example, when two types of models, a distant model and a near model, are used at a position where a three-dimensional object candidate is present, if a three-dimensional object candidate that is determined to be a false positive by the distant model is determined not to be a false positive by the near model, there is a possibility that the models have not been trained correctly. For this reason, the consistency of the estimation results between the multiple models may be verified.

[0059] As described above, according to the external environment recognition device of this embodiment, even when a stereo camera with parallel deviation is used, the three-dimensional objects around the vehicle can be correctly detected by removing erroneously detected three-dimensional objects as noise from among the detected three-dimensional objects. EXAMPLES

[0060] Next, an external environment recognition system according to a second embodiment of the present invention in which a plurality of in-vehicle external environment recognition devices and an external server are linked by wireless communication will be described with reference to Fig. 8 to Fig. 10. In the following, a duplicated description of the common points with the first embodiment will be omitted.

[0061] <Function block diagram of the external recognition system> 8 is a functional block diagram of the external environment recognition system of this embodiment in which the external environment recognition device 100A and the external server 200 are linked by wireless communication. Note that, although a configuration in which one external environment recognition device 100A is wirelessly connected to the external server 200 is illustrated here, multiple external environment recognition devices 100A can be wirelessly connected to the external server 200.

[0062] As shown in Fig. 8, the external environment recognition device 100A of this embodiment does not have the erroneous detection three-dimensional object memory unit 6 and the reliability calculation model learning unit 7 that the external environment recognition device 100 of the first embodiment has, and the functions corresponding to these are moved to the data accumulation unit 8 of the external server 200. Therefore, information on a three-dimensional object candidate that the three-dimensional object height analysis unit 51 judges to be a false detection is transmitted to the erroneous detection three-dimensional object memory unit 6 in the external environment recognition device 100 in the first embodiment, but is transmitted to the data accumulation unit 8 of the external server 200 in this embodiment.

[0063] The data storage unit 8, which has received information on the erroneously detected three-dimensional object candidate, re-learns the reliability calculation model based on the information, verifies the operation of the re-learned model, and then distributes the latest model (the re-learned and operation-verified model) to the external environment recognition device 100A of each vehicle. In addition, when a model update is requested from the external environment recognition device 100A of each vehicle, the data storage unit 8 outputs the model to the reliability calculation unit 3.

[0064] Here, the advantages of the external environment recognition system of this embodiment in cooperation with the external server 200 will be described with reference to Fig. 9. As shown here, in the external environment recognition system of this embodiment, information from the external environment recognition device 100A of each vehicle traveling in various environments such as urban areas, highways, rainy weather, forward rotation, backlighting, and nighttime is collected in the data storage unit 8 of the external server 200, so that the external server 200 can learn a highly accurate reliability calculation model that can handle more environments by using data rich in variation.

[0065] In addition, since the tendency of false positives, which are likely to occur depending on the environment around the vehicle, such as the weather, time, and the type of road on which the vehicle is traveling (urban areas, highways, unmaintained roads, etc.), changes, it is also possible to have multiple reliability calculation models specialized for each environment. For example, a rainy weather model for use in rainy environments and a nighttime model specialized for nighttime are possible. By classifying a wide variety of data collected from multiple vehicles by environment (day / night, sunny / rainy, urban areas / highways, etc.) and learning a dedicated model for each classified data, a reliability calculation model specialized for the target environment can be learned. Each vehicle can use a reliability calculation model with higher accuracy by downloading and using the most appropriate reliability calculation model based on the vehicle's position, time, weather, etc., and can reduce false alarms / miscontrols by appropriately eliminating false positives. In addition, a reliability calculation model specialized for a place where the vehicle has never traveled can be used, so that appropriate reliability calculation can be performed even in an environment where the vehicle has never traveled before.

[0066] <Data Storage Unit 8> The data storage unit 8 provided in the external server 200 re-learns the reliability calculation model based on the received data when the data storage unit 8 receives information on a three-dimensional object candidate that is determined to be a false positive by the false positive three-dimensional object determination unit 5 of the external environment recognition device 100A of any vehicle, and distributes the learned reliability calculation model to the external environment recognition device 100A of each vehicle. As shown in FIG. 10, the data storage unit 8 has a false positive three-dimensional object information receiving unit 81, a false positive three-dimensional object information storage unit 82, a re-learning timing determination unit 83, a model storage unit 84, a model learning unit 85, a model verification unit 86, and a model output unit 87. The false positive three-dimensional object information storage unit 82 and the re-learning timing determination unit 83 correspond to the function of the false positive three-dimensional object memory unit 6 in the first embodiment, and the model storage unit 84, the model learning unit 85, and the model verification unit 86 correspond to the function of the reliability calculation model learning unit 7 in the first embodiment. Each unit will be described in order below.

[0067] The false-detection three-dimensional object information receiving unit 81 receives the information transmitted by the false-detection three-dimensional object determining unit 5, and stores the received information in the false-detection three-dimensional object information storing unit 82. The received information is all texture information in a time series from when the three-dimensional object is first detected until it is determined to be falsely detected. It may also include time series information sampled at any interval, such as when the three-dimensional object is 2 m away, 15 m away, or 1 m away, depending on the distance from the vehicle. This may be classified and received by the time until the collision with the vehicle, regardless of the distance. It may also be possible to receive only the texture information when it is first detected, or only information of any distance from the vehicle.

[0068] In addition, when the erroneous detection three-dimensional object determination unit 5 transmits the vehicle position (GNSS information, etc.) at the moment when it determines that the three-dimensional object candidate is erroneously detected, that information may also be received. In addition, when environmental information including the time and weather conditions at the moment when it is determined that the three-dimensional object candidate is erroneously detected can be obtained, that information may also be received.

[0069] The erroneous detection three-dimensional object information storage unit 82 stores the data received by the erroneous detection three-dimensional object information receiving unit 81 .

[0070] The re-learning timing determination unit 83 determines the timing to re-learn the reliability calculation model. The timing to be determined is set to a certain time after the previous re-learning is performed. Also, while looking at the data stored in the erroneously detected three-dimensional object information storage unit 82, the time when a certain number / capacity of re-learned data is accumulated may be determined as the time to re-learn. Also, when there are multiple detection models, the re-learning timing may be determined for each model. For example, when the weather changes from rainy to sunny, the reliability calculation model for the rainy environment may be re-learned using erroneous detection data in the rainy environment. In addition, the re-learning timing may be determined based on any environmental information that can be acquired.

[0071] The model storage unit 84 stores the reliability calculation model to be distributed to each vehicle, and stores the latest model after re-learning. If there are multiple models, the multiple models are stored. Models distributed in the past may be managed and stored by version.

[0072] The model learning unit 85 re-learns the target model when the re-learning timing determination unit 83 determines that it is time to re-learn. The target model stored in the model storage unit 84 is copied, and re-learning is performed based on the data stored in the erroneous detection three-dimensional part information storage unit 82. The trained model is output to the model verification unit 86 to verify the effect of the training.

[0073] The model verification unit 86 receives the trained model retrained by the model training unit 85 as an input and verifies the effect of retraining. Using the driving data for verification, it verifies whether the retrained trained model outputs any abnormal values. If the verification result indicates that there is no problem, the model is stored in the model storage unit 84 as the latest version.

[0074] The model output unit 87 distributes the reliability calculation model stored in the model storage unit 84 to the external environment recognition device 100A of each vehicle. The output to each vehicle is executed when the latest version of the trained model is stored in the model storage unit. Alternatively, the output may be performed when a model update request is received from each vehicle. If an abnormality is found in the latest model, a version distributed in the past may be output. If environmental information about the vehicle's surroundings can be received from each vehicle, the reliability calculation model most suitable for that environment may be output. For example, when the vehicle is traveling in rainy weather, a model for rainy weather trained with rainy weather information is output.

[0075] According to the external environment recognition system of the present embodiment described above, a model can be retrained using data acquired by a plurality of external environment recognition devices 100A. Therefore, a model with higher accuracy can be used compared to the model retrained in the first embodiment. [Explanation of symbols]

[0076] 100, 100A external recognition device, 101 camera interface, 102 arithmetic processing unit, 103 storage section, 104 CAN interfaces, 1 Sensor unit, 11 Left camera, 12 Right camera, 13 Matching section, 14 Parallax calculation unit, 2 Three-dimensional object detection unit, 21 3D object extraction part, 22 Three-dimensional object information extraction section, 3. Reliability calculation unit, 4. Three-dimensional object tracking unit, 41 Tracking three-dimensional object information storage unit, 42 Three-dimensional object position prediction unit, 43 Three-dimensional object matching section, 5. False positive three-dimensional object determination unit; 51 Three-dimensional object height analysis unit, 52 False detection three-dimensional object information transmission unit, 6 False positive three-dimensional object memory section, 61 False positive three-dimensional object information storage unit, 62 Re-learning timing determination unit, 7 Reliability calculation model learning unit, 71 Model storage section, 72 Model Learning Section, 73 Model Verification Department, 200 Outside the vehicle server, 8 Data storage unit, 81 False detection three-dimensional object information receiving unit, 82 False positive three-dimensional object information storage unit, 83 Re-learning timing determination unit, 84 Model storage section, 85 Model Learning Department, 86 Model Verification Department, 87 Model Output Section

Claims

1. A sensor for detecting the surroundings of a vehicle; a three-dimensional object detection unit that detects three-dimensional object candidates based on external environment information acquired from a sensor; a reliability calculation unit that uses a reliability calculation model to determine the possibility that a three-dimensional object candidate detected based on the external environment information newly acquired from the sensor is a false-detection three-dimensional object; a three-dimensional object tracking unit that tracks and continues to detect the three-dimensional object candidate as the host vehicle advances; an erroneous detection three-dimensional object determination unit that determines whether or not the tracked three-dimensional object candidate is an erroneous detection based on the tracking detection result; a false positive three-dimensional object storage unit that stores at least a part of the external environment information regarding the three-dimensional object candidate when the three-dimensional object candidate is tracked until the three-dimensional object candidate is determined to be a false positive by the false positive three-dimensional object determination unit; a reliability calculation model learning unit that learns a reliability calculation model from the information stored in the erroneous detection three-dimensional object storage unit; Equipped with The false-detection three-dimensional object determination unit determines whether the three-dimensional object candidate is a false detection or not based on a change in shape of the three-dimensional object candidate while being tracked by the three-dimensional object tracking unit.

2. A sensor for detecting the surroundings of a vehicle; a three-dimensional object detection unit that detects three-dimensional object candidates based on external environment information acquired from a sensor; a reliability calculation unit that uses a reliability calculation model to determine the possibility that a three-dimensional object candidate detected based on the external environment information newly acquired from the sensor is a false-detection three-dimensional object; a three-dimensional object tracking unit that tracks and continues to detect the three-dimensional object candidate as the host vehicle advances; an erroneous detection three-dimensional object determination unit that determines whether or not the tracked three-dimensional object candidate is an erroneous detection based on the tracking detection result; a false positive three-dimensional object storage unit that stores at least a part of the external environment information regarding the three-dimensional object candidate when the three-dimensional object candidate is tracked until the three-dimensional object candidate is determined to be a false positive by the false positive three-dimensional object determination unit; a reliability calculation model learning unit that learns a reliability calculation model from the information stored in the erroneous detection three-dimensional object storage unit; Equipped with The false-detection three-dimensional object determination unit determines whether the three-dimensional object candidate is a false detection or not from a change in height of the three-dimensional object candidate while being tracked by the three-dimensional object tracking unit.

3. The external environment recognition device according to claim 1, The external environment information regarding the three-dimensional object candidate stored in the false-detection three-dimensional object memory unit is time-series external environment information when the three-dimensional object candidate was tracked until it was determined to be a false detection.

4. The external environment recognition device according to claim 1, When the erroneous detection three-dimensional object determination unit determines that the three-dimensional object candidate is not an erroneous detection, the external environment information regarding the three-dimensional object candidate is not stored in the erroneous detection three-dimensional object storage unit.

5. An external environment recognition system in which an in-vehicle external environment recognition device and an external server are wirelessly connected, The external environment recognition device includes: A sensor unit that detects the surroundings of the vehicle based on an image captured by a camera; a three-dimensional object detection unit that detects a three-dimensional object candidate based on external environment information acquired from the sensor unit; a reliability calculation unit that uses a reliability calculation model to determine the possibility that a three-dimensional object candidate detected based on the external environment information newly acquired from the sensor unit is a false-detection three-dimensional object; a three-dimensional object tracking unit that tracks and continues to detect the three-dimensional object candidate as the host vehicle advances; and an erroneous detection three-dimensional object determination unit that determines whether or not the three-dimensional object candidate being tracked is an erroneous detection based on the tracking result, The off-vehicle server includes: a data storage unit that stores, when the false-detection three-dimensional object determination unit determines that the three-dimensional object candidate is a false detection, at least a part of the external environment information regarding the three-dimensional object candidate when the three-dimensional object candidate was tracked until the false detection was determined, The false-detection three-dimensional object determination unit of the external environment recognition device determines whether the three-dimensional object candidate is a false detection or not from a change in the shape of the three-dimensional object candidate while being tracked by the three-dimensional object tracking unit, The external environment recognition system, wherein the false detection three-dimensional object determination unit of the external environment recognition device determines whether the three-dimensional object candidate is a false detection or not based on a change in height of the three-dimensional object candidate while being tracked by the three-dimensional object tracking unit.

6. The external environment recognition system according to claim 5, The reliability calculation model of the external environment recognition device is characterized in that an appropriate reliability calculation model is downloaded from a data storage unit of the external server in accordance with the environment around the vehicle.

7. The external environment recognition system according to claim 5, The external environment information regarding the three-dimensional object candidate stored in the data storage unit of the external server is time-series external environment information when the three-dimensional object candidate was tracked until it was determined to be a false positive.

8. The external environment recognition system according to claim 5, When the false positive three-dimensional object determination unit of the external environment recognition device determines that the three-dimensional object candidate is not a false positive, the external environment information regarding the three-dimensional object candidate is not stored in the data storage unit of the external server.

9. An external environment recognition system in which an in-vehicle external environment recognition device is wirelessly connected to an external server, The external environment recognition device includes: A sensor unit that detects the surroundings of the vehicle based on an image captured by a camera; a three-dimensional object detection unit that detects a three-dimensional object candidate based on external environment information acquired from the sensor unit; a reliability calculation unit that uses a reliability calculation model to determine the possibility that a three-dimensional object candidate detected based on the external environment information newly acquired from the sensor unit is a false-detection three-dimensional object; a three-dimensional object tracking unit that tracks and continues to detect the three-dimensional object candidate as the host vehicle advances; and an erroneous detection three-dimensional object determination unit that determines whether or not the three-dimensional object candidate being tracked is an erroneous detection based on the tracking result, The off-vehicle server includes: a data storage unit that stores, when the false-detection three-dimensional object determination unit determines that the three-dimensional object candidate is a false detection, at least a part of the external environment information regarding the three-dimensional object candidate when the three-dimensional object candidate was tracked until the false detection was determined, The false-detection three-dimensional object determination unit of the external environment recognition device determines whether the three-dimensional object candidate is a false detection or not from a change in the shape of the three-dimensional object candidate while being tracked by the three-dimensional object tracking unit, The data storage unit of the external server receives external environment information from at least one vehicle, and re-learns a reliability calculation model based on the received data; The data storage unit of the external server classifies data according to the environment around the vehicle when the external information is acquired, and trains a reliability calculation model for each classified data, thereby providing an external environment recognition system having multiple models specialized for specific environments.

10. The external environment recognition system according to claim 9, The data storage unit of the external server receives environmental information about the surroundings of the vehicle from each vehicle, and transmits a reliability calculation model that is most suitable for the received environment to the external recognition device of each vehicle.

Images