VEHICLE CONTROL SYSTEM

Abstract

Vehicle control system (100), which includes: a first sensor (111) and a second sensor (112) that detect an object in the vicinity of a vehicle; an information summarization device (120) that groups and outputs the first detection information and second detection information, the detection results of the first sensor (111) and the second sensor (112) detecting the same object; and a vehicle control device (130) that controls the vehicle based on the aggregated detection information, wherein the information summarisation device (120) includes an arithmetic device (121, 212) and a storage device (122), and the arithmetic device (121, 212) stores the first time series information, which is the time series information of the first detection information, and the second time series information, which is the time series information of the second detection information, in the storage device (122), The first time series information and the second time series information are grouped when the first sensor (111) and the second sensor (112) detect the same object in order to calculate a correction parameter of the first detection information. Correction information obtained by correcting the first Detection information is obtained using the correction parameter, calculated, and outputs the summarized detection information by instantaneous value grouping using the correction information and the second detection information, wherein the arithmetic device (121, 212) repeats the output of the aggregated detection information in a predetermined cycle and stores aggregated time series information, which is the time series information of the aggregated detection information, in the storage device (122), and wherein the arithmetic device (121, 212) calculates a calibration determination index based on the instability of the instantaneous value grouping and calculates the correction parameter when the calibration determination index becomes equal to or greater than a threshold value.

Description

Technical field

[0001] The present disclosure relates to a vehicle control system. State of the art

[0002] An invention relating to a calibration device used for a mining machine is known in the prior art (see PTL 1 below). PTL 1 discloses a calibration device having the following configuration.The calibration device includes: “a position deviation determination approval unit 114, which determines whether an obstacle is suitable for detecting a position deviation between a first measuring device 11 and a second measuring device 12, based on a geometric shape measured by the first measuring device 11; and a cross-sensor calibration unit 117, which determines a position deviation magnitude between the first measuring device 11 and the second measuring device 12 based on a detection position by the first measuring device 11 with respect to an obstacle that is determined to be suitable for detecting a position deviation by the position deviation determination approval unit 114, and a detection position by the second measuring device 12 with respect to the obstacle” (see summary section of this document).

[0003] Additionally, an invention is known relating to an action detection device and an action detection method (see PTL 2 below). An action detection device 30 of PTL 2 comprises: a storage unit 32 that stores training data for a person p, wherein the training data is a set of pairs (Xp, Yp) of sensor data Xp detected by a first sensor 10 and sensor data Yp detected by a second sensor 20; an estimation unit 332 that calculates an estimate Ye (Xp') of the sensor data Yp' for any sensor data Xp' based on the training data;a sensor data compilation unit 333 that calculates a similarity Cij between the estimated value Ye(Xp') and the collected sensor data Yj and calculates a compilation score Pij that indicates a degree of probability that person i and person j are the same person, based on temporal transitions of the calculated similarity Cij; and a person identification unit 334 that determines, based on a value of the compilation score Pij, whether person i and person j are the same person or not” (see summary section of this document).

[0004] DE 10 2018 220 024 B3, which represents the prior art, discloses a method for fusing sensor data from multiple sensors at different times. First, a first object hypothesis is formed for each sensor based on current sensor data. Then, a second object hypothesis is created for each sensor, incorporating current and historical sensor data predicted to the current time. The two object hypotheses are then fused. A correction signal is generated from the fusion result, which is used to correct further object hypotheses and / or sensor data.

[0005] WO 2019 / 039279 A1 discloses a signal processing device, a signal processing method, a program, a movable body, and a signal processing system in which a sensor unit, during calibration, records the times at which stereo camera images and radar images of a target with a radar reflector and marker were acquired while the distance to the target is varied. Time deviation values ​​are calculated from the recorded times, representing the differences between the acquisition times of the images at the same distance. During object detection, a radar image acquired first is temporarily stored and combined with a stereo camera image acquired with a corresponding time delay to generate an image of the object's distance. List of prior art patent literature PTL 1: JP 2016-223963 A PTL 2: JP 2018-13855 A Summary of the invention; Technical task

[0006] The conventional obstacle detection device described in PTL 1 determines that two obstacles are the same obstacle if the obstacle position detected by the first measuring device and the obstacle position detected by the second measuring device overlap at any time from t, t+1, t+2,... and t+n (see paragraph 0034 or the like of PTL 1). However, as in the conventional obstacle detection device, in the instantaneous value grouping used to determine whether the obstacles detected by the multiple measuring devices are the same at any given time, there is a possibility of error in the determination result, and it is difficult to perform the grouping with high accuracy.

[0007] Meanwhile, in the conventional action detection device described in PTL 2, the estimating unit calculates a composite score that indicates the degree of probability that people detected by the multiple sensors are the same person, based on a time-based transition of similarity between the multiple units of sensor data. As described above, time-series grouping based on the time-based transition of the multiple sensor data allows for grouping with higher accuracy than instantaneous value grouping. However, because time-series grouping requires time-series data, it takes longer than instantaneous value grouping, which does not.

[0008] The present disclosure provides a vehicle control system capable of grouping multiple detection results of the same object, detected by multiple sensors, with high accuracy in a short time. Solution to the task

[0009] The above problem is solved by the invention according to the subject matter of claim 1. Further preferred embodiments are described in the dependent claims.

[0010] The invention makes it possible to provide a vehicle control system that is capable of grouping multiple detection results of the same object, detected by multiple sensors, with high accuracy in a short time. Brief description of the drawings [ Fig. 1] Fig. Figure 1 is a block diagram representing a configuration of a vehicle control system according to a first embodiment of the present disclosure. [ Fig. 2] Fig. 2 is a flowchart that describes the processing of an information aggregation device that powers the vehicle's control system. Fig. 1 forms, represents. [ Fig. 3] Fig. 3 is a flowchart of the detection information summary, which is contained in the Fig. The information summarisation device is included as shown in section 2. [ Fig. 4] Fig. 4 is a schedule for the correction process, which is described in the Fig. The detection information summary shown in section 3 is included. [ Fig. 5] Fig. 5 is a flowchart for the correction parameter calculation, which is described in the Fig. The detection information summary shown in section 3 is included. [ Fig. 6] Fig. Figure 6 is a graph showing an example of time series information from an external sensor of the vehicle control system. Fig. 1 represents. [ Fig. 7A] Fig. Figure 7A is a graphic showing a grouping result of the conventional obstacle detection device. [ Fig. 7B] Fig. Figure 7B is a graphic showing a grouping result of the conventional obstacle detection device. [ Fig. 8A] Fig. 8A is a graphic that shows a grouping result from an information summarization device of the vehicle control system. Fig. 1 shows. [ Fig. 8B] Fig. 8B is a graphic showing the grouping result of the information summarization device of the vehicle control system. Fig. 1 shows. [ Fig. 9] Fig. Figure 9 is a flowchart for calibration determination by a vehicle control system according to a second embodiment of the present disclosure. [ Fig. 10] Fig. Figure 10 is a flowchart for the grouping of a vehicle control system according to a third embodiment of the present disclosure. Description of embodiments

[0011] The following describes embodiments of the vehicle control system of the present disclosure with reference to the drawings. [First embodiment]

[0012] Below, a configuration of a vehicle control system 100 according to a first embodiment of the present disclosure is described first, with reference to the Fig. 1 and Fig. 2 described, and next a process of detection information summarization S200 by an information summarization device 120 is described with reference to Fig. 3 described. (Vehicle control system)

[0013] Fig. Figure 1 is a block diagram illustrating a configuration example of the vehicle control system 100 of the present embodiment. The vehicle control system 100 is, for example, mounted on a vehicle such as a motor vehicle and forms an advanced driver assistance system (ADAS) or an automatic driving system (AD system) of the vehicle. The vehicle control system 100 includes, for example, an external sensor 110, an information aggregation device 120, and a vehicle control device 130. Furthermore, the vehicle control system 100 may, for example, include a vehicle sensor 140, a positioning system 150, a mapping unit 160, and an actuator 170.

[0014] The external sensor 110 is a sensor for detecting an object in the vicinity of the vehicle on which the vehicle control system 100 is mounted. The external sensor 110 detects, for example, the relative position and speed of an object in the vicinity of the vehicle on which the vehicle control system 100 is mounted, as well as the object's shape and type, and similar information. Objects in the vicinity of the vehicle include, for example, other vehicles, pedestrians, roads, sidewalks, road markings such as white lines, traffic signs, signals, curbs, guardrails, buildings, and other obstacles.

[0015] The external sensor 110, for example, contains two sensors: a first sensor 111 and a second sensor 112. The first sensor 111 and the second sensor 112 are, for example, different types of sensors with different object detection principles. Furthermore, the external sensor 110 can contain three or more different types of sensors or multiple sensors of the same type. The external sensor 110 is connected to the information aggregation device 120 and the vehicle control device 130 via a communication network 180, such as a CAN bus, Ethernet (registered trademark), or wireless communication.

[0016] Examples of the multiple sensors included in the external sensor 110 include a millimeter-wave radar device, an image acquisition device containing a monocular or stereo camera, an ultrasonic sonar, a laser radar device, and the like. The first sensor 111, for example, is a sensor that detects the three-dimensional shape of an object in the vicinity of the vehicle and the distance to that object. In particular, the first sensor 111 is, for example, an image acquisition device. The second sensor 112 is, for example, a sensor that measures the distance to an object in the vicinity of the vehicle. In particular, the second sensor 112 is, for example, a distance measuring device such as a millimeter-wave radar device or a laser radar device.

[0017] The information aggregation device 120 groups detection results from multiple sensors that have detected the same object among the multiple sensors contained in the external sensor 110 and outputs aggregated detection information summarizing these detection results. Specifically, the information aggregation device 120, for example, groups first detection information and second detection information, which are the detection results of the first sensor 111 and the second sensor 112 that detected the same object, and outputs aggregated detection information. The information aggregation device 120 is connected to, for example, the vehicle sensor 140, the external sensor 110, the positioning system 150, the mapping unit 160, and the vehicle control unit 130 via the communication network 180 for data transmission.

[0018] The information aggregation device 120 includes, for example, an arithmetic device 121 and a storage device 122. The arithmetic device 121 is, for example, one or more central processing units (CPUs). The storage device 122 includes, for example, one or more main memory devices and auxiliary memory devices. The storage device 122 can be configured, for example, as a hard disk or memory. Furthermore, the information aggregation device 120 includes, for example, a connection unit that inputs and outputs a signal, and an input device for entering a setting value, and the like.

[0019] Fig. Figure 2 is a flowchart that illustrates an example of the processing by the information aggregation device 120. The arithmetic device 5 121 of the information aggregation device 120 first acquires various types of information via the communication network 180 in the information memory S100 and stores the information in the storage device 122. As a result, the storage device 122 stores various types of information, such as behavioral and positional information of the vehicle on which the vehicle control system 100 is mounted, and detection information of objects in the vehicle's environment, road information, and map information.

[0020] In the detection information summary S200, the arithmetic device 121 outputs summarized detection information obtained by combining several units of detection information for the same object from among the units of detection information of the multiple sensors of the external sensor 110, which are stored in the storage device 122. Here, the multiple units of summarized detection information include, for example, first detection information and second detection information, which are the detection results of objects by the first sensor 111 and the second sensor 112.

[0021] In the environmental information summary S300, the arithmetic device 121 combines the summarized detection information described above with various types of information obtained from the storage device 122, such as the vehicle's behavior and position information, and the road and map information in the vehicle's vicinity, and outputs the environmental information. The output environmental information is then transferred to the Fig. 1 vehicle control device 130 entered.

[0022] The vehicle control device 130 controls the vehicle based on the aggregated detection information output by the information aggregation device 120. Specifically, the vehicle control device 130 controls the vehicle based on environmental information about the vehicle's surroundings, which includes the aggregated detection information of the object and various other information. The vehicle control device 130 is, for example, connected to the vehicle sensor 140, the external sensor 110, the positioning system 150, the mapping unit 160, the information aggregation device 120, and the actuator 170 via data transmission.

[0023] The vehicle control unit 130 generates a travel plan containing the route, speed, acceleration, and other parameters of the vehicle equipped with the vehicle control unit 100, based on, for example, outputs from the vehicle sensor 140, the external sensor 110, the positioning system 150, the map unit 160, and the information summarization device 120. The vehicle control unit 130 implements the ADAS and AD by controlling the actuator 170 based on the generated travel plan.

[0024] The vehicle sensor 140 is a sensor that measures the speed, acceleration, steering angle, yaw rate, and the like of the vehicle on which the vehicle control system 100 is mounted. In particular, the vehicle sensor 140 includes, for example, a gyroscope, a wheel speed sensor, a steering angle sensor, an acceleration sensor, a brake sensor, an accelerator pedal sensor, a displacement sensor, and the like. The vehicle sensor 140 is connected to the information aggregation device 120 and the vehicle control device 130 via, for example, the communication network 180, enabling data transmission.

[0025] The positioning system 150 measures the position of the vehicle on which the vehicle control system 100 is mounted. The positioning system 150 can be, for example, a satellite positioning system such as a global navigation satellite system (GNSS) or a global positioning system (GPS). The positioning system 150 is connected to the information aggregation device 120, the vehicle control device 130, and the map unit 160 via, for example, the communication network 180, enabling data transmission.

[0026] The map unit 160, for example, contains an arithmetic device and a storage device, acquires the position of the vehicle on which the vehicle control system 100 is mounted from the positioning system 150, and selects and outputs map information about the vehicle's surroundings. The map unit 150 is connected to the information aggregation device 120, the vehicle control device 130, and the positioning system 160 via, for example, the communication network 180, enabling data transmission.

[0027] The actuator 170 causes the vehicle, on which the vehicle control system 100 is mounted, to drive in accordance with, for example, command information from the vehicle control device 130. In particular, the actuator 170 implements the ADAS and AD of the vehicle, for example, by actuating an accelerator pedal, a brake, a transmission, a steering system, a turn signal indicator, and the like of the vehicle in accordance with the command information from the vehicle control device 100. (Detection information summary)

[0028] Next, the detection information summary S200, which is processed by the in Fig. The information summarisation device 120 shown is described in detail. Fig. 3 is a flowchart detailing the detection information summary S200, which is contained in the Fig. The processing of the information summarisation device 120 shown in section 2 is included.

[0029] The detection information summary S200, for example, contains correction S210, instantaneous value grouping S202, summary update S203, time series conversion S204 of the summarized detection information, storage S205 of the summarized time series information, instantaneous value conversion S206 of the summarized time series information, and forecast update S207 of the summarized detection information. The detection information summary S200, for example, contains time series conversion S208 of detection information, storage S209 of the time series information, calibration determination S210, calibration S220, and storage S211 of the correction parameter memory.

[0030] First, in correction S201, the arithmetic device 121 of the information summarization device 120 corrects detection information of the specific sensor from among the multiple units of detection information that are the detection results of the object by the multiple sensors of the external sensor 110, which are stored in the storage device 122. In particular, the arithmetic device 121, for example, corrects the first detection information of the first sensor 111 from among multiple units of detection information that include the first detection information of the first sensor 112 and the second detection information of the second sensor 111.

[0031] As described above, the first sensor 111 is, for example, an image acquisition device such as a stereo camera, and the first detection information is the position and velocity of the object detected by the image acquisition device. Additionally, the second sensor 112 is, for example, a millimeter-wave radar device, and the second detection information is the position and velocity of the object detected by the millimeter-wave radar device.

[0032] Fig. 4 is a flowchart for correction S201, which is in the detection information summary S200 in Fig. 3 is included. For example, correction S201 includes detection information selection S201a, sensor observation quantity calculation S201b, sensor observation quantity correction S201c, correction information calculation S201d and end determination S201e.

[0033] In the detection information selection S201a, the arithmetic device 121 selects detection information from one sensor among the multiple sensors of the external sensor 110. Specifically, the arithmetic device 121 selects, for example, the first detection information of the first sensor 111 in the detection information selection S201a. Next, in the sensor observation quantity calculation S201b, the arithmetic device 121 calculates an observation quantity of the selected sensor from the sensor's detection information. Finally, in the sensor observation quantity correction S201c, the arithmetic device 121 corrects an observation quantity of the sensor using a correction parameter acquired from the storage device 122.Next, in the correction information calculation S201d, the arithmetic device 121 calculates correction information, which is corrected detection information, using the sensor's observation quantity after correction.

[0034] In particular, in sensor observation quantity calculation S201b, the arithmetic device 121 calculates a pixel position from which the sensor observation quantity is derived, for example, the position and velocity of the object, which are the initial detection information of the first sensor 111, which is the image acquisition element. Next, in sensor observation quantity correction S201c, the arithmetic device 121 corrects the pixel position, which is the observation quantity of the first sensor 111, such as the image acquisition element, for example, using the correction parameter acquired from the storage device 122. Next, in correction information calculation S201d, the arithmetic device 121 calculates, for example, the position and velocity of the object based on the corrected pixel position as the correction information.It is noted that the observation parameter of the second sensor 112, which is, for example, a millimeter wave radar device, is, for example, a distance and an angle from the second sensor 112 to an object.

[0035] After calculating the correction information for the selected sensor, the arithmetic device 121 determines in the end determination S201e whether the calculation of the correction information for all sensors to be subjected to correction information has been completed. If it is determined that the calculation of the correction information has not been completed for all sensors for which correction information is to be calculated (No), the arithmetic device 121 returns to the detection information selection S201a. If it is determined that the calculation of the correction information has been completed for all sensors for which correction information is to be calculated (Yes), the arithmetic device 121 terminates correction S201.

[0036] For example, in an initial state, the storage device 122 stores correction parameters at the time of product shipment from several sensors, including the first sensor 111, which forms the external sensor 110, as correction parameters used in correction S201. The correction parameter stored in the storage device 122 is updated, for example, in a correction parameter memory S211 described below.

[0037] After the in the Fig. 3 and Fig. Once correction S201, as shown in Figure 4, is complete, the arithmetic device 121 performs the following processing in instantaneous value grouping S202. The arithmetic device 121 performs instantaneous value grouping using the correction information calculated for the target sensor in correction S201 and the detection information from the sensor that is not the target of correction S201, among the multiple sensors that comprise the external sensor 110. As a result, the arithmetic device 121 outputs aggregated detection information for the same object detected by the multiple sensors that comprise the external sensor 110.

[0038] In particular, the arithmetic device 121 performs instantaneous value grouping S202 using, for example, correction information obtained by correcting the first detection information of the first sensor 111, which is an image acquisition device, and, for example, second correction information of the second sensor 112, which is a millimeter-wave radar device. Here, the instantaneous value grouping uses a method that differs from time-series grouping based on the time-series information of the detection information and the correction information of the sensor, that is, the temporal transition of the detection information and the correction information.In instantaneous value grouping, the detection information and the correction information of the multiple sensors are grouped using the instantaneous value of the sensor's detection information, that is, the detection information captured in a sample, and the correction information thereof.

[0039] The arithmetic device 121 performs the instantaneous value grouping of the detection information and the correction information of the multiple sensors that constitute the external sensor 110, for example, using the following equation (1) stored in the storage device 122. Equation (1) represents a Mahalanobis distance. In the instantaneous value grouping S202, the arithmetic device 121 uses, for example, the detection information of the multiple sensors that constitute the external sensor 100 and the predictive information of the correction information. This predictive information is calculated in the prediction update S207 of the aggregated detection information, which is described below. [Equation 1] (xp,i,t−xs,j,t)tpp,i,t,s,j,t−1(xp,i,t−xs,j,t)

[0040] In equation (1) x denotes s,j,t for example, detection information or correction information, and xp,i,t This refers to the predictive information, the detection information, and the correction information. For example, it is assumed that the detection information or the correction information follows a normal distribution where the predicted value is a mean, and information that the covariance P p,i,t,s,j,t is obtained. It is noted that a subscript p represents a prediction, s represents a sensor, i represents an identification number of a prediction value, j represents an identification number of detection information or correction information, and t represents a time point.

[0041] In this case, the Mahalanobis distance between the detection information and the correction information and their predicted values ​​is expressed by the preceding equation (1). In instantaneous value grouping S202, for example, in a case where the Mahalanobis distance is equal to or less than a threshold value, the arithmetic device 121 sets the detection information or the correction information of the multiple sensors as a grouping target. The arithmetic device 121 then stores the corresponding predicted values ​​of the detection information and the correction information, and the identification numbers of the detection information or the correction information of the multiple sensors to be grouped, in the storage device 122.

[0042] After completing the instantaneous value grouping S202, the arithmetic device 121 outputs the summarized detection information in the summary update S203, based, for example, on the detection information and correction information of the multiple sensors that make up the external sensor 110, and the detection information identification number or correction information of the grouping target. This summarized detection information is, for example, information obtained by estimating the state of an object detected by the multiple sensors that make up the external sensor 110. Using the summarized detection information output in the summary update S203, the arithmetic device 121 outputs the environment information as described above in the Fig. The environmental information summary shown in section 2 is described for S300.

[0043] Additionally, the arithmetic device 121 adds the summarized detection information in the time series conversion S204, which is in Fig. Figure 3 shows the latest summarized detection information output in the summary update S203, added to the time series information, which is the time series of past summarized detection information stored in storage device 122. Then, the arithmetic device 121 outputs the latest summarized time series information to which the latest summarized detection information has been added. Additionally, the arithmetic device 121 stores the latest summarized time series information in storage device 122 in the summarized time series information memory S205.

[0044] Additionally, for example, the arithmetic device 121 converts the aggregated time series information stored in the storage device 122 into an instantaneous value in the instantaneous value conversion S206 of the aggregated time series information and outputs the prediction information in the prediction update S207 of the aggregated detection information, using the aggregated detection information converted into the instantaneous value. This prediction information is, for example, a result of predicting the state of the object detected by the multiple sensors that make up the external sensor 110 after the time corresponding to the time interval during which the sensor aggregation is performed has elapsed. This prediction information is used, for example, in the instantaneous value grouping S202 as described above.

[0045] Additionally, the storage device 122 stores, for example, time series information, which is a time series of previous detection information from several sensors that constitute the external sensor 110. In the time series conversion S208 of the detection information, the arithmetic device 121 outputs the latest time series information, which is obtained, for example, by adding the latest detection information from the several sensors that constitute the external sensor 110 to the time series information stored in the storage device 122. Furthermore, the arithmetic device 121 stores the latest time series information in the storage device 122 in the time series information memory S209.

[0046] Additionally, in calibration procedure S210, the arithmetic device 121 determines the need to calibrate at least one of the several sensors that make up the external sensor 110. For example, in calibration procedure S210, the arithmetic device 121 calculates a calibration determination index based on the instability of the instantaneous value grouping S202 and calculates the correction parameter in calibration S220 when the calibration determination index becomes equal to or greater than the threshold value. For example, the calibration determination index is obtained by summarizing an index that indicates the instability of the instantaneous value grouping S202 over time.

[0047] This means, for example, that in a normal state, where the calibration determination index in calibration determination S210 is equal to or less than the threshold, the arithmetic device 121 repeats the processing from correction S210 to calibration determination S210 in a predefined cycle. Then, the arithmetic device 121 calculates, for example, the correction parameter in calibration S200 only in an anomalous state, where the calibration determination index in calibration determination S210 exceeds the threshold. Calibration S220, for example, includes grouping S221 and the correction parameter calculation S222.

[0048] The calibration determination index can, for example, be a variable with an initial value of zero. In this case, the arithmetic device 121 adds 1 to the calibration determination index for each processing operation and stores the result in the memory device 122. For example, in calibration determination S210, the arithmetic device 121 outputs binary information indicating whether the calibration determination index is a predefined threshold or greater. For example, if the binary information of the calibration determination index is "true," that is, if the calibration determination index is greater than or equal to a predefined threshold, the arithmetic device 121 initializes the calibration determination index and stores the calibration determination index as 0 in the memory device 122.

[0049] If calibration procedure S210 determines that the sensor needs to be calibrated, the arithmetic device 121, for example, starts the time series grouping in grouping S221. In the Fig. In the example shown in Figure 3, the arithmetic device 121 uses the following two units of information as inputs for time series grouping. One is the time series, that is, the time series information of the detection data from the multiple sensors, which is output in the time series conversion S208 of the detection data. The other is the aggregated time series information, which is output in the time series conversion S204 of the aggregated detection data. In grouping S221, the arithmetic device 121 estimates information in which the same object is detected among the time series information from the multiple sensors using these two units of information, aggregates the units of information, and outputs the aggregated time series information.

[0050] In this grouping S221, the arithmetic device 121 performs time-series grouping of the time-series information from the multiple sensors using, for example, the preceding equation (1) and the following equation (2), which are stored in the storage device 122. Equation (2) represents a time-averaged Mahalanobis distance. The time-averaged Mahalanobis distance is, for example, a distance between time-series information from multiple sensors and aggregated time-series information that includes prediction information. In this case, the time-averaged Mahalanobis distance is obtained by averaging the Mahalanobis distance of the preceding equation (1) over a common time interval from t = t1 to t = t2, during which detection information from two objects existed. [Equation 2] 1t2−t1+1∑t=t1t2(xp,i,t−xs,j,t)tpρ,i,t,s,j,t−1(xp,i,t−xs,j,t)

[0051] In grouping S221, for example, in a case where the time-averaged Mahalanobis distance in the preceding equation (2) is equal to or less than the threshold, the arithmetic device 121 groups the sensor time series information output in the time series conversion S208 of the detection information and the aggregated time series information output in the time series conversion S204 of the aggregated detection information. The arithmetic device 121 then stores the identification number of the sensor time series information grouped into the aggregated time series information in the storage device 122.

[0052] Fig. 5 is a flowchart that provides an example of the correction parameter calculation S222, which is included in the detection information summary S200 in Fig. 3 is included. The correction parameter calculation S222 includes, for example, sensor selection S222a, grouping determination S222b, calculation S222c of reference time series information, calculation S222d of sensor observation quantity time series information, calculation S222e of a correction parameter, end determination S222f and deletion S222g of time series information.

[0053] In sensor selection S222a, the arithmetic device 121 selects, for example, one sensor as a target for calculating the correction parameter from among the several sensors that make up the external sensor 110. Specifically, in sensor selection S222a, the arithmetic device 121 selects, for example, the first sensor 111, which is the image acquisition sensor. Next, in grouping determination S222b, the arithmetic device 121 determines whether the time series information of the grouped detection information exists in the aggregated time series information for the selected sensor.

[0054] In grouping determination S222b, if the time series information of the detection information grouped in the summarized time series information does not exist (No), the arithmetic device 121 proceeds to the end determination S222f described below. Additionally, in grouping determination S222b, if the time series information of the detection information grouped in the summarized time series information exists (Yes), the arithmetic device 121 proceeds to the reference time series information calculation S222c. Specifically, in grouping determination S222b, the arithmetic device 121 determines whether the time series information of the detection information grouped in the summarized time series information exists for the selected first sensor 111.

[0055] In calculation S222c of the reference time series information, the arithmetic device 121 calculates the time series information of the sensor's observed quantity, which serves as the correction reference for the selected sensor, as the reference time series information. Specifically, in calculation S222c of the reference time series information, the arithmetic device 121 calculates the pixel position time series information, which serves as the correction reference, as the reference time series information based on the aggregated time series information for the first sensor 111, which is the selected image acquisition device.

[0056] In calculation S222d of the sensor observation time series information, the arithmetic device 121 calculates the time series information of the sensor observation as the sensor observation time series information based on the time series information that is the detection information for the selected sensor. Specifically, in calculation S222d of the sensor observation time series information, the arithmetic device 121 calculates the time series information of the pixel position to be corrected as the sensor observation time series information based on the time series information for the first sensor 111, which is the selected image acquisition device.

[0057] In the calculation S222e of the correction parameter, the arithmetic device 121 calculates a mean value of the differences between the reference time series information and the sensor observation time series information at each time point for the selected sensor and stores the mean value of the differences in the storage device 122 as the correction parameter. Specifically, in the calculation S222e of the correction parameter, the arithmetic device 121 calculates a mean value of the differences between the pixel positions at each time point of the reference time series information and the sensor observation time series information for the first sensor 111, which is the selected image acquisition element, and stores the mean value of the differences between the pixel positions in the storage device 122 as the correction parameter.

[0058] In the final determination S222f, the arithmetic device 121 determines whether the calculation of the correction parameters for all sensors to be selected from among the multiple sensors comprising the external sensor 110 is complete. If the calculation of the correction parameters is not complete for all sensors to be selected (No), the arithmetic device 121 returns to sensor selection S222a, and if the calculation of the correction parameters is complete for all sensors to be selected (Yes), the arithmetic device 121 proceeds to the deletion S222g of the time series information.

[0059] In the deletion S222g of the time series information, the arithmetic device 121 deletes the time series information prior to the specified time from the sensor's time series information, which is stored in the memory device 122 in memory S209 of the time series information, and terminates the calculation S222 of the correction parameter. Afterwards, the arithmetic device 121 stores the correction parameter in memory S211, which is located in Fig. 3 shows the correction parameter calculated in the calculation S222e of the correction parameter in the storage device 122.

[0060] The operation of the vehicle control system 100 of the present embodiment is described below on the basis of a comparison with the prior art. Fig. Figure 6 is a graph showing an example of time series information from several sensors in the external sensor 110 of the vehicle control system 100. Fig. 1 is included, represents. Fig. 7A and Fig. Figure 7B shows graphs depicting grouping results from conventional obstacle detection devices. Fig. 8A and Fig. 8B are graphs showing grouping results by the information summarization device 120 of the vehicle control system 100 in Fig. 1.

[0061] In each graph, a horizontal axis t represents time, and a vertical axis x represents detection information, such as the distance and angle of an object detected by a sensor. Furthermore, in each graph, a solid line and a dashed line, for example, represent time series information of detection data for the same object detected by different types of sensors, and a dashed line represents time series information of detection data for another object detected by a different type of sensor from among those sensors.

[0062] The vehicle control system 100 performs sensor detection summaries, which compile information about objects detected by several different types of external sensors (e.g., the external sensor 110) for the detection of a motor vehicle running the AD. A travel plan is prepared within the AD, and a travel determination is made based on the result of an object detection in the vehicle's environment by a sensor. Various sensors are used to detect an object, such as radar, cameras, sonar, and laser radar.

[0063] These sensors have different requirements, such as detection range, detectable object, detection accuracy, and cost. Therefore, it is necessary to combine multiple sensors for a specific purpose. When several sensors are combined, multiple units of information, each detecting the same object, can be generated. It is necessary to identify these units of detection information as being about the same object, aggregate them, and output highly accurate, summarized detection information. This process of identifying detection information about the same object from multiple sensors is called grouping.

[0064] As described above, the conventional obstacle detection device determines that the two obstacles are the same obstacle if the obstacle position detected by the first measuring device and the obstacle position detected by the second measuring device overlap at the time t, t+1, t+2,... and t+n with respect to the multiple units of time series information, as shown in Fig. Figure 6 shows that the sensor's observation parameter can change over time, for example, due to the influence of vibration or thermal stress.

[0065] Therefore, for example, if the sensor's observation parameter changes over time, the instantaneous value grouping of the conventional obstacle detection device, which determines whether the obstacles detected by the multiple measuring devices are the same at any given time, may contain an error in the determination result. In particular, for example, as in the Fig. 7A and Fig. As shown in Figure 7B, there is a possibility that the detection information units from several different objects are grouped as detection information from the same object, and grouping with high accuracy is difficult. If the sensors are calibrated in a situation where a grouping error occurs as described above, there is a concern that the calibration accuracy of the sensors may be degraded.

[0066] Furthermore, in the conventional action detection device described above, grouping can be performed with higher accuracy than instantaneous grouping by time-series grouping based on the temporal transition of multiple sensor data units. However, in this conventional action detection device, because the sensor is not calibrated, an error in the detection information of the object detected by each sensor cannot be removed. Therefore, in the conventional obstacle detection device, it is conceivable to calibrate the sensor by time-series grouping instead of instantaneous grouping. However, since time-series grouping requires time-series information, it takes longer than instantaneous grouping, which does not require time-series data.

[0067] To solve such a problem from the prior art, the vehicle control system 100 of the present embodiment includes the first sensor 111 and the second sensor 112, which form the external sensor 110, the information aggregation device 120, and the vehicle control device 130. The first sensor 111 and the second sensor 112 detect an object in the vehicle's environment. The information aggregation device 120, for example, groups the first detection information and the second detection information, which are the detection results of the first sensor 111 and the second sensor 112 that detected the same object, and outputs aggregated detection information. The vehicle control device 130 controls the vehicle based on the aggregated detection information. The information aggregation device 120 includes the arithmetic device 121 and the storage device 122.The arithmetic device 121 stores in the storage device 122 first time series information, which is the time series information of the first detection information, and the second time series information, which is the time series information of the second detection information, in the storage device S209 of the time series information, which is in . Fig. 3 is shown. Additionally, the arithmetic device 121 groups in the in Fig. In the calibration S220 shown in Figure 3, the first and second time series information are used when the first sensor 111 and the second sensor 112 detect the same object in order to calculate the correction parameter of the first detection information. Additionally, the arithmetic device 121 calculates correction information, in which the first detection information is corrected, using the correction parameter in correction S201. Fig. 3. Then the arithmetic device 121 outputs the summarized detection information through the instantaneous value grouping using the correction information and the second detection information in the instantaneous value grouping S202 and the summary update S203, which are in Fig. 3 is shown.

[0068] With this configuration, the vehicle control system 100 of the present embodiment can perform the instantaneous value grouping S202 by the information summarization device 120 using the correction information obtained by correcting the first detection information of the first sensor 111 during normal driving. In other words, the vehicle control system 100 of the present exemplary embodiment can perform the calibration S220 only when the arithmetic device 121 of the information summarization device 120 determines in the calibration determination S210 that calibration is necessary, and can perform the time series grouping in the grouping S221. That is, since the vehicle control system 100 does not perform the time series grouping during normal driving, it is not necessary to wait for the time series information of the sensor's detection information to be prepared.

[0069] As a result, during normal operation, the processing time for grouping the detection information of the multiple sensors contained in the external sensor, for example, the first sensor 111 and the second sensor 111, can be reduced compared to time-series grouping. Additionally, when the arithmetic device 121 determines that calibration is necessary, the correction parameter of the first detection information of the first sensor 111 is calculated using the time-series information of the first detection information. Thus, the correction information of the first detection information can be calculated accurately, independent of any temporal changes in the observation parameter of the first sensor 111. By using this correction information, as described in the Fig. 8A and Fig. As shown in Figure 8B, it is possible to prevent detection information from several different objects from being grouped as detection information from the same object, and to perform the grouping with high accuracy compared to the conventional obstacle detection device.

[0070] Additionally, in the vehicle control system 100 of the present embodiment, the arithmetic device 121 of the information summarization device 120, for example, repeats the output of the summarized detection information in a predetermined cycle in the summary update S203, as shown in Fig. 3 represents. Furthermore, the arithmetic device 121 stores the summarized time series information, which is the time series information of the summarized detection information, in the storage device 122 in the time series conversion S204 of the summarized detection information and the storage S205 of the summarized time series information.

[0071] With this configuration, the vehicle control system 100 of the present embodiment can perform the grouping S221 using the aggregated time series information stored in the storage device 122 when it performs the calibration S220 by the arithmetic device 121 of the information aggregation device 120. Therefore, in the grouping S221, the grouping S221 can be performed using the aggregated time series information stored in the storage device 122 without waiting for the aggregated detection information to be prepared.Furthermore, the arithmetic device 121 can calculate the detection information and the prediction information of the correction information from the multiple sensors that make up the external sensor 110 in the prediction update S207 of the aggregated detection information and can perform the instantaneous value grouping using the prediction information in the instantaneous value grouping S202. As a result, the instantaneous value grouping S202 can be performed more easily and with high accuracy.

[0072] Additionally, in the vehicle control system 100 of the present embodiment, the arithmetic device 121 of the information summarization device 120 calculates the calibration determination index, for example, based on the instability of the instantaneous value grouping in the Fig. 3. Calibration determination S210 shown. Then, if the calibration determination index becomes equal to or greater than the threshold value, the arithmetic device 121 performs the calibration S220 to calculate the correction parameter.

[0073] With this configuration, the vehicle control system 100 of the present embodiment can detect a change in the observation parameter of the sensor contained in the external sensor 110, such as the first sensor 111, based on the calibration determination index by the arithmetic device 121 of the information summarization device 120. Accordingly, the correction parameter can be automatically updated according to the change in the sensor's observation parameter, and the instantaneous value grouping S202 by the arithmetic device 212 can be performed with higher accuracy.

[0074] In addition, in the vehicle control system 100 of the present embodiment, the arithmetic device 121 of the information summarization device 120 calculates the correction parameter by grouping based on the time transition of the first time series information of the first sensor 111 and the time transition of the second time series information of the second sensor 112.

[0075] In particular, as in Fig. 3 shown, the arithmetic device 121 in grouping S221 which is in the Fig. 8A and Fig. Figure 8B shows a time series grouping using the time series information, which is the time series of detection information from the multiple sensors output in the time series conversion S208 of the detection information. Then the arithmetic device 121 calculates the correction parameter in the Fig. 5 shown correction parameter calculation S222. With this configuration, a more accurate correction parameter can be calculated, and the arithmetic device 121 can perform the instantaneous value grouping S202 with higher accuracy.

[0076] As described above, according to the present embodiment it is possible to provide a vehicle control system 100 that is capable of grouping the multiple detection results of the same object detected by the multiple sensors with high accuracy in a short time. [Second embodiment]

[0077] Next, a vehicle control system 100 according to a second embodiment of the present disclosure is described with reference to the Fig. 1 to 8 of the first embodiment described above and Fig. 9 described. The vehicle control system 100 of the present embodiment differs from that of the first embodiment in one configuration of the information summarization device 120. Since the other points of the vehicle control system 100 of the present embodiment are the same as those of the vehicle control system 100 according to the first embodiment, the same reference numerals are assigned to the same parts, and their descriptions are omitted.

[0078] In the vehicle control system 100 of the present embodiment, the arithmetic device 121 of the information summarization device 120 has the following features. If the result of the instantaneous value grouping S202 using the latest correction information from the first sensor 111 and the second detection information from the second sensor 112 differs from the result of the instantaneous value grouping S202 using the past correction information and the second detection information, the arithmetic device 121 calculates the calibration determination index based on the number of units of differing information between the result of the latest instantaneous value grouping S202 and the result of the past instantaneous value grouping S202. The arithmetic device 121 then calculates the correction parameter when the calibration determination index becomes equal to or greater than the threshold value.

[0079] Fig. Figure 9 is a flowchart of the calibration determination S210 by the arithmetic device 121 of the present embodiment. The calibration determination S210 includes, for example, detection information selection S210a, grouping cancellation addition S210b, grouping addition S210c, selection end determination S210d, binary information output setting S210e, and index initialization S210f.

[0080] In the detection information selection S210a, the arithmetic device 121 selects one of the aggregated detection information units grouped in the earlier and latest instantaneous value groupings S202. In the grouping extinction addition S210b, the arithmetic device 121 adds to the calibration determination index the number of detection information units excluded from the target grouping in the latest instantaneous value grouping S202, from among the latest detection information of the multiple sensors that are aggregated into one unit of aggregated detection information in the earlier instantaneous value grouping S202.

[0081] In the grouping addition S210c, the arithmetic device 121 adds the number of detection information units to be grouped in the latest instantaneous value grouping S202 to the calibration determination index, including the latest detection information from the multiple sensors that were not grouped in the earlier instantaneous value grouping S202. In the selection completion determination S210d, the arithmetic device 121 determines whether all units of aggregated detection information that were grouped in the earlier and latest instantaneous value grouping S202 have been selected.

[0082] In selection end condition S210d, the arithmetic device 121 returns to detection information selection S210a if the selection of all units of summarized detection information grouped in the earlier and latest instantaneous value groupings S202 is not complete (No). In selection end condition S210d, the arithmetic device 121 proceeds to binary information output setting S210e if the selection of all units of summarized detection information grouped in the earlier and latest instantaneous value groupings S202 is complete (Yes).

[0083] In binary information output setting S210e, the arithmetic device 121 determines whether the calibration determination index is equal to or greater than the threshold and, for example, sets the binary information as true or false as an output. Specifically, in binary information output setting S210e, the arithmetic device 121 sets false as the output if the calibration determination index is less than the threshold, sets true as the output if the calibration determination index is equal to or greater than the threshold, and proceeds to index initialization S210f. If true as the output is set in binary information output setting S210e, the arithmetic device 121 initializes the calibration determination index to 0 in index initialization S210f, outputs true in the binary information, and terminates calibration determination S210.

[0084] According to the vehicle control system 100 of the present embodiment, if the correction parameter in the information summarization device 120 is unsuitable, the correction parameter can be updated to a suitable value earlier than in the vehicle control system 100 according to the first embodiment described above. However, if the correction parameter in the information summarization device 120 is suitable, the time until the correction parameter is updated can be extended compared to the vehicle control system 199 according to the first embodiment described above. As a result, the frequency of the calibration determination S210 by the arithmetic device 121 can be reduced, and the processing time can be reduced. [Third embodiment]

[0085] Next, a vehicle control system 100 according to a second embodiment of the present disclosure is described with reference to the Fig. 1 to 8 of the first embodiment described above and Fig. 10 described. The vehicle control system 100 of the present embodiment differs from that of the first embodiment in one configuration of the information summarization device 120. Since the other points of the vehicle control system 100 of the present embodiment are the same as those of the vehicle control system 100 according to the first embodiment, the same reference numerals are assigned to the same parts, and their descriptions are omitted.

[0086] In the vehicle control system 100 of the present embodiment, the arithmetic device 121 of the information summarization device 120 calculates the correction parameter by grouping based on a time series interval between the first time series information from the first sensor 111 and the second time series information from the second sensor 112. That is, in the present embodiment, the arithmetic device 121 of the information summarization device 120 performs the grouping S221 based on the time series interval between the first time series information and the second time series information, instead of grouping S221 using the time transition between the first time series information and the time transition between the second time series information.In other words, in the present embodiment, the arithmetic device 121 in the grouping S221 summarizes the time series information, which is the time series of the detection information of the multiple sensors, by clustering, which is one of the unsupervised learning methods.

[0087] Fig. Figure 10 is a flowchart of the grouping S221 in the calibration S220 by the arithmetic device 121 of the present embodiment. The grouping S221 of the calibration S220 includes, for example, the generation S221a of a prioritized queue, the calculation S221b of the time-averaged Mahalanobis distance, the minimum value output S221c, the minimum value determination S221d, the time series information summary S221e, and the update S221f of the time-averaged Mahalanobis distance.

[0088] In the generation of the prioritized queue S221a, the arithmetic device 121 creates an empty priority queue in the storage device 122. The prioritized queue is, for example, a collection from which a minimum value can be preferentially retrieved. In the computation S221b of the time-averaged Mahalanobis distance, the arithmetic device 121 calculates the time-averaged Mahalanobis distance between the time series information of the multiple sensors using the preceding equation (2) and adds the time-averaged Mahalanobis distance to the priority queue of the storage device 122.

[0089] In the minimum value output S221c, the arithmetic device 121 extracts a combination of the sensor's time series information that minimizes the time-averaged Mahalanobis distance output from the priority queue. In the minimum value determination S221d, the arithmetic device 121 determines whether the time-averaged Mahalanobis distance of the combination of extracted time series information is equal to or less than the threshold. If the time-averaged Mahalanobis distance exceeds the threshold in the minimum value determination S221d (No), the arithmetic device 121 terminates the grouping S221. Conversely, if the time-averaged Mahalanobis distance is equal to or less than the threshold in the minimum value determination S221d (Yes), the arithmetic device 121 proceeds to the time series information summarization S221e.

[0090] In the time series information summary S221e, the arithmetic device 121 summarizes the time series information of the sensor whose time-averaged Mahalanobis distance is equal to or less than the threshold. In the time-averaged Mahalanobis distance update S221f, the arithmetic device 121 updates the time-averaged Mahalanobis distance between the time series information of the multiple sensors. Furthermore, the arithmetic device 121 removes the time-averaged Mahalanobis distance relating to the combination of the sensors' time series information before the summary from the priority queue, adds the time-averaged Mahalanobis distance relating to the time series information after the summary to the priority queue, and returns to the minimum value output S221c.

[0091] According to the vehicle control system 100 of the present embodiment, the aggregated time series information in the grouping S221 of the calibration S220 is not used by the arithmetic device 121 of the information aggregation device 120. As a result, even if faulty grouping is performed in the instantaneous value grouping S202, it is possible to prevent an error in the grouping result from propagating to the storage S205 of the aggregated time series information and the storage S211 of the correction parameter. Therefore, the influence of the error in the correction parameter calculation S222 can be eliminated, and the sensor's detection information can be corrected with high accuracy in the correction S201.

[0092] Although the embodiments of the vehicle control system have been precisely described in accordance with the present disclosure with reference to the drawings, the specific configuration is not limited to the embodiments, and even if there are changes and the like in the area which does not deviate from the core statement of the disclosure, these changes to the design are included in this disclosure. Reference symbol list 100 Vehicle control system 111 first sensor 112 second sensor 120 Information summarization device 121 Arithmetic device 122 Storage device 130 Vehicle control device

Claims

[1] Vehicle control system (100) comprising: a first sensor (111) and a second sensor (112) that detect an object in the vicinity of a vehicle; an information summarization device (120) that groups and outputs the first detection information and second detection information, the detection results of the first sensor (111) and the second sensor (112) detecting the same object; and a vehicle control device (130) that controls the vehicle based on the aggregated detection information, wherein the information summarisation device (120) includes an arithmetic device (121, 212) and a storage device (122), and the arithmetic device (121, 212) stores the first time series information, which is the time series information of the first detection information, and the second time series information, which is the time series information of the second detection information, in the storage device (122), The first time series information and the second time series information are grouped when the first sensor (111) and the second sensor (112) detect the same object in order to calculate a correction parameter of the first detection information. Correction information obtained by correcting the first Detection information is obtained using the correction parameter, calculated, and outputs the summarized detection information by instantaneous value grouping using the correction information and the second detection information, wherein the arithmetic device (121, 212) repeats the output of the aggregated detection information in a predetermined cycle and stores aggregated time series information, which is the time series information of the aggregated detection information, in the storage device (122), and wherein the arithmetic device (121, 212) calculates a calibration determination index based on the instability of the instantaneous value grouping and calculates the correction parameter when the calibration determination index becomes equal to or greater than a threshold value. [2] Vehicle control system (100) according to claim 1, wherein, when a result of grouping using new correction information and the second detection information differs from a result of grouping using the previous correction information and the second detection information, the arithmetic device (121, 212) calculates a calibration determination index based on the number of units of different information between the new grouping result and the previous grouping result, and when the calibration determination index is equal to or greater than a threshold value, the arithmetic device (121, 212) calculates the correction parameter. [3] Vehicle control system (100) according to claim 1, wherein the arithmetic device (121, 212) calculates the correction parameter by grouping based on a time transition of the first time series information and a time transition of the second time series information. [4] Vehicle control system (100) according to claim 1, wherein the arithmetic device (121, 212) calculates the correction parameter by grouping based on a time series interval of the first time series information and the second time series information.

Images