Vehicle control device, vehicle control method, and vehicle control program

Abstract

To provide a vehicle control device and a vehicle control method that can improve safety during automatic driving.SOLUTION: An automatic driving device 20 comprises: a fusion unit F2 that recognizes a driving environment based on an output signal of a surrounding monitoring sensor; and a control planning unit F3 that performs control planning based on a recognition result of the fusion unit F2. Also, a diagnostic device 30 comprises an abnormality detection unit G4 that detects an abnormality in a recognition system from the surrounding monitoring sensor to the fusion unit F2 by monitoring the output signal of the surrounding monitoring sensor and the recognition result of the fusion unit F2 in a time sequential manner. The diagnostic device 30 requests the control planning unit F3 to execute an MRM based on a fact that the abnormality detection unit G4 has detected the abnormality in the recognition system.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to techniques for generating control plans for autonomous vehicles. [Background technology]

[0002] Patent Document 1 discloses a configuration for generating a vehicle travel plan, in other words a control plan, for autonomous driving using a mathematical formula model called an RSS (Responsibility Sensitive Safety) model and map data.

[0003] In the RSS model, the planner, which is a functional block that formulates control plans, uses map data to calculate the potential accident liability value for each of multiple control plans, and adopts a control plan whose potential accident liability value is within an acceptable range. The potential accident liability value is a parameter that indicates the degree of responsibility of the vehicle in the event of an accident between the vehicle and surrounding vehicles around the vehicle. The potential accident liability value is a value that takes into account whether the distance between the vehicle and surrounding vehicles is shorter than the safe distance determined based on the road structure, etc. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 115963 Summary of the Invention [Problem to be solved by the invention]

[0005] In the RSS model, the planner calculates the potential accident liability value for each control plan generated based on map data and sensing information from surrounding monitoring sensors such as cameras, and selects the plan that is judged to be relatively safe. However, if there is an abnormality in the surrounding monitoring sensor itself or in the system that performs object recognition processing based on the observation data from the surrounding monitoring sensor (hereinafter referred to as the external recognition system), there is a possibility that the calculation of the potential accident liability value will be incorrect. Naturally, if the evaluation of the potential accident liability value is incorrect, the safety of autonomous driving will be reduced.

[0006] The present disclosure has been made based on the above circumstances, and an object of the present disclosure is to provide a vehicle control device that can improve safety during autonomous driving. [Means for solving the problem]

[0007] A vehicle control device for achieving this purpose is, for example, a vehicle control device used in a vehicle that includes a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, and detects an object present within a predetermined detection range. multiple and an abnormality detection unit (G4) that detects an abnormality in an external recognition system including the periphery monitoring sensors and the recognition unit based on at least one of output signals of the periphery monitoring sensors (11, 11A to 11D, 17, 17A to 17D) and a recognition result of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object existing around the vehicle based on the output signals of the periphery monitoring sensors, and a vehicle stop processing unit (G6) that executes a process to stop the vehicle based on the detection of an abnormality in the external recognition system by the abnormality detection unit, and the abnormality detection unit is used for the control planning unit. By external recognition system Used for processing and control planning By external recognition system Used in the safety evaluation section as well as processing By external recognition system If the results are different, External recognition system It is configured to determine that an abnormality has occurred. The process by the external recognition system used in the control planning section and the process by the external recognition system used in the safety evaluation section are processes that fuse the recognition results of multiple surrounding monitoring sensors. do. Further, a vehicle control device for achieving the above object is a vehicle control device used in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, and the vehicle control device is configured to recognize an external environment recognition system including the surroundings monitoring sensor and the recognition unit based on at least one of an output signal of at least one surroundings monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range, and a recognition result of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the surroundings monitoring sensor. and a vehicle stop processing unit (G6) that executes processing to stop the vehicle based on the abnormality of the external recognition system detected by the abnormality detection unit. The abnormality detection unit is configured to compare processing by the external recognition system used in the control planning unit with processing by an external recognition system used in the safety evaluation unit which is similar to the processing by the external recognition system used in the control planning unit, and determine that an abnormality has occurred in the external recognition system if the results differ. The processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processing to identify the vehicle's position on a map. A vehicle control device for achieving the above object is a vehicle control device used in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, and includes an abnormality detection unit (G) that detects an abnormality in an external recognition system including the surroundings monitoring sensor and the recognition unit based on at least one of an output signal of at least one surroundings monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range, and a recognition result of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the surroundings monitoring sensor. 4), and a vehicle stop processing unit (G6) that executes processing to stop the vehicle based on the abnormality detection unit detecting an abnormality in the external recognition system, wherein the abnormality detection unit is configured to compare processing by the external recognition system used in the control planning unit with processing by an external recognition system used in the safety evaluation unit which is similar to the processing by the external recognition system used in the control planning unit, and determine that an abnormality has occurred in the external recognition system when the results differ, and the processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processing for recognizing the environment around the vehicle by combining the recognition results of the perimeter monitoring sensors and map data.

[0009] Further, a vehicle control method for achieving the above object is a vehicle control method for creating a control plan for autonomously driving the vehicle, the method being used in a vehicle including a control planning unit (F3) for creating a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) for evaluating the safety of the control plan created by the control planning unit, the method being executed by at least one processor (21, 31), the method comprising: detecting an object present within a predetermined detection range; multiple The method includes an abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensors and the recognition units based on at least one of output signals of the periphery monitoring sensors (11, 11A to 11D, 17, 17A to 17D) and output signals of recognition units (F1, F2, H1, H2, X2) that recognize objects present around the vehicle based on the output signals of the periphery monitoring sensors, and a vehicle stop processing step (S211) of executing processing for stopping the vehicle based on the detection of an abnormality in the external recognition system in the abnormality detection step, By external recognition system Used for processing and control planning By external recognition system Used in the safety evaluation section as well as processing By external recognition system If there are differences, External recognition system Determines that an abnormality has occurred The process by the external recognition system used in the control planning section and the process by the external recognition system used in the safety evaluation section are processes that fuse the recognition results of multiple surrounding monitoring sensors. do. Further, a vehicle control method for achieving the above object is a vehicle control method for creating a control plan for autonomously driving the vehicle, which is used in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, and is executed by at least one processor (21, 31). The vehicle control method includes an abnormality detection step (S201) of detecting an abnormality in an external environment recognition system including the surrounding monitoring sensor and the recognition unit based on at least one of an output signal of at least one surrounding monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the surrounding monitoring sensor, and a stop processing step (S211) of executing processing for stopping the vehicle based on the detection of an abnormality in the external environment recognition system in the abnormality detection step. By external recognition system Used for processing and control planning By external recognition system Used in the safety evaluation section as well as processing By external recognition system If there are differences, External recognition system Determines that an abnormality has occurred The process by the external recognition system used in the control planning section and the process by the external recognition system used in the safety evaluation section are processes for identifying the vehicle position on a map. do. Further, a vehicle control method for achieving the above object is a vehicle control method for creating a control plan for autonomously driving the vehicle, which is used in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, and is executed by at least one processor (21, 31). The vehicle control method includes an abnormality detection step (S201) of detecting an abnormality in an external environment recognition system including the surrounding monitoring sensor and the recognition unit based on at least one of an output signal of at least one surrounding monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the surrounding monitoring sensor, and a stop processing step (S211) of executing processing for stopping the vehicle based on the detection of an abnormality in the external environment recognition system in the abnormality detection step. By external recognition system Used for processing and control planning By external recognition system Used in the safety evaluation section as well as processing By external recognition system If there are differences, External recognition system Determines that an abnormality has occurred The processing by the external recognition system used in the control planning section and the processing by the external recognition system used in the safety evaluation section are processes that recognize the environment around the vehicle by combining the recognition results of the surrounding monitoring sensors and map data. do. A vehicle control program for achieving the above object is a vehicle control program used in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, the vehicle control program including at least one processor (21, 31) that detects an object present within a predetermined detection range, multipleThe control system is configured to execute an abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensors and the recognition units based on at least one of output signals of the periphery monitoring sensors (11, 11A to 11D, 17, 17A to 17D) and output signals of recognition units (F1, F2, H1, H2, X2) that recognize objects present around the vehicle based on the output signals of the periphery monitoring sensors, and a vehicle stop processing step (S211) of executing processing for stopping the vehicle based on the detection of an abnormality in the external recognition system in the abnormality detection step, By external recognition system Used for processing and control planning By external recognition system Used in the safety evaluation section as well as processing By external recognition system If the results are different, External recognition system Determines that an abnormality has occurred The process by the external recognition system used in the control planning section and the process by the external recognition system used in the safety evaluation section are processes that fuse the recognition results of multiple surrounding monitoring sensors. do. A vehicle control program for achieving the above object is a vehicle control program used in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, and is configured to cause at least one processor (21, 31) to execute an abnormality detection step (S201) of detecting an abnormality in an external environment recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor, and a stop processing step (S211) of executing processing to stop the vehicle based on the detection of an abnormality in the external environment recognition system in the abnormality detection step. By external recognition system Used for processing and control planning By external recognition system Used in the safety evaluation section as well as processing By external recognition system If the results are different, External recognition system Determines that an abnormality has occurred The process by the external recognition system used in the control planning section and the process by the external recognition system used in the safety evaluation section are processes for identifying the vehicle position on a map. do. A vehicle control program for achieving the above object is a vehicle control program used in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates the safety of the control plan created by the control planning unit, and is configured to cause at least one processor (21, 31) to execute an abnormality detection step (S201) of detecting an abnormality in an external environment recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor, and a stop processing step (S211) of executing processing to stop the vehicle based on the detection of an abnormality in the external environment recognition system in the abnormality detection step. By external recognition system Used for processing and control planning By external recognition system Used in the safety evaluation section as well as processing By external recognition system If the results are different, External recognition system Determines that an abnormality has occurred The processing by the external recognition system used in the control planning section and the processing by the external recognition system used in the safety evaluation section are processes that recognize the environment around the vehicle by combining the recognition results of the surrounding monitoring sensors and map data. do. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining the overall configuration of an autonomous driving system. [Diagram 2] 1 is a block diagram showing an overall configuration of a vehicle control system 1. FIG. [Diagram 3] FIG. 2 is a block diagram for explaining the functions of an automatic driving device 20 and a diagnoser 30. [Figure 4] FIG. 13 is a diagram for explaining the operation of the danger level determination unit G5. [Diagram 5] FIG. 13 is a diagram for explaining the operation of the danger level determination unit G5. [Figure 6] FIG. 13 is a diagram for explaining the operation of the danger level determination unit G5. [Figure 7] 13 is a flowchart for explaining the operation of the abnormality detection unit G4. [Figure 8] 13 is a diagram for explaining the operation of the abnormality detection unit G4. FIG. [Figure 9] 13 is a diagram for explaining the operation of the abnormality detection unit G4. FIG. [Figure 10] 4 is a flowchart for explaining the operation of the diagnoser 30. [Figure 11] FIG. 2 is a diagram for explaining the operation of a recognition system. [Figure 12] FIG. 2 is a diagram for explaining the operation of a recognition system. [Figure 13] FIG. 13 is a diagram showing a modified example of the system configuration. [Figure 14] FIG. 11 is a block diagram showing the overall configuration of a vehicle control system 1 according to a second embodiment. [Figure 15] FIG. 4 is a block diagram for explaining the operation of a diagnostic device 30 of a second embodiment. [Figure 16] 13 is a diagram for explaining the operation of an abnormality detection unit G4 in the second embodiment. FIG. [Figure 17] 13 is a diagram for explaining the operation of an abnormality detection unit G4 in the second embodiment. FIG. [Figure 18] 13 is a diagram for explaining the operation of an abnormality detection unit G4 in the second embodiment. FIG. [Figure 19] FIG. 13 is a diagram showing a modified example of the configuration of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] A number of embodiments of a vehicle control system 1 to which a vehicle control device according to the present disclosure is applied will be described with reference to the drawings. In the following, an example will be described in which driving on the left side of the road is legalized. In an area where driving on the right side of the road is legalized, the left and right can be reversed. The present disclosure can be modified as appropriate to conform to the laws, regulations, and customs of the area in which the vehicle control system 1 is used.

[0013] [First embodiment] Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a schematic configuration of an automatic driving system according to the present disclosure. As shown in FIG. 1, the automatic driving system includes a vehicle control system 1 constructed in a vehicle Ma and an external server Sv. The vehicle control system 1 can be mounted on a vehicle capable of traveling on a road, and the vehicle Ma may be a four-wheeled vehicle, a two-wheeled vehicle, a three-wheeled vehicle, or the like. A motorized bicycle can also be included in the two-wheeled vehicle. The vehicle Ma may be a privately owned car, a shared car, or a service car. The service car includes a taxi, a route bus, a shared bus, and the like. The taxi or bus may be a robot taxi without a driver on board.

[0014] The vehicle control system 1 performs wireless communication with the external server Sv, and downloads partial map data, which is local high-precision map data, from the external server Sv to use for automatic driving and navigation. Hereinafter, the vehicle on which the vehicle control system 1 is mounted is also referred to as the vehicle itself Ma, and the occupant sitting in the driver's seat of the vehicle itself Ma (i.e., the driver's seat occupant) is also referred to as the user. The concept of the driver's seat occupant also includes an operator who has the authority to remotely operate the vehicle itself Ma. In the following description, the front-rear, left-right, and up-down directions are defined based on the vehicle itself Ma. Specifically, the front-rear direction corresponds to the longitudinal direction of the vehicle itself Ma. The left-right direction corresponds to the width direction of the vehicle itself Ma. The up-down direction corresponds to the vehicle height direction. From another perspective, the up-down direction corresponds to the direction perpendicular to a plane parallel to the front-rear and left-right directions.

[0015] <About map data> Here, the map data held by the external server Sv will be described first. The map data corresponds to map data that indicates the road structure and the position coordinates of features located along the road with an accuracy usable for autonomous driving. The map data includes node data, link data, feature data, etc. The node data is composed of each data item such as a node ID, which is a unique number assigned to each node on the map, node coordinates, node name, node type, and a connection link ID, which describes the link ID of the link connecting to the node.

[0016] The link data is data on links, which are road sections connecting nodes. The link data includes each data such as a link ID, which is a unique identifier for each link, shape information of the link (hereinafter, link shape), node coordinates of the start and end of the link, and road attributes. The link shape may be expressed by a coordinate sequence that represents the shape of the road edge. The link shape corresponds to the road shape. The link shape may be expressed by a cubic spline curve. The road attributes include, for example, the road name, road type, road width, lane number information that represents the number of lanes, and speed limit value. The link data may also include data indicating the road type, such as whether it is a motorway or a general road. A motorway here is a road on which pedestrians and bicycles are prohibited from entering, such as a toll road such as an expressway. The link data may include attribute information indicating whether or not the road is permitted for autonomous driving. The link data may be described by being subdivided into lane units.

[0017] The feature data includes demarcation line data and landmark data. The demarcation line data includes a demarcation line ID for each demarcation line and a coordinate point group representing the installation portion. The demarcation line data includes pattern information such as a broken line, a solid line, a road stud, etc. The demarcation line data is associated with lane information such as a lane ID and a link ID at the lane level. A landmark is a three-dimensional structure arranged along a road. Examples of three-dimensional structures arranged along a road include a guard rail, a curb, a tree, a utility pole, a road sign, and a traffic light. Road signs include guide signs such as a direction sign and a road name sign. The landmark data represents the position and type of each landmark. The shape and position of each feature are expressed by a coordinate point group. The POI data is data indicating the position and type of a feature that affects a vehicle's driving plan, such as a branch point for exiting a main line of a highway, a junction point, a speed limit change point, a lane change point, a congestion section, a construction section, an intersection, a tunnel, a toll gate, etc. The POI data includes type and position information.

[0018] The map data may be three-dimensional map data including a point cloud of road shapes and characteristic points of structures. The three-dimensional map data corresponds to map data expressing the positions of features such as road edges, lane markings, and road signs in three-dimensional coordinates. The three-dimensional map may be generated based on captured images using REM (Road Experience Management). The map data may also include a driving trajectory model. The driving trajectory model is trajectory data generated by statistically integrating the driving trajectories of multiple vehicles. The driving trajectory model is, for example, an average of the driving trajectories for each lane. The driving trajectory model corresponds to data indicating a driving trajectory that serves as a reference during autonomous driving.

[0019] Map data is managed by dividing it into a number of patches. Each patch corresponds to map data for a different area. The map data is stored in units of map tiles, which are obtained by dividing the map recording area into rectangular shapes of 2 km square, as shown in FIG. 1, for example. A map tile corresponds to a lower-level concept of the above-mentioned patch. Each map tile is assigned information indicating the area in the real world to which the map tile corresponds. The information indicating the area in the real world is expressed, for example, by latitude, longitude, and altitude. In addition, each map tile is assigned a unique ID (hereinafter, tile ID). The map data for each patch or map tile is a part of the entire map recording area, in other words, local map data. A map tile corresponds to partial map data. The external server Sv distributes partial map data according to the position of the vehicle control system 1 based on a request from the vehicle control system 1.

[0020] The shape of the map tile is not limited to a rectangular shape of 2 km square. It may be a rectangular shape of 1 km square or 4 km square. The map tiles may be hexagonal or circular. Each map tile may be set to partially overlap with an adjacent map tile. The map recording area may be the entire country where the vehicle is used, or only a part of the area. For example, the map recording area may be only an area where automatic driving of general vehicles is permitted or an area where automatic driving mobile services are provided. In addition, the division mode of the map data may be defined by the data size. In other words, the map recording area may be divided and managed within a range defined by the data size. In that case, each patch is set so that the data amount is less than a predetermined value. According to such an embodiment, the data size in one distribution can be set to a certain value or less. The map data is updated at any time by, for example, integrating probe data uploaded from multiple vehicles.

[0021] <Schematic configuration of vehicle control system 1> Next, the configuration of the vehicle control system 1 of the first embodiment will be described with reference to FIG. 2. The vehicle control system 1 shown in FIG. 2 is used in a vehicle capable of autonomous driving (hereinafter, an autonomous driving vehicle). As shown in FIG. 2, the vehicle control system 1 includes a periphery monitoring sensor 11, a vehicle state sensor 12, a V2X vehicle-mounted device 13, a map storage unit 14, a locator 15, a travel actuator 16, an autonomous driving device 20, and a diagnostic device 30. Note that HMI in the names of the components is an abbreviation for Human Machine Interface. V2X is an abbreviation for Vehicle to X (Everything), and refers to a communication technology that connects a vehicle to various things. The "V" in V2X refers to a car as the vehicle Ma, and the "X" can refer to various entities other than the vehicle Ma, such as pedestrians, other vehicles, road facilities, networks, and servers.

[0022] The level referred to by "automated driving" in the present disclosure may be, for example, equivalent to level 3 defined by the Society of Automotive Engineers (SAE International), or level 4 or higher. Hereinafter, an example will be described in which the vehicle Ma performs automatic driving at least at automation level 3 or higher. Level 3 refers to a level where the system performs all driving tasks within the operational design domain (ODD), while in an emergency, the system transfers operation authority to the user. The ODD specifies the conditions under which automatic driving can be performed, such as the driving position being within a highway. In level 3, the user is required to be able to respond quickly when the system requests a change of driving. The person who takes over the driving operation may be an operator outside the vehicle. Level 3 corresponds to so-called conditional automatic driving. Level 4 is a level where the system can perform all driving tasks except under specific circumstances such as incompatible roads and extreme environments. Level 5 is a level where the system can perform all driving tasks in any environment. Levels 3 to 5 can also be called autonomous driving levels where all control related to the driving of the vehicle is automatically performed.

[0023] The perimeter monitoring sensor 11 is a sensor that monitors the perimeter of the vehicle. The perimeter monitoring sensor 11 is configured to detect the presence and position of a predetermined detection object. The detection object includes, for example, a moving object such as a pedestrian or another vehicle. The other vehicles include a bicycle, a motorbike, and a motorcycle. The perimeter monitoring sensor 11 is also configured to be able to detect a predetermined feature. The features that the perimeter monitoring sensor 11 detects include road edges, road markings, and three-dimensional structures installed along the road. Road markings refer to paint that is painted on the road surface for traffic control and traffic regulation. For example, lane markings that indicate the boundaries of lanes, crosswalks, stop lines, guiding strips, safety zones, and regulatory arrows are included in the road markings. Lane markings are also called lane marks or lane markers. Lane markings include those that are realized by road studs such as chatter bars and Bott's dots. As described above, three-dimensional structures installed along the road include, for example, guardrails, road signs, and traffic lights. That is, it is preferable that the perimeter monitoring sensor 11 is configured to be able to detect landmarks. Also, curbs, guard rails, walls, etc., fall under the above-mentioned three-dimensional structures. The perimeter monitoring sensor 11 may also be configured to be able to detect objects that have fallen onto the road.

[0024] As the perimeter monitoring sensor 11, for example, a perimeter monitoring camera, a millimeter wave radar, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a sonar, etc. can be adopted. The perimeter monitoring camera is an in-vehicle camera arranged to capture the outside of the vehicle in a predetermined direction. The perimeter monitoring camera includes a front camera arranged on the upper end of the windshield on the interior side of the vehicle, the front grill, etc. to capture the front of the vehicle Ma. The millimeter wave radar is a device that detects the relative position and relative speed of an object with respect to the vehicle Ma by transmitting millimeter waves or quasi-millimeter waves in a predetermined direction and analyzing the received data of the reflected waves that are reflected by the object and returned. The millimeter wave radar generates, as observation data, for example, data indicating the reception strength and relative speed for each detection direction and distance, or data indicating the relative position and reception strength of the detected object. The LiDAR is a device that generates three-dimensional point cloud data indicating the position of a reflection point for each detection direction by irradiating a laser light. The LiDAR is also called a laser radar. The LiDAR may be a scan type or a flash type. A sonar is a device that transmits ultrasonic waves in a specified direction and detects the relative position and relative speed of an object in relation to the vehicle Ma by analyzing the received data of the reflected waves that are returned after the transmitted waves are reflected by the object.

[0025] For example, the vehicle control system 1 includes a front camera 11A, a millimeter wave radar 11B, a LiDAR 11C, and a sonar 11D as the periphery monitoring sensor 11, as shown in Fig. 3. The front camera 11A, the millimeter wave radar 11B, the LiDAR 11C, and the sonar 11D are all configured to include the traveling direction (for example, the front of the vehicle) in their detection ranges. Note that a plurality of sonars 11D may be arranged on the front bumper and rear bumper of the vehicle.

[0026] The forward camera 11A, LiDAR 11C, etc. detect the above-mentioned detection target using a classifier using, for example, a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network), etc. The millimeter wave radar 11B and sonar 11D detect the above-mentioned detection target by analyzing the intensity of the received reflected wave, the distribution of detection points, etc.

[0027] Each of the multiple perimeter monitoring sensors 11 outputs a signal indicating the relative position, type, moving speed, etc. of each detected object as a detection result. The identification result of the type of detected object includes correct answer probability value data indicating the likelihood of the identification result. For example, the correct answer probability value data includes a probability value that the detected object is a vehicle, a probability value that the detected object is a pedestrian, a probability value that the detected object is a cyclist, and a probability value that the detected object is a three-dimensional structure such as a signboard. The probability value for each type can be a score indicating the degree of match of the feature amount. Here, a cyclist refers to a bicycle with a rider on it, or a rider riding a bicycle. The detection result of the perimeter monitoring sensor 11 can also be interpreted as a recognition result or an identification result.

[0028] In addition, each periphery monitoring sensor 11 outputs an error signal to the in-vehicle network Nw when an internal failure or the like occurs. For example, the front camera 11A outputs an error signal when it detects an abnormality in the imaging element or processing circuit. The error signal output by the periphery monitoring sensor 11 is input to, for example, the automatic driving device 20 or the diagnostic device 30.

[0029] The type of sensor used by the vehicle control system 1 as the perimeter monitoring sensor 11 may be appropriately designed, and it is not necessary to include all of the above-mentioned sensors. Furthermore, the object recognition process based on the observation data generated by the perimeter monitoring sensor 11 may be executed by an ECU (Electronic Control Unit) outside the sensor, such as the automatic driving device 20. A part or all of the object recognition functions of the perimeter monitoring sensor 11, such as the forward camera 11A and the millimeter wave radar, may be provided in the automatic driving device 20. In that case, the various perimeter monitoring sensors 11 may provide the automatic driving device 20 with observation data, such as image data and distance measurement data, as detection result data.

[0030] The vehicle state sensor 12 is a group of sensors that detect state quantities related to the driving control of the host vehicle Ma. The vehicle state sensor 12 includes a vehicle speed sensor, a steering sensor, an acceleration sensor, a yaw rate sensor, and the like. The vehicle speed sensor detects the vehicle speed of the host vehicle. The steering sensor detects the steering angle of the host vehicle. The acceleration sensor detects acceleration such as the longitudinal acceleration and lateral acceleration of the host vehicle. The acceleration sensor may also detect deceleration, which is acceleration in the negative direction. The yaw rate sensor detects the angular velocity of the host vehicle. Note that the type of sensor used by the vehicle control system 1 as the vehicle state sensor 12 may be designed as appropriate, and it is not necessary to include all of the sensors described above.

[0031] The V2X vehicle-mounted device 13 is a device for the vehicle Ma to perform wireless communication with other devices. The V2X vehicle-mounted device 13 includes a wide-area communication unit and a narrow-area communication unit as communication modules. The wide-area communication unit is a communication module for performing wireless communication in accordance with a predetermined wide-area wireless communication standard. Various wide-area wireless communication standards, such as LTE (Long Term Evolution), 4G, and 5G, can be adopted here. The wide-area communication unit may be configured to perform wireless communication directly with other devices, in other words, without a base station, by a method in accordance with the wide-area wireless communication standard, in addition to communication via a wireless base station. In other words, the wide-area communication unit may be configured to perform cellular V2X. The vehicle Ma becomes a connected car that can connect to the Internet by being equipped with the V2X vehicle-mounted device 13. For example, the automatic driving device 20 downloads the latest partial map data corresponding to the current position of the vehicle Ma from the external server Sv in cooperation with the V2X vehicle-mounted device 13. In addition, the V2X vehicle-mounted device 13 acquires traffic congestion information and weather information from an external server Sv, a roadside device, etc. The traffic congestion information includes location information such as the start point and end point of a traffic congestion section.

[0032] The short-range communication unit included in the V2X vehicle-mounted device 13 is a communication module for directly performing wireless communication with other moving objects and roadside devices present around the vehicle Ma by a communication standard (hereinafter, short-range communication standard) in which the communication distance is limited to within several hundred meters. The other moving objects are not limited to vehicles, but may include pedestrians and bicycles. As the short-range communication standard, any standard may be adopted, such as the Wireless Access in Vehicular Environment (WAVE) standard disclosed in IEEE1709 or the Dedicated Short Range Communications (DSRC) standard. For example, the short-range communication unit broadcasts vehicle information about the vehicle Ma to surrounding vehicles at a predetermined transmission period, and receives vehicle information transmitted from other vehicles. The vehicle information includes a vehicle ID, a current position, a traveling direction, a moving speed, an operating state of a turn signal, a timestamp, and the like.

[0033] The map storage unit 14 is a device that stores partial map data corresponding to the current position of the vehicle Ma, which the V2X vehicle-mounted device 13 acquires from the external server Sv. The map storage unit 14 stores, for example, partial map data related to roads that the vehicle Ma is scheduled to pass through within a predetermined time. The map storage unit 14 is a non-transient storage medium. The map storage unit 14 may be realized, for example, by using a part of the storage area of ​​the storage 23 or the RAM 22. The map storage unit 14 may be built into the automatic driving device 20. The process of acquiring map data from the external server Sv in cooperation with the V2X vehicle-mounted device 13 and storing the data in the map storage unit 14 may be controlled by the automatic driving device 20 or the locator 15. The map storage unit 14 may be a non-volatile storage device that stores all map data.

[0034] The locator 15 is a device that generates highly accurate position information of the vehicle Ma by composite positioning that combines a plurality of pieces of information. The locator 15 is configured using, for example, a GNSS receiver. The GNSS receiver is a device that sequentially detects the current position of the GNSS receiver by receiving navigation signals (hereinafter, positioning signals) transmitted from positioning satellites that constitute the GNSS (Global Navigation Satellite System). For example, when the GNSS receiver is able to receive positioning signals from four or more positioning satellites, it outputs positioning results every 100 milliseconds. As the GNSS, GPS, GLONASS, Galileo, IRNSS, QZSS, Beidou, etc. can be adopted.

[0035] The locator 15 sequentially measures the position of the vehicle Ma by combining the positioning result of the GNSS receiver and the output of the inertial sensor. For example, when the GNSS receiver cannot receive the GNSS signal, such as in a tunnel, the locator 15 performs dead reckoning (i.e., autonomous navigation) using the yaw rate and the vehicle speed. The yaw rate used for dead reckoning may be calculated by a front camera using SfM technology, or may be detected by a yaw rate sensor. The locator 15 may perform dead reckoning using the output of an acceleration sensor or a gyro sensor. The vehicle position is expressed by three-dimensional coordinates of, for example, latitude, longitude, and altitude. The measured vehicle position information is output to the in-vehicle network Nw and used by the automatic driving device 20, etc.

[0036] The locator 15 may be configured to perform a localization process. The localization process refers to a process of identifying the detailed position of the vehicle Ma by comparing the coordinates of a landmark identified based on an image captured by an on-board camera such as the forward camera 11A with the coordinates of a landmark registered in map data. The locator 15 may also be configured to identify a driving lane ID, which is an identifier of the lane in which the vehicle Ma is traveling, based on the distance from the road edge detected by the forward camera or the millimeter wave radar. The driving lane ID indicates, for example, which lane the vehicle Ma is traveling in from the left or right edge of the road. Some or all of the functions of the locator 15 may be provided by the automatic driving device 20.

[0037] The traveling actuators 16 are actuators for traveling. The traveling actuators 16 include, for example, a brake actuator as a braking device, an electronic throttle, a steering actuator, and the like. The steering actuator also includes an EPS (Electric Power Steering) motor. The traveling actuators 16 are controlled by an automatic driving device 20. Note that a steering ECU that performs steering control, a power unit control ECU that performs acceleration / deceleration control, a brake ECU, and the like may be interposed between the automatic driving device 20 and the traveling actuators.

[0038] The automatic driving device 20 is an ECU (Electronic Control Unit) that controls the driving actuator 16 based on the detection results of the surrounding monitoring sensor 11 and the like, thereby performing part or all of the driving operation on behalf of the driver. Here, as an example, the automatic driving device 20 is configured to be able to execute up to automation level 5, and is configured to be able to switch between operation modes corresponding to each automation level. For convenience, the operation mode corresponding to the automatic driving level N (N=0 to 5) will be referred to as the level N mode hereinafter. The following description will be continued assuming that the automatic driving device 20 is operating in an automation level 3 or higher mode. In a driving mode of level 3 or higher, the automatic driving device 20 automatically performs vehicle steering, acceleration, deceleration (in other words, braking), etc., so that the host vehicle Ma travels along the road to the destination set by the driver or operator. The switching of the operation mode is automatically performed due to the user's operation, system limits, exiting the ODD, etc.

[0039] The automatic driving device 20 is mainly composed of a computer including a processing unit 21, a RAM 22, a storage 23, a communication interface 24, and a bus connecting these. The processing unit 21 is hardware for arithmetic processing coupled to the RAM 22. The processing unit 21 includes at least one arithmetic core such as a CPU (Central Processing Unit). The processing unit 21 accesses the RAM 22 to execute various processes for realizing the functions of each functional unit described later. The storage 23 includes a non-volatile storage medium such as a flash memory. The storage 23 stores a program executed by the processing unit 21 (hereinafter, an automatic driving program). The processing unit 21 executing the automatic driving program corresponds to the execution of a method corresponding to the automatic driving program as a vehicle control method. The communication interface 24 is a circuit for communicating with other devices via the in-vehicle network Nw. The communication interface 24 may be realized using analog circuit elements, ICs, or the like.

[0040] The communication interface 24 corresponds to a configuration for acquiring, for example, the detection result (i.e., sensing information) as an output signal of the periphery monitoring sensor 11 and the detection result of the vehicle state sensor 12. The sensing information includes the positions and movement speeds of other moving bodies, features, obstacles, etc. present around the host vehicle Ma. For example, the information includes the distance between the host vehicle Ma and a forward vehicle that is a vehicle traveling in front of the host vehicle Ma, and the movement speed of the forward vehicle. The forward vehicle here can include a so-called preceding vehicle that travels in the same lane as the host vehicle, as well as a vehicle traveling in an adjacent lane. In other words, the forward here can include not only the direction directly in front of the host vehicle Ma, but also the diagonally forward direction. The sensing information includes the lateral distance to the road edge, the travel lane ID, the offset amount from the center line of the travel lane, etc.

[0041] The autonomous driving device 20 receives inputs of various signals indicating the traveling speed, acceleration, yaw rate, etc. of the host vehicle Ma from the vehicle state sensor 12, host vehicle position information detected by the locator 15, and map data stored in the map storage unit 14. In addition, traffic information acquired by the V2X vehicle-mounted device 13 from a roadside device or the like, and other vehicle information acquired by vehicle-to-vehicle communication are also input. Details of the operation of the autonomous driving device 20 based on these input signals will be described separately later.

[0042] The diagnoser 30 is configured to detect an abnormality in the external environment recognition system based on the output signal from the periphery monitoring sensor 11 and the output signal from the automatic driving device 20. The external environment recognition system here includes the individual periphery monitoring sensors 11. The external environment recognition system also includes a module, such as a fusion unit F2 described later, that performs processing to recognize the environment around the vehicle in the automatic driving device 20 based on the output signal from the periphery monitoring sensor 11. Recognizing the driving environment here also includes recognizing the type and position of objects present around the vehicle.

[0043] The diagnoser 30 is mainly composed of a computer including a processing unit 31, a RAM 32, a storage 33, a communication interface 34, and a bus connecting these components. A program (hereinafter, a vehicle control program) executed by the processing unit 31 is stored in the storage 33. The execution of the vehicle control program by the processing unit 31 corresponds to the execution of a method corresponding to the vehicle control program as a vehicle control method.

[0044] The diagnoser 30 receives as input data indicating the detection results output by the various surrounding monitoring sensors 11, error signals, and the operating status of the fusion unit F2 included in the automatic driving device 20. Data indicating the operating status of the fusion unit F2 and the like includes, for example, the results of fusion and the utilization rate (in other words, weight) of the surrounding monitoring sensors 11 used in sensor fusion. The diagnoser 30 also outputs a signal indicating the diagnosis result to the automatic driving device 20. The signal indicating the diagnosis result includes whether the various surrounding monitoring sensors 11 are available or not. Furthermore, if there is an abnormality in the recognition system, a control signal according to the degree of influence of the abnormal event is output to the automatic driving device 20. Details of the diagnoser 30 will be described separately below. The diagnoser 30 corresponds to a vehicle control device.

[0045] <Configuration of the automatic driving device 20 and the diagnostic device 30> Here, the configurations of the autonomous driving device 20 and the diagnoser 30 will be described with reference to Fig. 3. The autonomous driving device 20 includes a map linking unit F1, a fusion unit F2, a control planning unit F3, and a safety evaluation unit F4 as functional units realized by the processing unit 21 executing the autonomous driving program. The diagnoser 30 includes a diagnostic material acquisition unit G1, a recognition result storage unit G2, a correspondence relationship specification unit G3, an abnormality detection unit G4, a risk determination unit G5, and a risk response unit G6 as functional units realized by the processing unit 31 executing the vehicle control program.

[0046] The map linking unit F1 recognizes the environment around the vehicle by combining the recognition result of the front camera 11A with the map data. For example, the map linking unit F1 corrects the positions and types of structures and road markings in front of the vehicle. The map data is used to improve the reliability and amount of information of the recognition result by the front camera 11A and to correct it. The map linking unit F1 may also perform a so-called localization process to identify the vehicle's position on the map by comparing landmark structures shown in the map data with landmark information recognized by the front camera 11A.

[0047] The map linking unit F1 outputs the recognition result data determined by using the recognition result of the front camera 11A and the map data in a complementary manner to the diagnoser 30 and the fusion unit F2. For convenience, the map linking unit F1 is also referred to as the primary recognition unit because it corresponds to a configuration that performs pre-processing of the fusion unit F2, and the recognition result of the map linking unit F1 is also referred to as the primary recognition result. The object recognition result data as the primary recognition result includes information indicating the position and type of each detected object. The type information of the detected object includes correct answer probability value data for each type, as described above. For example, the correct answer probability value data includes a probability value that the detected object is a vehicle, a probability value that the detected object is a pedestrian, etc.

[0048] The map linking unit F1 also outputs sensor usage ratio data for each detected object to the diagnoser 30 and the fusion unit F2. The sensor usage ratio data for a certain object indicates the weight of the recognition result of the front camera 11A and the map data used to determine the type of the object, in other words, the weight. The sensor usage ratio may be a constant value or may be variable depending on the distance from the vehicle to the detected object. Furthermore, it may be changed depending on the type of object recognized by the front camera 11A and its correct answer probability value. In addition, the sensor usage ratio may be changed depending on the driving scene. For example, when the image recognition performance may deteriorate, such as at night or in rainy weather, the weight for the recognition result by the front camera 11A is set relatively small. In addition to the above, driving scenes in which the image recognition performance may deteriorate may include a case where the vehicle is exposed to the setting sun, near the entrance or exit of a tunnel, while driving around a curve, and when fog is generated.

[0049] In this embodiment, the map linking unit F1 is disposed outside the fusion unit F2, but the function of the map linking unit F1 may be provided in the fusion unit F2. The output signals of the front camera 11A and the locator 15 and the map data may be directly input to the fusion unit F2. The map linking unit F1 is an optional element. The map linking unit F1 and the fusion unit F2 correspond to a recognition unit.

[0050] The fusion unit F2 recognizes the driving environment of the host vehicle Ma by a sensor fusion process that integrates the detection results of various surrounding monitoring sensors 11 with a predetermined weight (in other words, a sensor usage ratio). The driving environment includes the positions, types, and moving speeds of objects around the vehicle, as well as the curvature of the road, the number of lanes, the driving lane as the driving position within the road, the weather, and the road surface condition. The weather and road surface condition can be identified by combining the recognition result of the front camera 11A and the weather information acquired by the V2X vehicle-mounted device 13. The road structure is identified mainly based on the information input from the map linking unit F1.

[0051] The data showing the recognition result of the fusion unit F2 is also referred to as fusion result data to distinguish it from the recognition result of the map linking unit F1. For example, the fusion unit F2 generates fusion result data by integrating the detection results of the map linking unit F1, the millimeter wave radar 11B, the LiDAR 11C, and the sonar 11D at a predetermined or dynamically set sensor usage ratio. The fusion result data is output to the diagnoser 30, the control planning unit F3, and the safety evaluation unit F4.

[0052] The fusion result data also includes information indicating the position and type of each detected object. For example, the fusion result data includes the type, position, shape, size, relative speed, etc. of each detected object. As described above, the type information includes correct answer probability value data for each type. Note that the fusion unit F2 can be called a secondary recognition unit because it is configured to perform object recognition processing again using the recognition processing result of the map linking unit F1 as the primary recognition unit. Therefore, the recognition result in the fusion unit F2 is also referred to as a secondary recognition result. The recognition result data in the fusion unit F2 is also referred to as a secondary recognition result. The secondary recognition result data in the following can be called fusion result data.

[0053] The fusion unit F2 also outputs sensor usage ratio data for each detected object to the diagnoser 30, the control planning unit F3, and the safety evaluation unit F4. As described above, the sensor usage ratio data for a certain object indicates the weight of each information source used to determine the type of the object. The information sources here correspond to the perimeter monitoring sensors 11 such as the millimeter wave radar 11B and the LiDAR 11C. The map linking unit F1 also corresponds to an information source for the fusion unit F2. The map linking unit F1 is also configured to recognize objects present around the vehicle based on the recognition results of the front camera 11A. Therefore, the map linking unit F1 can be included in the concept of the perimeter monitoring sensor 11.

[0054] The sensor usage ratio in the fusion unit F2 may be a constant value, or may be variable depending on the distance from the vehicle to the detected object. The sensor usage ratio in the fusion unit F2 may be set to a value depending on the correct answer probability output by each perimeter monitoring sensor 11. For example, the weight of a perimeter monitoring sensor 11 with a correct answer probability of 80% may be 0.8 (equivalent to 80%). The usage ratio of the perimeter monitoring sensor 11 for a certain object may be a value obtained by multiplying the correct answer probability of the perimeter monitoring sensor 11 by a predetermined conversion coefficient. It may also be changed depending on the type of the object. Furthermore, the sensor usage ratio may be changed depending on the driving scene. In addition, the sensor usage ratio may be changed for each detection target item of the object. The detection target items include distance, horizontal azimuth, height azimuth, moving speed, size, type, etc. For example, with respect to distance, height azimuth, and moving speed, the weight of the millimeter wave radar 11B and the LiDAR 11C may be larger than that of the map linking unit F1 based on the detection result of the front camera 11A. In addition, in terms of type and horizontal azimuth angle, the weighting of the LiDAR 11C and the map linking unit F1 may be greater than that of the millimeter wave radar 11B.

[0055] The control planning unit F3 uses the map data and the recognition result of the fusion unit F2 to generate a driving plan for autonomously driving the vehicle Ma by automatic driving, in other words, a control plan. For example, the control planning unit F3 performs a route search process as a medium- to long-term driving plan to generate a recommended route for moving from the vehicle's position to a destination. In addition, the control planning unit F3 generates a driving plan for lane change, a driving plan for driving in the center of the lane, a driving plan for following a preceding vehicle, and a driving plan for obstacle avoidance as a short-term control plan for driving according to the medium- to long-term driving plan. Note that the map data is used, for example, to identify an area in which the vehicle can travel based on the number of lanes and the road width, and to set a steering amount and a target speed based on the curvature of the road ahead.

[0056] The control planning unit F3 generates, as a short-term control plan, for example, a route that is a certain distance or a center from the recognized lane marking as a driving plan, or generates a route that follows the behavior or driving trajectory of the recognized preceding vehicle as a driving plan. When the driving path of the host vehicle corresponds to a road with multiple lanes on one side, the control planning unit F3 may generate a plan candidate for changing lanes to an adjacent lane in the same direction as the driving lane of the host vehicle. When the fusion unit F2 recognizes that there is an obstacle in front of the host vehicle Ma, the control planning unit F3 may generate a driving plan for passing the side of the obstacle. When the fusion unit F2 recognizes that there is an obstacle in front of the host vehicle Ma, the control planning unit F3 may generate a driving plan for decelerating to stop in front of the obstacle. The control planning unit F3 may be configured to generate a driving plan that is determined to be optimal by machine learning or the like.

[0057] The control planning unit F3 calculates, for example, one or more plan candidates as candidates for a short-term driving plan. The multiple plan candidates each have different acceleration / deceleration amounts, jerk, steering amount, and timing for performing various controls. In other words, the short-term driving plan may include schedule information for acceleration / deceleration for speed adjustment on the calculated route. The plan candidates may also be called route candidates. The control planning unit F3 outputs data indicating at least one of the generated plan candidates to the safety evaluation unit F4.

[0058] The safety evaluation unit F4 is configured to determine a final execution plan from among the control plans generated by the control planning unit F3 based on the recognition result in the fusion unit F2 and the map data, and to output a control signal corresponding to the plan to the traveling actuator 16. The map data is used for the calculation process of the safety distance based on the road structure and traffic rules, and the calculation process of the potential accident responsibility value. The safety evaluation unit F4 includes a responsibility value calculation unit F41 and an action decision unit F42 as sub-functions.

[0059] The responsibility value calculation unit F41 corresponds to a configuration for evaluating the safety of the driving plan generated by the control planning unit F3. As an example, the responsibility value calculation unit F41 evaluates the safety based on whether or not the distance between the vehicle and the surrounding objects (hereinafter, the object distance) is equal to or greater than a safe distance determined using a mathematical formula model that mathematically represents the concept of safe driving. For example, when the vehicle Ma travels along each plan candidate planned by the control planning unit F3, the responsibility value calculation unit F41 determines a potential accident responsibility value that indicates the degree of responsibility of the vehicle Ma when the vehicle Ma travels along the plan candidate and an accident occurs in the vehicle Ma. The potential accident responsibility value is determined using a comparison result between the vehicle distance between the vehicle Ma and the surrounding vehicles when the vehicle Ma travels along the plan candidate and the safe distance as one of the factors.

[0060] The lower the responsibility, the smaller the potential accident responsibility value. Therefore, the safer the vehicle Ma is driving, the smaller the potential accident responsibility value. For example, if a sufficient distance is maintained between the vehicles, the potential accident responsibility value will be small. In addition, the potential accident responsibility value can be large when the vehicle Ma accelerates or decelerates suddenly.

[0061] In addition, the responsibility value calculation unit F41 can set the potential accident responsibility value to a low value when the host vehicle Ma is traveling in accordance with the traffic rules. In other words, whether or not the route at the host vehicle position complies with the traffic rules can also be adopted as a factor that affects the potential accident responsibility value. In order to determine whether or not the host vehicle Ma is traveling in accordance with the traffic rules, the responsibility value calculation unit F41 can be configured to acquire the traffic rules at the location where the host vehicle Ma is traveling. The traffic rules at the location where the host vehicle Ma is traveling may be acquired from a predetermined database, or the traffic rules at the current location may be acquired by analyzing an image captured by a camera that captures the surroundings of the host vehicle Ma to detect signs, traffic lights, road markings, etc. The traffic rules may be included in the map data.

[0062] The safety distance used by the responsibility value calculation unit F41 is a parameter that is a standard for evaluating the safety between an object, such as the vehicle itself and a preceding vehicle, and is dynamically determined according to the driving environment. The safety distance is set based on at least information on the behavior of the vehicle itself, such as the acceleration. Since various models can be adopted as a method for calculating the safety distance, detailed explanation of the calculation method will be omitted here. Note that, for example, the RSS (Responsibility Sensitive Safety) model can be used as a mathematical formula model for calculating the safety distance. In addition, the SFF (Safety Force Field, registered trademark) can also be adopted as a mathematical formula model for calculating the safety distance. The safety distance includes a safety distance between the vehicle itself and a preceding vehicle, that is, a safety distance in the vertical direction, and a safety distance in the left-right direction, that is, a lateral direction. The above-mentioned mathematical formula model includes a model for determining these two types of safety distance.

[0063] The above mathematical formula model does not guarantee that an accident will not occur at all, but is intended to guarantee that the vehicle will not be held responsible for the accident if appropriate action is taken to avoid a collision when the safe distance is not reached. An example of the appropriate action to avoid a collision referred to here is braking with a reasonable force. Braking with a reasonable force is, for example, braking with the maximum deceleration possible for the vehicle. The safe distance calculated by the mathematical formula model can be rephrased as the minimum distance that the vehicle should keep between itself and an obstacle in order to avoid the vehicle coming too close to the obstacle.

[0064] The action decision unit F42 is configured to determine a final execution plan from among a plurality of control plans based on the potential accident responsibility value calculated by the responsibility value calculation unit F41. For example, the action decision unit F42 adopts, as the final execution plan, a plan from among the control plans generated by the control planning unit F3 that has the smallest potential accident responsibility value calculated by the responsibility value calculation unit F41 (described later) or a plan whose potential accident responsibility value is at an acceptable level.

[0065] The safety evaluation unit F4 outputs a control signal corresponding to the control plan determined by the action determination unit F42 to the traveling actuator 16 to be controlled. For example, when deceleration is planned, the safety evaluation unit F4 outputs a control signal for realizing the planned deceleration to the brake actuator or the electronic throttle.

[0066] The diagnostic material acquisition unit G1 is configured to acquire information as diagnostic material for determining whether or not an abnormality has occurred in any of the external recognition systems, i.e., the perimeter monitoring sensor 11, the map linking unit F1, and the fusion unit F2. As diagnostic material, signals indicating the detection results of each perimeter monitoring sensor 11, the recognition result data of the map linking unit F1, and the recognition result data of the fusion unit F2 can be adopted. In addition, when an error signal is output from the perimeter monitoring sensor 11, the error signal can also be adopted as diagnostic material. Note that determining whether or not an abnormality has occurred corresponds to detecting that an abnormality has occurred. An abnormality here refers to a state in which the system is not operating normally due to some kind of malfunction. Abnormal states include a state in which the recognition result is not output due to a malfunction, and a state in which the output signal is fixed. The diagnostic material acquisition unit G1 sequentially acquires a predetermined type of data that can be used as diagnostic material from each of the surrounding monitoring sensor 11, the map linkage unit F1, and the fusion unit F2. The diagnostic material acquisition unit G1 stores the data acquired from each component in the recognition result storage unit G2, classifying them by information type. In addition, data of the same type acquired at different times is sorted and stored in chronological order, for example, with the most recent data at the top. The various data are configured to be able to identify the order of acquisition, for example, by adding a timestamp corresponding to the acquisition time. Note that data that has been saved for a certain period of time may be discarded in sequence. The diagnostic material acquisition unit G1 corresponds to the recognition result acquisition unit.

[0067] The recognition result storage unit G2 is realized, for example, by using a part of the storage area of ​​the RAM 32. The recognition result storage unit G2 is configured to store data acquired within the last 10 seconds, for example. The recognition result storage unit G2 corresponds to a memory that temporarily pools data as diagnostic material.

[0068] The correspondence specification unit G3 performs correspondence between an object detected at a previous time and an object detected at a next time based on the position and moving speed of the detected object. That is, the correspondence specification unit G3 corresponds to a configuration that tracks (in other words, tracks) an object once detected. Various methods can be used to track an object. For example, the correspondence specification unit G3 can estimate the position at the current time for each detected object based on the position and moving speed at the previous time, and can regard an object closest to the estimated position in the current observation data as the same object. In addition, the correspondence specification unit G3 may be configured to perform correspondence (in other words, tracking) to determine whether the objects are the same or not using the similarity of the feature amount of the detected object. As the feature amount of the detected object, for example, a color histogram, size, reflection intensity, or other items according to the characteristics of each perimeter monitoring sensor 11 can be adopted.

[0069] The correspondence specification unit G3 also performs correspondence between objects commonly detected by different perimeter monitoring sensors 11 based on the position information of the detected objects. For example, when the millimeter wave radar 11B detects another vehicle within a predetermined distance from the position where the front camera 11A detects another vehicle, the two are considered to be the same object. It is preferable that the same detected object ID is assigned to the same object to facilitate tracking over time. The detected object ID is an identifier for identifying the detected object. The result of the specification by the correspondence specification unit G3 is reflected in the recognition result data stored in the recognition result storage unit G2.

[0070] The abnormality detection unit G4 is configured to detect an abnormality in the external recognition system based on the recognition result data stored in the recognition result storage unit G2. Details of the abnormality detection unit G4 will be described separately. The surrounding monitoring sensor 11 that the abnormality detection unit G4 determines is not operating normally will be referred to as an abnormal sensor hereinafter. From another perspective, the abnormality detection unit G4 corresponds to a configuration that determines the presence or absence of an abnormal sensor.

[0071] The risk level determination unit G5 is configured to determine the degree of risk based on the presence or absence of an abnormal sensor and the degree of use of the abnormal sensor in the sensor fusion process. Here, as an example, the risk level is expressed in three stages from 0 to 2. Risk level 0 corresponds to a non-hazardous state. Risk level 1 corresponds to a state in which it is better to execute control on the safer side than usual. Control on the safer side refers to processing such as extending the safe distance and transferring operating authority to the user. Risk level 2 refers to a state in which it is better to stop the vehicle.

[0072] For example, as shown in FIG. 4, when there is no abnormal sensor, the risk level is determined to be 0. Also, as shown in FIG. 5, even when there is an abnormal sensor, when the usage level of the abnormal sensor is less than a predetermined risk determination threshold, the risk level is determined to be 1. As shown in FIG. 6, when there is an abnormal sensor and the usage level of the abnormal sensor is equal to or greater than the risk determination threshold, the risk level is determined to be 2. Note that sensor A shown in FIGS. 4 to 6 corresponds to, for example, the front camera 11A or the map linking unit F1. Sensor B is, for example, a millimeter wave radar 11B, and sensor C is, for example, a LiDAR 11C. Sensor D is, for example, a sonar 11D. The risk determination threshold can be, for example, 10% or 25%.

[0073] The usage degree of each perimeter monitoring sensor 11 may be, for example, the weight in sensor fusion itself. In addition, when the weight coefficient is different for each object, the usage degree of each perimeter monitoring sensor 11 may be the average value or maximum value of the weight for each object. Furthermore, the usage degree of the perimeter monitoring sensor 11 may be the ratio of the number of times the recognition result of the perimeter monitoring sensor 11 to be calculated is used in the sensor fusion process within a certain time. That is, the usage degree of each perimeter monitoring sensor 11 may be a time-dependent one. Furthermore, the usage degree of each perimeter monitoring sensor 11 in sensor fusion may be the ratio of the number of objects whose detection result of the target perimeter monitoring sensor 11 is used in sensor fusion to the total number of objects detected within a predetermined distance from the vehicle. The perimeter monitoring sensor 11 used in sensor fusion may be, for example, a sensor whose weight is 25% or more of the total. Note that if the weight of the perimeter monitoring sensor 11 to be calculated is not 0, the sensor may be considered to be used in sensor fusion and the usage degree may be calculated. Of course, the usage degree may be calculated taking the weight into consideration.

[0074] In addition, when the risk determination unit G5 determines in an individual diagnosis process described later that the fusion unit F2 is not operating normally, it determines the risk level to be 2. As another aspect, when an abnormality occurs in the fusion unit F2, the risk level may be 1. In addition, when it is determined that the map linkage unit F1 is not operating normally, it may determine the risk level to be 1.

[0075] The risk response unit G6 is configured to request the safety evaluation unit F4 to execute vehicle control according to the risk level based on the determination that the risk level is not 0. The risk response unit G6 requests the safety evaluation unit F4 to execute a predetermined safety action based on the determination that the risk level is 1. The safety action can be, for example, vehicle control to stop the vehicle gradually over about 10 seconds to 1 minute. The safety action may also be a handover request process. The handover request process corresponds to a request to take over driving operations to the driver's seat occupant or operator in conjunction with an HMI (Human Machine Interface) system such as a display. The handover request process can be called a handover request. The safety action may be to extend the safety distance used in the calculation of the potential accident liability value. The safety action can be, for example, a process to reduce the vehicle's running speed by a predetermined amount from the scheduled target speed. The safety action corresponds to an action to shift the vehicle behavior to the safer side than usual while maintaining running.

[0076] The risk level is 1 when an abnormality occurs in the perimeter monitoring sensor 11 whose rate of use in sensor fusion is equal to or higher than a predetermined value. In other words, the above configuration corresponds to a configuration in which a process for implementing a safety behavior is performed based on the occurrence of an abnormality in the perimeter monitoring sensor 11 whose rate of use in sensor fusion is less than a predetermined threshold value.

[0077] Furthermore, the risk response unit G6 requests the safety evaluation unit F4 to take a predetermined emergency action based on the risk level being determined to be 2. The emergency action can be, for example, MRM. The specific content of MRM can be, for example, a process of autonomously driving the vehicle to a safe place while issuing an alarm to the surroundings and parking it. The safe place can be a shoulder having a width of a predetermined value or more, or a place specified as an emergency evacuation area. The content of MRM may be a process of stopping the vehicle in the lane in which the vehicle is currently traveling by slow deceleration. As the deceleration at that time, it is preferable to adopt a value of 4 [m / s^2] or less, such as 2 [m / s^2] or 3 [m / s^2]. Of course, when it is necessary to avoid a collision with a preceding vehicle, a deceleration exceeding 4 [m / s^2] can also be adopted. The deceleration during MRM may be dynamically determined and sequentially updated in consideration of the running speed at the time of starting MRM and the distance between the vehicle and the following vehicle, within a range in which the vehicle can be stopped within 10 seconds, for example. Initiating MRM is equivalent to initiating deceleration for an emergency stop.

[0078] The above configuration is a configuration that executes a process to make an emergency stop of the vehicle based on the determination that the risk level is 2. For example, the risk response unit G6 outputs a signal to the safety evaluation unit F4 requesting execution of MRM. The risk response unit G6 may instruct the control planning unit F3 to generate a plan for implementing MRM as an emergency action. The signal requesting execution of the emergency action may be output to the control planning unit F3. The risk response unit G6 corresponds to a stop processing unit that executes a process to stop the host vehicle Ma.

[0079] <Details of operation of abnormality detection unit G4> Here, the individual diagnostic process, which is a process for determining whether each of the multiple periphery monitoring sensors 11 is operating normally or in an abnormal state, will be described with reference to the flowchart shown in FIG. 7. The sensor diagnostic process is executed sequentially at a predetermined execution cycle, for example, while the vehicle's running power source is on or while the automatic driving function is enabled. The execution cycle can be set to, for example, 200 milliseconds. The running power source here is a power source for running the vehicle, and refers to an ignition power source if the vehicle is a gasoline vehicle. Also, refers to a system main relay if the vehicle is an electric vehicle or a hybrid car. The sensor diagnostic process of this embodiment includes, as an example, steps S101 to S105. The individual diagnostic process is executed individually for each periphery monitoring sensor 11. Hereinafter, the periphery monitoring sensor 11 that is the target of the diagnostic process is also referred to as a diagnostic target sensor.

[0080] In step S101, the abnormality detection unit G4 sets any object detected by the diagnosis target sensor as the target object to be used in subsequent processing, and proceeds to step S102. In step S102, the abnormality detection unit G4 accesses the recognition result storage unit G2 and acquires the history of the recognition results of the diagnosis target sensor for the target object within a certain period of time (in other words, time series data). The recognition result time series data indicates the transition of the correct answer probability value data for each type of the target object. For example, the recognition result time series data indicates the transition of the probability that the target object is a vehicle or the transition of the probability that the target object is a pedestrian. The sampling period that specifies the reference range of past recognition results can be, for example, 4 seconds or 6 seconds.

[0081] In step S103, the abnormality detection unit G4 judges whether the recognition result for the target object is stable based on the time series data of the recognition result acquired in step S102. For example, as shown in FIG. 8, if the recognition result for the target object is stable (step S103 YES), the flow proceeds to step S104. In step S104, it is judged that the diagnosis target sensor is likely to be operating normally, and this flow ends. On the other hand, as shown in FIG. 9, if the recognition result for the target object fluctuates greatly, that is, if it is unstable (step S103 NO), the flow proceeds to step S105, and it is judged that there is a possibility that an abnormality has occurred in the diagnosis target sensor. In FIG. 8 and FIG. 9, the vertical axis indicates the probability value, and the horizontal axis indicates the time. The dashed line indicates the transition of the probability value p1 that the type of the target object is a vehicle, for example, and the dashed double-dot line indicates the transition of the probability value p2 that the type of the target object is a pedestrian, for example. The higher the amplitude of p1 and p2, or the higher the frequency of up-down switching, the more unstable the recognition result is. The amplitude of the probability value for each type and the threshold value for the frequency of switching for determining the possibility of abnormal operation may be appropriately designed.

[0082] The abnormality detection unit G4 performs the above process for all objects detected by the sensor to be diagnosed, and ultimately determines whether the sensor to be diagnosed is operating normally based on the number of abnormality determinations, which is the number of times it has been determined that there is a possibility of an abnormality occurring. For example, if the number of abnormality determinations is equal to or greater than a predetermined threshold, it is determined that there is an abnormality in the sensor to be diagnosed. The abnormality detection unit G4 may determine that there is an abnormality in the sensor to be diagnosed when the ratio of the number of abnormality determinations to the total number of determinations, which is the sum of the number of normal determinations, which is the number of times it has been determined that there is a high possibility of the sensor operating normally, and the number of abnormality determinations, is equal to or greater than a predetermined threshold. The abnormality detection unit G4 may also be configured to perform the above diagnostic process for multiple objects selected randomly or based on a predetermined rule, rather than for all objects detected by the sensor to be diagnosed. Furthermore, the abnormality detection unit G4 may perform the above diagnostic process for only one object selected randomly or based on a predetermined rule, and determine whether the sensor to be diagnosed is operating normally. However, if there is only one object to be used as a diagnostic material, there is a risk that the perimeter monitoring sensor 11, which is operating normally, may be erroneously determined to be not operating normally due to the characteristics of the object. For this reason, it is preferable that the abnormality detection unit G4 performs statistical processing on the diagnosis results using a plurality of objects to determine whether the sensor to be diagnosed is normal or not.

[0083] The abnormality detection unit G4 may determine that an abnormality has occurred in the periphery monitoring sensor 11 that is outputting the error signal. In other words, the periphery monitoring sensor 11 that is outputting the error signal can be regarded as an abnormal sensor.

[0084] The abnormality detection unit G4 also performs processing similar to the individual diagnosis processing shown in Fig. 7 on the map linking unit F1 and the fusion unit F2. This determines whether or not the map linking unit F1 and the fusion unit F2, which are targets of diagnosis, are operating normally from the viewpoint of the stability of the recognition result. The criteria for determining that they are operating abnormally can be the same as those for the perimeter monitoring sensor 11. Hereinafter, each of the individual elements constituting the external world recognition system, such as the perimeter monitoring sensor 11, the map linking unit F1, and the fusion unit F2, will be referred to as a recognition module.

[0085] <Operation of the danger level determination unit G5> Here, the operation of the diagnoser 30 based on the operating state of each of the periphery monitoring sensors 11 will be described with reference to the flowchart shown in Fig. 10. The flowchart shown in Fig. 10 is executed sequentially at a predetermined execution cycle, for example, while the power source for driving the vehicle is on or while the automatic driving function is enabled.

[0086] First, in step S201, the diagnosis result for each recognition module is obtained, and the process proceeds to step S202. As described above, the recognition module includes each surroundings monitoring sensor 11, the map linkage unit F1, and the fusion unit F2. Note that the individual diagnosis process described above may be executed for each recognition module as step S201. Step S201 corresponds to an abnormality detection step.

[0087] In step S202, it is determined whether or not an abnormal sensor is present. If no abnormal sensor is present, a negative decision is made in step S202 and the process proceeds to step S203. If an abnormal sensor is present, a positive decision is made in step S202 and the process proceeds to step S209.

[0088] In step S203, it is read whether the abnormality detection unit G4 has determined that the fusion unit F2 is normal. If the abnormality detection unit G4 has determined that the fusion unit F2 is normal, an affirmative decision is made in step S203 and the process proceeds to step S204. On the other hand, if the abnormality detection unit G4 has detected an abnormality in the fusion unit F2, a negative decision is made in step S203 and the process proceeds to step S210.

[0089] In step S204, it is read whether the map linking unit F1 is determined to be normal by the abnormality detection unit G4. If the map linking unit F1 is determined to be normal by the abnormality detection unit G4, an affirmative decision is made in step S204 and the process proceeds to step S205. On the other hand, if the abnormality detection unit G4 detects an abnormality in the map linking unit F1, a negative decision is made in step S204 and the process proceeds to step S207.

[0090] In step S205, the risk level is set to 0, and the process proceeds to step S206. In step S206, a signal indicating that no abnormality has occurred in the external recognition system is output to the safety evaluation unit F4. This allows normal automated driving to be maintained.

[0091] In step S207, the risk level is set to 1, and the process proceeds to step S208. In step S208, a signal requesting the safety evaluation unit F4 to perform a safe action is output. As a result, a safe action, such as extending a safe distance, is performed.

[0092] In step S209, it is determined whether the degree of use (in other words, the degree of dependency) of the abnormal sensor is equal to or greater than a predetermined danger determination threshold. If the degree of use of the abnormal sensor is equal to or greater than the danger determination threshold, a positive judgment is made in step S209 and the process proceeds to step S210. On the other hand, if the degree of use of the abnormal sensor is less than the danger determination threshold, a negative judgment is made in step S209 and the process proceeds to step S207.

[0093] In step S210, the risk level is set to 2, and the process proceeds to step S211. In step S211, a signal requesting the safety evaluation unit F4 to take emergency action is output. This causes emergency action such as MRM to be executed. Since MRM is a control for safely stopping the vehicle, step S211 corresponds to a vehicle stopping processing step.

[0094] According to the above configuration, when an abnormality occurs in the external recognition system, a safe action or an emergency action is adopted. This reduces the possibility that the autonomous driving will continue in a situation where the calculation result of the potential accident responsibility value may be erroneous. As a result, the safety during autonomous driving can be improved.

[0095] When the risk level is 1, the diagnoser 30 may continue normal control without implementing a safety action. This is because when the risk level is 1, the influence of the abnormality sensor on the recognition result of the driving environment is small, and it is possible to implement a control plan that generally uses the observation results. In another aspect, when the risk level is 1, the diagnoser 30 may implement an emergency action in the same way as when the risk level is 2.

[0096] <Additional information on abnormality detection section G4> Various methods can be used to detect abnormalities, such as a watchdog timer method or a homework answer method. The watchdog timer method is a method in which if a watchdog timer in a monitoring device expires without being cleared by a watchdog pulse input from a monitored device, the monitored device is determined to be not operating normally. Here, the abnormality detection unit G4 corresponds to the monitoring configuration, and each of the perimeter monitoring sensors 11, the map linkage unit F1, the fusion unit F2, etc. correspond to the monitored device.

[0097] <Supplementary information on variations of perimeter monitoring sensor 11> The vehicle control system 1 may include, as the front camera 11A, a camera for monitoring the surroundings configured to capture images at a relatively wide angle and at a short distance, in addition to a camera arranged at the upper end of the windshield on the interior side of the vehicle and configured to capture images at a relatively long distance. In other words, the front camera 11A may include a plurality of cameras with different angles of view. For example, three types of cameras, a mid-range camera, a telephoto camera, and a wide-angle camera, may be included. The mid-range camera is a camera with an angle of view of about 50 degrees, and a lens configured to capture images up to 150 m, for example. The telephoto camera is a camera configured to have a relatively narrow angle of view so that it can capture images at a longer distance than the mid-range camera. For example, the telephoto camera is configured to have an angle of view of about 30 degrees to 40 degrees, and can capture images at a distance of 250 m or more. The wide-angle camera is a camera configured to capture images of the surroundings of the vehicle in a wide range. The wide-angle camera is configured to have an angle of view of about 120 degrees to 150 degrees, and can capture images up to 50 m ahead of the vehicle, for example.

[0098] The vehicle control system 1 may also include a rear camera, a right side camera, a left side camera, a right rear radar, and a left rear radar as the periphery monitoring sensor 11. The rear camera is a camera that captures the rear of the vehicle at a predetermined angle of view. The rear camera is arranged at a predetermined position on the rear of the body, such as near the rear license plate or near the rear window. The right side camera and the left side camera are cameras that capture the sides of the vehicle at a predetermined angle of view, and are arranged at, for example, the side mirrors or predetermined positions on the left and right sides of the body (for example, near the base of the A-pillar). The horizontal angle of view of the right side camera and the left side camera is set to 180°, and the vertical angle of view is set to 120°. The right rear radar is a millimeter wave radar that detects a predetermined range on the right rear of the vehicle by transmitting a search wave toward the right rear of the vehicle, and is installed, for example, at the right corner of the rear bumper. The left rear radar is a millimeter wave radar that detects a predetermined range on the left rear of the vehicle by transmitting a search wave toward the left rear of the vehicle, and is installed, for example, at the left corner of the rear bumper.

[0099] In addition, the position information of obstacles and other vehicles acquired through road-to-vehicle communication or vehicle-to-vehicle communication also corresponds to information indicating the environment around the host vehicle. Therefore, the V2X vehicle-mounted device 13 can also be regarded as one of the surroundings monitoring sensors 11.

[0100] <Additional information on the operation of the external recognition system> The fusion unit F2 determines the position, speed, type, etc. of each detected object by integrating the recognition results of each perimeter monitoring sensor 11 with the probability value and the weight for each perimeter monitoring sensor 11. However, for each perimeter monitoring sensor 11, there are types of objects that are theoretically difficult to recognize, and situations in which the recognition accuracy deteriorates. In consideration of such circumstances, it is preferable to correct the correct answer probability value of the recognition result based on the characteristics (strengths and weaknesses) of each perimeter monitoring sensor 11 before use. For example, the external world recognition system can be configured as follows.

[0101] As shown in Fig. 11, each perimeter monitoring sensor 11 includes a sensing unit B1, a recognition processing unit B2, a sensor setting storage unit B3, a correction information output unit B4, a correction information acquisition unit B5, a probability value correction unit B6, and a detection result output unit B7. Note that, although each component will be described here using the front camera 11A as an example, perimeter monitoring sensors other than the front camera 11A, such as the millimeter wave radar 11B and the LiDAR 11C, also include similar functional units. In Fig. 11, the map linkage unit F1 and other components are omitted.

[0102] The sensing unit B1 is configured to generate observation data used in object recognition processing and output it to the recognition processing unit B2. The sensing unit B1 includes a sensor element for generating the observation data. For example, the sensor element for the front camera 11A is an image sensor. For the millimeter wave radar 11B and the sonar 11D, the sensing unit B1 corresponds to a configuration including a circuit for receiving radio waves or ultrasonic waves as a search wave. In the LiDAR 11C, the sensing unit B1 corresponds to a configuration including a light receiving element for receiving a reflected wave of an irradiated laser light.

[0103] The recognition processing unit B2 is configured to recognize the position, speed, type, etc. of the detected object based on the observation data input from the sensing unit B1. The type is identified based on the feature value indicated in the observation data. HOG features, etc. can be adopted as the feature value of the image data. In addition, in a configuration using radio waves, ultrasonic waves, radar light, etc. as the detection medium, for example, the correlation between the detection distance and the reception intensity, the height, width, contour shape, number of detection points, density of detection points, etc. of the object can be adopted as the feature value. The recognition result of the recognition processing unit B2 outputs a signal indicating the position information and type, etc. of each detected object. As described above, the identification result of the type of the detected object includes correct answer probability value data indicating the likelihood of the identification result. It includes correct answer probability value data for each type. For example, the correct answer probability value data indicates the probability value that the detected object is a vehicle, a probability value that the detected object is a pedestrian, a probability value that the detected object is a cyclist, and a probability value that the detected object is a three-dimensional structure such as a signboard.

[0104] The sensor setting storage unit B3 is a storage medium that stores sensor position data indicating the mounting position and detection direction of each perimeter monitoring sensor 11. The sensor setting storage unit B3 also stores sensor characteristic data, which is data indicating the characteristics of each perimeter monitoring sensor 11. Such a sensor setting storage unit B3 can also be called a sensor characteristic storage unit. The sensor characteristic data indicates, for each perimeter monitoring sensor, difficult objects that are difficult to detect due to the characteristics of the detection principle, and difficult situations that are situations in which the detection performance of an object may deteriorate. Note that difficult objects include objects that are likely to be mistaken for other types of objects and objects whose detection results are unstable.

[0105] For example, the sonar 11D is poor at detecting objects that do not easily reflect ultrasonic waves, such as sponge-like objects and net-like objects. In addition, depending on the vehicle's posture, the sonar 11D may or may not be able to recognize floating structures, which are structures arranged at a certain distance above the road surface, such as beams, and low-profile three-dimensional objects arranged on the road surface, such as curbs and wheel stops. In other words, for the sonar 11D, floating three-dimensional objects and low-profile three-dimensional objects correspond to detection targets whose detection results are unstable. Note that a floating structure refers to a three-dimensional object arranged, for example, 1 m or more above the road surface. A low-profile three-dimensional object refers to a three-dimensional object whose height is less than a certain threshold (for example, 0.2 m). Furthermore, there are cases in which the sonar 11D erroneously recognizes reflections from the road surface as three-dimensional objects such as other vehicles or walls. In other words, sponge-like objects, net-like objects, floating three-dimensional objects, low-profile three-dimensional objects, and the road surface correspond to objects that the sonar 11D is poor at detecting.

[0106] In addition, in the case of the front camera 11A, when a part of the image is blown out due to backlighting, light beams, smears, etc., there are cases where a part of the image frame cannot be recognized. In other words, a situation where a part of the image is blown out due to backlighting, light beams, smears, etc. corresponds to a poor situation.

[0107] The LiDAR11C is not good at detecting black objects. This is because black objects absorb laser light. The LiDAR11C also has difficulty detecting vehicles with a silver body color. The aluminum flakes contained in the paint of a silver body have the property of scattering light. If the vehicle is directly in front of the vehicle, it is easy to detect it, but since the reflection of light changes when the vehicle is at an angle, a silver vehicle that is diagonally in front or diagonally behind the vehicle can be an object that the LiDAR11C is not good at detecting.

[0108] Moreover, the millimeter wave radar 11B has a lower spatial resolution than the forward camera 11A and the LiDAR 11C. Therefore, in a case where a plurality of objects are close to each other or overlap each other, it is difficult to detect them as separate objects. In other words, a case where a plurality of objects are close to each other or overlap each other corresponds to a situation where the millimeter wave radar 11B is not good at detecting them. In addition, the millimeter wave radar 11B may recognize an object having a complex shape as a separate object. In other words, an object having a complex shape corresponds to an object that the millimeter wave radar 11B is not good at detecting. Furthermore, in the millimeter wave radar 11B, the search wave passes under the preceding vehicle, and the vehicle before the preceding vehicle, which is a vehicle further ahead of the preceding vehicle, may or may not be recognized. In other words, the vehicle before the preceding vehicle corresponds to an object that the millimeter wave radar 11B is not good at detecting. In addition, the millimeter wave radar 11B may erroneously detect a manhole as a three-dimensional object. Therefore, a manhole can also be included in the objects that the millimeter wave radar 11B is not good at detecting.

[0109] In addition, in any of the perimeter monitoring sensors 11, when an object is present near the sensing unit B1, there are cases where the object behind the sensing unit B1 cannot be recognized due to obstruction by the nearby object. In other words, in any of the perimeter monitoring sensors 11, a situation where an object is present near the mounting position of the sensing unit B1 may be considered a poor situation. Note that the term "nearby" in this disclosure refers to, for example, within 0.3 m.

[0110] The correction information output unit B4 judges whether or not an object or a situation that is unsuitable for another sensor, which is a certain surroundings monitoring sensor 11, has been detected based on the output signal of the recognition processing unit B2 and the data stored in the sensor setting storage unit B3. If an object or a situation that is unsuitable for the other sensor has been detected, the correction information output unit B4 transmits predetermined probability value correction information to the unsuitable sensor, which is the other sensor that is unsuitable for detecting the object or situation.

[0111] The probability value correction information is information that notifies another sensor of the presence of an object or situation that the other sensor is not good at detecting. The probability value correction information includes, for example, information indicating the type of the object or situation that is not good at detecting, and its position. Note that the probability value correction information may include the direction in which the object or situation that is not good at detecting exists, instead of the position of the object or situation that is not good at detecting. The probability value correction information may also include both the position and the direction of the object or situation that is not good at detecting.

[0112] For example, when the correction information output unit B4 of the front camera 11A recognizes at least one of a sponge-like object, a net-like object, a floating three-dimensional object, a low-profile three-dimensional object, and a road surface area by image analysis, it outputs probability value correction information indicating the position or direction of the object toward the sonar 11D. Also, when the correction information output unit B4 of the front camera 11A detects a black object by image analysis, it outputs probability value correction information indicating the direction or position of the black object toward the LiDAR 11C. Furthermore, when the correction information output unit B4 of the front camera 11A detects the presence of a vehicle ahead of the vehicle by image analysis, it outputs probability value correction information indicating the direction or position of the vehicle ahead of the vehicle to the millimeter wave radar 11B.

[0113] In addition, when each correction information output unit B4 detects the presence of an object near another sensor that is attached at a different position from its own sensor, it generates probability value correction information indicating the position of the nearby object, etc., and outputs it to the other sensor. For example, when the right side camera detects the presence of a bicycle near the right rear radar, it outputs probability value correction information indicating this to the right rear radar. In addition, when a certain camera can detect the position of the sun or a light with a predetermined brightness or higher, it outputs probability value correction information indicating the position or direction of the light source to a camera that is affected by the light source.

[0114] The correction information acquisition unit B5 acquires probability value correction information addressed to the sensor itself output from other sensors and provides it to the probability value correction unit B6. For example, the correction information acquisition unit B5 of the front camera 11A acquires the direction in which the sun, headlights, etc. are present as probability value correction information from another camera as the other sensor. Also, for example, the correction information acquisition unit B5 of the sonar 11D acquires probability value correction information indicating the area or direction in which a sponge-like object, a net-like object, a floating three-dimensional object, a low-profile three-dimensional object, a road surface area, etc. are present from the front camera 11A and LiDAR 11C as other sensors.

[0115] The probability value correction unit B6 lowers the probability of correct recognition of the object for the position or direction where the object or situation is not good, based on the probability value correction information acquired by the correction information acquisition unit B5. For example, the front camera 11A lowers the probability of correct recognition of the area where backlight, smear, light beam, etc. may exist. The sonar 11D also lowers the probability of correct recognition of the area where a spongy object exists, the road surface area, etc., detected by other sensors. When the probability value correction unit B6 of the LiDAR 11C is notified of an area where a black object exists as probability value correction information from the front camera 11A, it lowers the probability of correct recognition of the area.

[0116] Furthermore, the probability value correction unit B6 of the millimeter wave radar 11B reduces the accuracy rate of the recognition result for the area where multiple objects are close to or overlapping, as detected by the front camera 11A or the LiDAR 11C. For example, when the presence of the vehicle before the vehicle ahead is notified from the front camera 11A, the LiDAR 11C, or the V2X vehicle-mounted device 13, the probability value correction unit B6 of the millimeter wave radar 11B reduces the accuracy rate of the recognition result of the target on the extension of the vehicle ahead. This can reduce the risk of the detection result for the area ahead of the vehicle ahead being unstable. As another aspect, the probability value correction unit B6 of the millimeter wave radar 11B may increase the accuracy rate of the recognition result of the target on the extension of the vehicle ahead when the presence of the vehicle before the vehicle ahead is notified from the front camera 11A, etc. This can also reduce the risk of the detection result of the millimeter wave radar 11B becoming unstable.

[0117] The reduction or increase of the correct answer probability in the probability value correction unit B6 may be a constant value. For example, the reduction or increase of the correct answer probability may be 10% or 20%. The reduction of the correct answer probability may be changed according to the type of the output source of the probability value correction information. The reduction of the correct answer probability in the probability value correction unit B6 on the receiving side may be changed according to the correct answer probability for the target object in the output source of the probability value correction information. When the same probability value correction information is acquired from multiple other sensors, the reduction may be made larger. It is preferable that the probability value correction information is configured to be reduced to a larger reduction amount as the reliability of the probability value correction information is higher. When the contents notified in the probability value correction information are consistent with the recognition result of the own sensor, the correct answer probability of the recognition result for the target area may be increased. The probability value correction unit B6 adopts the output of the recognition processing unit B2 as it is for the recognition result for the position or direction for which the probability value correction information is not acquired.

[0118] The detection result output unit B7 outputs the generated recognition results, i.e., the recognition results of the own sensor and data indicating the correct answer probability, to the map linking unit F1 or the fusion unit F2. The fusion unit F2 integrates the recognition results of each perimeter monitoring sensor 11 to calculate the final type, position, moving speed, acceleration, etc. of the detected object. This is used for calculating the safe distance for each target as a detected object and for tracking processing.

[0119] According to the above configuration, when an object / situation that a certain perimeter monitoring sensor 11 is not good at is present, the result of the recognition processing unit B2 of the sensor itself is not adopted as is, but the correct answer probability value of the recognition result is lowered by a predetermined amount based on the probability value correction information from the other sensors. This makes it possible to increase the correct answer probability of the recognition result after integration. Note that the perimeter monitoring sensor 11 described above corresponds to a configuration that communicates with other sensors and outputs the correct answer probability of the recognition result of the sensor itself with a lowered value based on the recognition results of the other sensors.

[0120] Incidentally, in the above description, the related perimeter monitoring sensors 11 communicate with each other, and the correct answer probability is corrected within the perimeter monitoring sensor 11 and output to the fusion unit F2, etc., but the method for realizing the above idea is not limited to this.

[0121] As shown in Fig. 12, each perimeter monitoring sensor 11 may not include the correction information acquisition unit F5 and the probability value correction unit B6. Instead, the fusion unit F2 includes the correction information acquisition unit F5 and the probability value correction unit B6. In the configuration example shown in Fig. 12, each perimeter monitoring sensor 11 notifies the fusion unit F2 of the recognition result and correct answer probability of the own sensor. In addition, when each perimeter monitoring sensor 11 recognizes an object or situation that is difficult for the other sensors, it outputs probability value correction information including information of the target other sensor to the fusion unit F2.

[0122] The probability value correction information here includes identification information of the other target sensors (e.g., sensor ID), the position or direction of the unsuitable object or unsuitable situation, and their type. The probability value correction unit B6 included in the fusion unit F2 corrects the correct answer probability of the recognition result for each sensor using the probability value correction information. With such a configuration, the fusion unit F2 integrates the detection results of each perimeter monitoring sensor 11 after correcting the correct answer probability of the recognition result for each sensor using the probability value correction information, thereby increasing the correct answer probability of the recognition result after integration.

[0123] For example, when integrating the recognition results of the sonar 11D and the rear camera, if the rear camera detects a low object such as a curb or a wheel stopper, the fusion unit F2 integrates the information by lowering the weight of the recognition result of the sonar 11D for the corresponding position. According to this, if the sonar 11D recognizes that no three-dimensional object exists in the corresponding area, it is possible to reduce the possibility that the recognition result after integration will be erroneous due to the influence of the recognition result of the sonar 11D. Also, for example, if the presence of an object in front of a certain perimeter monitoring sensor is detected by another sensor with a different mounting position, the accuracy rate of the recognition result of the perimeter monitoring sensor 11 that is close to the object is reduced. This makes it possible to reduce cases in which the recognition result after integration is erroneous. In addition, when the fusion unit F2 determines that there is snowfall of a predetermined amount or more by the image sensor or LiDAR, it may integrate the information by lowering the weight of the recognition result by the sonar, taking into consideration the possibility that snow is attached to the sonar.

[0124] The above configuration corresponds to a configuration in which, when the first sensor, which is an arbitrary perimeter monitoring sensor, detects an object / situation that the second sensor, which is another sensor, is poor at, the reliability of the recognition result of the second sensor is determined to be low based on the detection result. Also, the fusion unit F2, which integrates the sensing information, corresponds to a configuration in which the weight of the recognition result of the second sensor is lowered to integrate the sensing information. That is, the above fusion unit F2 corresponds to a configuration in which, in a configuration in which the recognition results of a plurality of perimeter monitoring sensors 11 are integrated, the weight of the recognition results of other sensors is lowered based on the recognition result of a certain sensor to integrate the information. Note that, in a configuration in which the weight of each sensor in sensor fusion is determined based on the accuracy probability of the recognition result, correcting the accuracy probability corresponds to correcting the weight of each sensor during sensor fusion. The second sensor corresponds to a poor sensor.

[0125] In the above, when a first sensor, which is an arbitrary perimeter monitoring sensor, detects an object / situation that a second sensor, which is another sensor, is not good at, a mode has been disclosed in which the accuracy rate of the recognition result of the second sensor is lowered based on the detection result. However, while there are objects / situations that the sensor is not good at, there may also be objects and situations that the sensor is good at. An object that is good at for a certain perimeter monitoring sensor is an object that is easy to detect. A situation that is good at for a perimeter monitoring sensor refers to a situation in which the detection accuracy does not deteriorate. When the first sensor detects an object / situation that the second sensor is good at, the accuracy rate of the recognition result of the second sensor may be increased based on the detection result. Such a configuration can also increase the accuracy rate of the recognition result after integration by the fusion unit F2. When the accuracy rate of the recognition result of the second sensor is increased based on the recognition result of the first sensor, probability value correction information may be transmitted and received between the sensors or between the sensor and the fusion unit F2. Note that a person skilled in the art would be able to understand, based on the above disclosure, how to realize a configuration that increases the accuracy rate of the recognition result of the second sensor based on the recognition result of the first sensor, and therefore detailed explanation thereof will be omitted.

[0126] <System Configuration Variations> As shown in FIG. 13, a safety evaluator 40 equivalent to the safety evaluation unit F4 may be provided independently of the automatic driving device 20. The safety evaluator 40 is a module having the same function as the safety evaluation unit F4, and is configured to determine the final control content based on the potential accident liability value for the control plan generated by the automatic driving device 20. The above-mentioned description of the safety evaluation unit F4 and the description of the safety evaluator 40 can be implemented by replacing each other. The safety evaluator 40 is configured as a computer including a processing unit 41 and a memory 42. The processing unit 41 is configured using a processor. The memory 42 includes a RAM and a flash memory.

[0127] [Second embodiment] The vehicle control system 1 may include an external recognition system for the safety evaluator 40 to calculate a potential accident responsibility value, in addition to the external recognition system for the control planning unit F3 to generate a control plan. That is, the external recognition system may be configured as a dual system. Hereinafter, a configuration corresponding to such a technical idea will be described as a second embodiment with reference to Figs. 14 and 15. For convenience, the external recognition system for generating a control plan will be described as a recognition system for a control plan. On the other hand, the external recognition system for generating a recognition result for calculating a potential accident responsibility value will be described as a recognition system for a safety evaluation (hereinafter, SE: Safety Evaluation). SE can be described as SA (Safety Assessment).

[0128] As shown in FIG. 14, the vehicle control system 1 of the second embodiment includes a second periphery monitoring sensor 17 in addition to the first periphery monitoring sensor 11 corresponding to the above-mentioned periphery monitoring sensor 11. The first periphery monitoring sensor 11 includes the above-mentioned front camera 11A, millimeter wave radar 11B, LiDAR 11C, and sonar 11D. The first periphery monitoring sensor 11 is a group of sensors used by an automatic driving device 20 including a control planning unit F3. The map linking unit F1 and the fusion unit F2 included in the automatic driving device 20 recognize the driving environment based on the output signal of the first periphery monitoring sensor 11. The first periphery monitoring sensor 11, the map linking unit F1, and the fusion unit F2 correspond to a recognition module that configures a recognition system for control planning. The map linking unit F1 and the fusion unit F2 correspond to a first recognition unit.

[0129] The second periphery monitoring sensor 17 is, for example, a front camera 17A, a millimeter wave radar 17B, a LiDAR 17C, a sonar 17D, etc. The front camera 17A is a sensor that is physically separate from the front camera 11A. The same is true for the millimeter wave radar 17B, the LiDAR 17C, the sonar 17D, etc.

[0130] For example, the various first periphery monitoring sensors 11 may be sensors mounted on the vehicle body, while the second periphery monitoring sensor 17 may be a group of sensors attached to the roof of the vehicle. The second periphery monitoring sensor 17 may be configured to be detachable from the vehicle body as a set for automatic driving, for example. The positions of the first periphery monitoring sensor 11 and the second periphery monitoring sensor 17 may be interchangeable. That is, the first periphery monitoring sensor 11 may be a detachable sensor attached to the roof of the vehicle or the like.

[0131] The vehicle control system 1 of the second embodiment also includes a recognition device 50 for SE that recognizes the driving environment based on an output signal of the second periphery monitoring sensor 17. The recognition device 50 for SE is configured as a computer including a processing unit 51 and a memory 52. ​​The processing unit 51 is configured using a processor such as a GPU, and the memory 52 includes a RAM and a flash memory.

[0132] The recognition device 50 for the SE includes a map linking unit H1 and a fusion unit H2, as shown in FIG. 15, in the same manner as the automatic driving device 20. The map linking unit H1 is configured to perform object recognition based on the recognition result of the front camera 17A and map data. The map linking unit H1 can be configured to perform the same processing as the map linking unit F1 included in the automatic driving device 20. The fusion unit H2 is configured to perform object recognition based on the recognition results of the map linking unit H1, the millimeter wave radar 17B, the LiDAR 17C, and the sonar 17D. The fusion unit H2 can also be configured to perform the same processing as the fusion unit F2 included in the automatic driving device 20. The second periphery monitoring sensor 17, the map linking unit H1, and the fusion unit H2 correspond to a recognition module that configures a recognition system for the SE. The map linking unit H1 and the fusion unit F2 correspond to a recognition unit and a second recognition unit.

[0133] The diagnoser 30 receives the output signal of the first periphery monitoring sensor 11, the recognition result of the map linking unit F1, the recognition result of the fusion unit F2, as well as the output signal of the second periphery monitoring sensor 17, the recognition result of the map linking unit H1, and the recognition result of the fusion unit H2. The abnormality detection unit G4 of the diagnoser 30 also performs the above-mentioned individual diagnosis process on each of the various second periphery monitoring sensors 17, the map linking unit H1, and the fusion unit H2. In other words, the individual diagnosis process is also performed on the recognition modules that constitute the recognition system for the SE.

[0134] Furthermore, the abnormality detection unit G4 detects an abnormality in either one of the recognition results of the fusion unit F2 for the control plan and the fusion unit F2 for the SE by comparing the recognition results of the fusion unit F2 for the control plan. For example, as shown in FIG. 16, when the recognition results of the fusion unit F2 for the control plan and the fusion unit H2 for the SE are similar, it is determined that both are normal. Note that the vertical axis of FIG. 16 indicates the probability value, and the horizontal axis indicates the time. The dashed line and the two-dot chain line indicate the recognition results of the fusion unit F2 for the control plan, and the dashed line and the dotted line indicate the recognition results of the fusion unit H2 for the SE. Specifically, the dashed line indicates the transition of the probability value p1f that the type of the target object is a vehicle in the fusion unit F2 for the control plan, and the two-dot chain line indicates the transition of the probability value p2f that the type of the target object is a pedestrian in the fusion unit F2. The dashed line indicates the transition of the probability value p1h that the type of the target object is a vehicle in the fusion section H2 for SE, and the dotted line indicates the transition of the probability value p2h that the type of the target object is a pedestrian in the fusion section H2.

[0135] For example, the anomaly detection unit G4 calculates the similarity between the two recognition results by comparing history data of probability values ​​of recognition results for a predetermined target object by pattern matching or the like. Then, if the similarity between the recognition results is equal to or greater than a predetermined threshold (e.g., 75%), it is determined that both are operating normally. On the other hand, if the similarity between the two recognition results is less than the predetermined threshold and is different, the anomaly detection unit G4 determines that an abnormality has occurred in either the fusion unit F2 for control plan or the fusion unit H2 for SE. In that case, it identifies which one has an abnormality based on the result of the individual diagnosis process. Determining that the recognition results are similar corresponds to determining that the similarity results are consistent.

[0136] FIG. 17 illustrates a pattern in which, although the classifications themselves as recognition results are consistent, it is determined that an abnormality has occurred in either the fusion unit for control plan F2 or the fusion unit for SE H2 based on a large difference in probability values. Specifically, it illustrates a pattern in which the fusion unit for control plan F2 and the fusion unit for SE H2 both determine that the classification of the target object is a vehicle, but the probability values ​​differ by a predetermined threshold value (e.g., 30%) or more. FIG. 18 illustrates a pattern in which, based on a difference in classifications themselves as recognition results, it is determined that an abnormality has occurred in either the fusion unit for control plan F2 or the fusion unit for SE H2. Specifically, it illustrates a case in which the fusion unit for control plan F2 determines that the target object is a vehicle, while the fusion unit for SE H2 determines that the target object is a pedestrian. Items indicated by symbols and line types in FIG. 17 and FIG. 18 are the same as those in FIG. 16.

[0137] If the recognition results of the fusion units F2 and H2 for control planning and SE are inconsistent, i.e., if one of them is abnormal, the abnormality detection unit G4 determines which one is abnormal based on the results of individual diagnosis processing.

[0138] The above describes the process of detecting an abnormality in either one of the map linking units F1 and H1 by comparing the recognition results of the fusion unit F2 for control planning and the fusion unit H2 for SE. The abnormality detection unit G4 also performs a similar comparison process on the map linking units F1 and H1. That is, the abnormality detection unit FG4 detects an abnormality in either one of the map linking units F1 and H1 by comparing the recognition results of the map linking unit F1 for control planning and the map linking unit H1 for SE. This makes it possible to more accurately determine whether an abnormality has occurred in the map linking units F1 and H1. In addition, when the recognition results of the map linking units F1 and H1 for control planning and SE are not consistent, that is, when one of them is abnormal, the abnormality detection unit G4 identifies which one has an abnormality based on the result of the individual diagnosis process.

[0139] If an abnormality occurs in the recognition system for the SE, the diagnoser 30 may notify the safety evaluator 40 of the abnormality and may cause the safety evaluator 40 to calculate the potential accident responsibility value using the recognition result of the fusion unit F2 for the control plan. With this configuration, the calculation of the potential accident responsibility value can be continued until the vehicle stops. As a result, it becomes possible to select a control plan that stops the vehicle in a manner other than full braking. By being able to adopt a deceleration control manner other than full braking, it is possible to reduce the risk of tires slipping or locking.

[0140] Furthermore, if an abnormality occurs in the recognition system for the control plan, the diagnoser 30 notifies the automatic driving device 20 and the safety evaluator 40 of the abnormality. In addition, if an abnormality occurs in the recognition system for the control plan, the diagnoser 30 may cause the control plan unit F3 to create a control plan for stopping the vehicle using the recognition result of the fusion unit H2 for SE. With this configuration, the vehicle can be stopped more safely.

[0141] In addition, the risk determination unit G5 of the second embodiment may determine the risk as 1 when it is determined that the fusion unit H2 for SE is not operating normally. The risk may be 2 when an abnormality occurs in the fusion unit H2. Even if it is determined that the fusion unit H2 for SE is not operating normally, the risk may be considered to be 0 as long as the fusion unit F2 for control plan is operating normally. This is because, as long as the fusion unit F2 for control plan is operating normally, the safety evaluator 40 uses the recognition result of the fusion unit F2 for control plan, and the calculation of the potential personal liability value can be continued.

[0142] The risk level determination unit G5 may be configured to determine the risk level as 2 when an abnormality occurs in both the fusion unit F2 for control plan and the fusion unit H2 for SE, and to set the risk level to 1 or less as long as either one of them is operating normally. Alternatively, the risk level may be determined to be 1 based on the determination that the map linkage unit H1 is not operating normally. Alternatively, the risk level may be determined to be 1 when an abnormal sensor is present in the second periphery monitoring sensor 17 and the usage level of the abnormal sensor is equal to or greater than a predetermined risk determination threshold.

[0143] According to the above-mentioned configuration, the recognition results of the two recognition systems are compared, and furthermore, an abnormality diagnosis is performed for each system. By performing an abnormality diagnosis in each recognition system in this way, even if an abnormality occurs in both systems at the same time, it is possible to identify that an abnormality has occurred in both systems. Furthermore, when an abnormality is detected in one of the recognition systems by comparing the recognition results of the two recognition systems, it is possible to identify which system has an abnormality based on the result of the individual diagnosis process. When an abnormality is detected in either the recognition system for the control plan or the recognition system for the SE, it is preferable that the diagnoser 30 executes at least a safety action such as deceleration or a handover request.

[0144] As shown in FIG. 19, the vehicle control system 1 may include a third fusion unit X2 that recognizes the surrounding environment by integrating sensing information of a part of the first periphery monitoring sensor 11 and sensing information of a part of the second periphery monitoring sensor 17. For example, the third fusion unit X2 may be configured to integrate the recognition results of the front camera 11A and the millimeter wave radar 11B as the first periphery monitoring sensor 11 and the recognition results of the LiDAR 17C and the sonar 17D as the second periphery monitoring sensor 17. The third fusion unit X2 may be included in the automatic driving device 20 or the recognition device 50 for SE. The third fusion unit X2 may be included in another ECU such as a driving assistance ECU, or may be built in the front camera 11A. The recognition result of the third fusion unit X2 is also input to the diagnoser 30.

[0145] According to the above configuration, the diagnoser 30 can detect any abnormal operation by comparing the recognition results of the three fusion units F2, H2, and X2. For example, the abnormality detection unit G4 may identify the fusion unit that is operating abnormally using a concept such as majority vote. The third fusion unit X2 can be realized by logically combining the first periphery monitoring sensor 11 and the second periphery monitoring sensor 17. Therefore, it is possible to provide triple redundancy without adding a new periphery monitoring sensor 11 in terms of hardware.

[0146] <Additional remarks> The control unit and the method described in the present disclosure may be realized by a dedicated computer that configures a processor programmed to execute one or more functions embodied by a computer program. The device and the method described in the present disclosure may also be realized by a dedicated hardware logic circuit. The device and the method described in the present disclosure may also be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. The computer program may also be stored in a computer-readable non-transient tangible recording medium as instructions executed by a computer. In other words, the means and / or functions provided by the automatic driving device 20 and the diagnoser 30, etc. can be provided by software recorded in a substantial memory device and a computer that executes the software, software only, hardware only, or a combination thereof. For example, some or all of the functions provided by the diagnoser 30 may be realized as hardware. Aspects in which a certain function is realized as hardware include aspects in which it is realized using one or more ICs, etc. The processing units 21, 31, 41, and 51 may be realized using an MPU or GPU instead of a CPU. The processing units 21, 31, 41, and 51 may be realized by combining a plurality of types of arithmetic processing devices, such as a CPU, an MPU, and a GPU. The processing units 21, 31, 41, and 51 may be realized as a system-on-chip (SoC). Furthermore, the various processing units may be realized using a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The various programs may be stored in a non-transitive tangible storage medium.A variety of storage media can be used as the program storage medium, such as a hard-disk drive (HDD), a solid-state drive (SSD), an erasable programmable ROM (EPROM), a flash memory, a USB memory, or a secure digital (SD) memory card. [Explanation of symbols]

[0147] 11 Periphery monitoring sensor (first peripheral monitoring sensor), 11A Front camera (camera), 11B Millimeter wave radar, 11C LiDAR (laser radar), 11D Sonar, 12 Vehicle state sensor, 13 V2X vehicle-mounted device, 14 Map storage unit, 15 Locator, 16 Travel actuator, 20 Autonomous driving device, 21 Processing unit (processor), 30 Diagnosis device, 31 Processing unit (processor), 40 Safety evaluator, 41 Processing unit, 50 SE recognition device, 51 Processing unit, F1 Map linkage unit (recognition unit, first recognition unit), F2 Fusion unit (recognition unit, first recognition unit), F3 Control planning unit, F4 Safety evaluation unit, G1 Diagnostic material acquisition unit (recognition result acquisition unit), G2 Recognition result storage unit, G3 Correspondence relationship identification unit, G4 Anomaly detection unit, G5 Danger level determination unit, G6 danger level response unit (vehicle stop processing unit), H1 map linkage unit (recognition unit, second recognition unit), H2 fusion unit (recognition unit, second recognition unit), B1 sensing unit, B2 recognition unit, B3 sensor characteristic storage unit, B4 correction information output unit, B5 correction information acquisition unit, B6 Probability value correction section, B7 detection result output section

Claims

1. A vehicle control device for use in a vehicle, the vehicle control device comprising: a control planning unit (F3) that creates a control plan for autonomously driving the vehicle; and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, an abnormality detection unit (G4) that detects an abnormality in an external recognition system including the periphery monitoring sensors and the recognition units based on at least one of output signals of a plurality of periphery monitoring sensors (11, 11A-11D, 17, 17A-17D) that detect objects present within a predetermined detection range and a recognition result of a recognition unit (F1, F2, H1, H2, X2) that recognizes objects present around the vehicle based on the output signals of the periphery monitoring sensors; A vehicle stop processing unit (G6) that executes a process for stopping the vehicle based on the abnormality detection unit detecting an abnormality in the external environment recognition system, The abnormality detection unit is configured to compare the processing by the external recognition system used in the control planning unit with the processing by the external recognition system used in the safety evaluation unit that is similar to the processing by the external recognition system used in the control planning unit, and when the results are different, determine that an abnormality has occurred in the external recognition system; A vehicle control device in which the processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processes that fuse the recognition results of multiple surrounding monitoring sensors.

2. The vehicle control device according to claim 1, A vehicle control device, wherein the abnormality detection unit determines that the abnormality has occurred in either the control planning unit or the safety evaluation unit when the type of the target object as a recognition result is different.

3. A vehicle control device for use in a vehicle, the vehicle control device comprising: a control planning unit (F3) that creates a control plan for autonomously driving the vehicle; and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, an abnormality detection unit (G4) that detects an abnormality in an external recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A-11D, 17, 17A-17D) that detects an object present within a predetermined detection range and a recognition result of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor; A vehicle stop processing unit (G6) that executes a process for stopping the vehicle based on the abnormality detection unit detecting an abnormality in the external environment recognition system, The abnormality detection unit is configured to compare the processing by the external recognition system used in the control planning unit with the processing by the external recognition system used in the safety evaluation unit that is similar to the processing by the external recognition system used in the control planning unit, and when the results are different, determine that an abnormality has occurred in the external recognition system; A vehicle control device, wherein the processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processing for identifying the vehicle's position on a map.

4. A vehicle control device for use in a vehicle, the vehicle control device comprising: a control planning unit (F3) that creates a control plan for autonomously driving the vehicle; and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, an abnormality detection unit (G4) that detects an abnormality in an external recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A-11D, 17, 17A-17D) that detects an object present within a predetermined detection range and a recognition result of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor; A vehicle stop processing unit (G6) that executes a process for stopping the vehicle based on the abnormality detection unit detecting an abnormality in the external environment recognition system, The abnormality detection unit is configured to compare the processing by the external recognition system used in the control planning unit with the processing by the external recognition system used in the safety evaluation unit that is similar to the processing by the external recognition system used in the control planning unit, and when the results are different, determine that an abnormality has occurred in the external recognition system; The processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processing for recognizing the environment around the vehicle by combining the recognition results of the surrounding monitoring sensors and map data.

5. The vehicle control device according to any one of claims 1 to 4, When it is determined that the abnormality has occurred in the case where the difference is present, the abnormality detection unit executes an individual diagnosis process to identify which of the external environment recognition system used in the control planning unit and the external environment recognition system used in the safety evaluation unit has the abnormality, When it is determined that the abnormality has occurred in the processing by the external recognition system used in the safety evaluation unit, the vehicle control device notifies the external recognition system used in the safety evaluation unit of the occurrence of the abnormality, so as to cause the safety evaluation unit to continue calculating parameters that are used as a reference for evaluating safety using results of the processing by the external recognition system used in the control planning unit.

6. The vehicle control device according to claim 5, When it is determined that the abnormality has occurred in the processing by the external recognition system used in the control planning unit, the vehicle control device notifies the external recognition system used in the control planning unit of the occurrence of the abnormality, so as to have the control planning unit create the control plan using a result of the processing by the external recognition system used in the safety evaluation unit.

7. The vehicle control device according to any one of claims 1 to 6, The abnormality detection unit calculates a similarity between a result of processing by the external environment recognition system used in the control planning unit and a result of processing by the external environment recognition system used in the safety evaluation unit, and if the similarity is less than a predetermined threshold, determines that the abnormality has occurred in either the external environment recognition system used in the control planning unit or the external environment recognition system used in the safety evaluation unit.

8. The vehicle control device according to claim 7, The abnormality detection unit determines that the external environment recognition system used in the control planning unit and the external environment recognition system used in the safety evaluation unit are normal when the similarity is equal to or greater than the threshold value.

9. A vehicle control method for use in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, the vehicle control method being executed by at least one processor (21, 31), An abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensors and the recognition unit based on at least one of output signals of a plurality of periphery monitoring sensors (11, 11A to 11D, 17, 17A to 17D) that detect an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signals of the periphery monitoring sensors; A vehicle stop processing step (S211) of executing a process for stopping the vehicle based on the detection of an abnormality in the external environment recognition system in the abnormality detection step, In the abnormality detection step, a process by the external recognition system used in the control planning unit is compared with a process by the external recognition system used in the safety evaluation unit that is similar to the process by the external recognition system used in the control planning unit, and if the results are different, it is determined that an abnormality has occurred in the external recognition system; A vehicle control method in which the processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processes that fuse the recognition results of multiple surrounding monitoring sensors.

10. A vehicle control method for use in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, the vehicle control method being executed by at least one processor (21, 31), An abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor; A vehicle stop processing step (S211) of executing a process for stopping the vehicle based on the detection of an abnormality in the external environment recognition system in the abnormality detection step, In the abnormality detection step, a process by the external recognition system used in the control planning unit is compared with a process by the external recognition system used in the safety evaluation unit that is similar to the process by the external recognition system used in the control planning unit, and if the results are different, it is determined that an abnormality has occurred in the external recognition system; A vehicle control method, wherein the processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processing for identifying the vehicle's position on a map.

11. A vehicle control method for use in a vehicle including a control planning unit (F3) that creates a control plan for autonomously driving the vehicle, and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, the vehicle control method being executed by at least one processor (21, 31), An abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor; A vehicle stop processing step (S211) of executing a process for stopping the vehicle based on the detection of an abnormality in the external environment recognition system in the abnormality detection step, In the abnormality detection step, a process by the external recognition system used in the control planning unit is compared with a process by the external recognition system used in the safety evaluation unit that is similar to the process by the external recognition system used in the control planning unit, and if the results are different, it is determined that an abnormality has occurred in the external recognition system; The processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processing for recognizing the environment around the vehicle by combining the recognition results of the surrounding monitoring sensors and map data.

12. A vehicle control program for use in a vehicle, the vehicle control program comprising: a control planning unit (F3) that creates a control plan for autonomously driving the vehicle; and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, At least one processor (21, 31) An abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensors and the recognition unit based on at least one of output signals of a plurality of periphery monitoring sensors (11, 11A to 11D, 17, 17A to 17D) that detect an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signals of the periphery monitoring sensors; A vehicle stop processing step (S211) for executing a process for stopping the vehicle based on the detection of an abnormality in the external recognition system in the abnormality detection step, In the abnormality detection step, a process by the external recognition system used in the control planning unit is compared with a process by the external recognition system used in the safety evaluation unit that is similar to the process by the external recognition system used in the control planning unit, and if the results are different, it is determined that an abnormality has occurred in the external recognition system; A vehicle control program in which the processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processes that fuse the recognition results of multiple surrounding monitoring sensors.

13. A vehicle control program for use in a vehicle, the vehicle control program comprising: a control planning unit (F3) that creates a control plan for autonomously driving the vehicle; and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, At least one processor (21, 31) An abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor; A vehicle stop processing step (S211) for executing a process for stopping the vehicle based on the detection of an abnormality in the external recognition system in the abnormality detection step, In the abnormality detection step, a process by the external recognition system used in the control planning unit is compared with a process by the external recognition system used in the safety evaluation unit that is similar to the process by the external recognition system used in the control planning unit, and if the results are different, it is determined that an abnormality has occurred in the external recognition system; A vehicle control program in which the processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are processing for identifying the vehicle's position on a map.

14. A vehicle control program for use in a vehicle, the vehicle control program comprising: a control planning unit (F3) that creates a control plan for autonomously driving the vehicle; and a safety evaluation unit (40) that evaluates safety of the control plan created by the control planning unit, At least one processor (21, 31) An abnormality detection step (S201) of detecting an abnormality in an external recognition system including the periphery monitoring sensor and the recognition unit based on at least one of an output signal of at least one periphery monitoring sensor (11, 11A to 11D, 17, 17A to 17D) that detects an object present within a predetermined detection range and an output signal of a recognition unit (F1, F2, H1, H2, X2) that recognizes an object present around the vehicle based on the output signal of the periphery monitoring sensor; A vehicle stop processing step (S211) for executing a process for stopping the vehicle based on the detection of an abnormality in the external recognition system in the abnormality detection step, In the abnormality detection step, a process by the external recognition system used in the control planning unit is compared with a process by the external recognition system used in the safety evaluation unit that is similar to the process by the external recognition system used in the control planning unit, and if the results are different, it is determined that an abnormality has occurred in the external recognition system; The processing by the external recognition system used in the control planning unit and the processing by the external recognition system used in the safety evaluation unit are a vehicle control program that recognizes the environment around the vehicle by combining the recognition results of the surrounding monitoring sensors and map data.

Images