Vehicle control device and vehicle control method

Abstract

A vehicle control device and a vehicle control method are disclosed. An automatic driving device includes a fusion unit (F2) that identifies a traveling environment of a vehicle based on output signals of surrounding monitoring sensors and a control plan unit (F3) that generates a control plan based on an identification result of the fusion unit. A diagnosis device (30) includes an abnormality detection unit (G4) that detects an abnormality of a real-time identification system from the surrounding monitoring sensors to the fusion unit (F2) by monitoring the output signals of the surrounding monitoring sensors and the identification result of the fusion unit over time. The diagnosis device (30) requests the control plan unit (F3) to perform an MRM based on the detection of the abnormality of the identification system by the abnormality detection unit (G4).

Description

Technical Field

[0001] This disclosure generally relates to techniques for generating control plans for autonomous vehicles. Background Technology

[0002] Patent document 1 discloses the following configuration: in this configuration, a vehicle driving plan, in other words, a control plan, is generated using a mathematical formula model called the RSS (Responsibility Sensitive Safety) model and map data during autonomous driving.

[0003] In the RSS model, the planner, acting as a functional block for developing control plans, uses map data to calculate the potential accident liability value for each of multiple control plans, and adopts a control plan where the potential accident liability value is within the permissible range. The potential accident liability value is a parameter indicating the degree of responsibility of the vehicle in the event of an accident between the vehicle and nearby vehicles. The potential accident liability value takes into account whether the inter-vehicle distance between the vehicle and nearby vehicles is less than a safe distance determined based on road structure, etc.

[0004] [Patent Document 1] WO 2018 / 115963 A1

[0005] In the RSS model, the potential accident liability value for each control plan generated by the planner is calculated based on map data and sensing information from surrounding surveillance sensors, such as cameras, and the plan determined to be relatively safe is selected. However, anomalies in the surrounding surveillance sensors themselves or in the system that performs object recognition processing based on the observation data from the surrounding surveillance sensors (hereinafter referred to as the scene recognition system) can lead to miscalculations of the potential accident liability value. Ultimately, if the assessment of the potential accident liability value is incorrect, the safety of autonomous driving deteriorates. Summary of the Invention

[0006] The purpose of this disclosure is to provide a vehicle control device that can improve safety during autonomous driving based on the above assumptions.

[0007] The vehicle control device for achieving this purpose is, for example, a vehicle control device that creates a control plan for the autonomous driving of the vehicle, the vehicle control device comprising:

[0008] (i) An anomaly detection unit that detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: (a) the output signal of at least one ambient monitoring sensor detecting objects within a predetermined detection range, and (b) the identification unit identifying an object near the vehicle based on the output signal of at least one ambient monitoring sensor, and

[0009] (ii) A vehicle stop processing unit that performs processing to stop the vehicle based on the detection of anomalies in the field identification system by the anomaly detection unit.

[0010] The above configuration corresponds to or can be set to the following configuration: in which, when the anomaly detection unit of the vehicle control device for autonomous driving detects an anomaly in the recognition system as a trigger, it plans to stop vehicle control. Using this configuration, the vehicle control device can reduce the chance of continuing autonomous driving in a state that causes an anomaly in the recognition system. In other words, safety during autonomous driving can be improved.

[0011] Furthermore, the vehicle control method for achieving the above objectives is a vehicle control method executed by at least one processor to create a control plan for autonomous driving of the vehicle, the vehicle control method comprising:

[0012] (i) Anomaly detection step, which detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: (a) detecting the output signal of at least one ambient monitoring sensor of an object within a predetermined detection range, and (b) the identification unit identifying the identification result of an object near the vehicle based on the output signal of at least one ambient monitoring sensor, and

[0013] (ii) A vehicle stopping procedure, which is performed to stop the vehicle based on the detection of anomalies in the field identification system by the anomaly detection unit.

[0014] Here, the reference numerals enclosed in parentheses in the claims indicate a correspondence with the specific means described in the embodiments that serve as examples of this disclosure. Therefore, the technical scope of this disclosure is not necessarily limited thereto. Attached Figure Description

[0015] The purpose, features, and advantages of this disclosure will become more apparent from the following detailed description with reference to the accompanying drawings, in which:

[0016] Figure 1 This is a diagram used to illustrate the configuration of an autonomous driving system;

[0017] Figure 2 This is a block diagram showing the configuration of the vehicle control system 1;

[0018] Figure 3 This is a block diagram used to illustrate the functions of the automatic drive device 20 and the diagnostic device 30;

[0019] Figure 4 This is a diagram used to illustrate the operation of the risk assessment unit G5;

[0020] Figure 5This is another diagram used to illustrate the operation of the risk determination unit G5;

[0021] Figure 6 This is another diagram used to illustrate the operation of the risk assessment unit G5;

[0022] Figure 7 This is a flowchart illustrating the operation of the anomaly detection unit G4;

[0023] Figure 8 This is a diagram used to illustrate the operation of the anomaly detection unit G4;

[0024] Figure 9 This is a diagram used to illustrate the operation of the anomaly detection unit G4;

[0025] Figure 10 This is a flowchart illustrating the operation of the diagnostic device 30;

[0026] Figure 11 This is a diagram used to illustrate the operation of the identification system;

[0027] Figure 12 This is a diagram used to illustrate the operation of the identification system;

[0028] Figure 13 This is a diagram showing the modifications to the system configuration;

[0029] Figure 14 This is a block diagram showing the configuration of the vehicle control system 1 according to the second embodiment;

[0030] Figure 15 This is a block diagram illustrating the operation of the diagnostic device 30 according to the second embodiment;

[0031] Figure 16 This is a diagram illustrating the operation of the anomaly detection unit G4 in the second embodiment;

[0032] Figure 17 This is a diagram illustrating the operation of the anomaly detection unit G4 in the second embodiment;

[0033] Figure 18 This is a diagram illustrating the operation of the anomaly detection unit G4 in the second embodiment; and

[0034] Figure 19 This is a diagram showing a modification of the configuration of the second embodiment. Detailed Implementation

[0035] An embodiment of a vehicle control system 1 to which the vehicle control device according to this disclosure is applied will be described with reference to the accompanying drawings. The following description uses a region where left-hand traffic is legal as an example. In regions where right-hand traffic is legal, the left and right sides can be reversed. This disclosure can be appropriately modified and implemented to comply with the laws and customs of the regions where the vehicle control system 1 is used.

[0036] (First Implementation)

[0037] In the following description, a first embodiment of the present disclosure will be described with reference to the accompanying drawings. Figure 1 This is a diagram illustrating an example configuration of an autonomous driving system according to this disclosure. (See diagram for example.) Figure 1 As shown, the autonomous driving system includes a vehicle control system 1 built on a vehicle Ma and an external server Sv. The vehicle control system 1 can be installed / assembled on a vehicle that can travel on roads, and the vehicle Ma can be a four-wheeled vehicle, a two-wheeled vehicle, a three-wheeled vehicle, etc. Motorized bicycles can also be included in two-wheeled vehicles. The vehicle Ma can be a privately owned car, a shared car, or a service vehicle. Service vehicles include taxis, fixed-route buses, and shared buses. Taxis or buses can be robot taxis without a driver on board, etc.

[0038] The vehicle control system 1 downloads partial map data as local high-precision map data from an external server Sv via wireless communication and uses it for autonomous driving and navigation. In the following description, the vehicle on which the vehicle control system 1 is installed is also described as vehicle Ma, and the occupant sitting in the driver's seat of vehicle Ma (i.e., the driver's seat occupant) can also be described as a user. The concept of driver's seat occupant also includes an operator with the authority to remotely control vehicle Ma. The forward, left, right, and up / down directions in the following description are defined with reference to vehicle Ma. Specifically, the forward / backward direction corresponds to the longitudinal direction of vehicle Ma. The left / right direction corresponds to the width direction of vehicle Ma. The vertical direction corresponds to the height direction of the vehicle. From another perspective, the vertical direction corresponds to a direction perpendicular to a plane parallel to both the forward / backward and left / right directions.

[0039] <Map Data>

[0040] The following describes the map data maintained by the external server Sv. The map data corresponds to map data that shows the road structure, the location coordinates of feature objects on the ground and along the road, etc., with appropriate accuracy for autonomous driving. The map data includes node data, link data, feature object data, etc. Node data includes various data such as a unique node ID assigned to each node on the map, node coordinates, node name, node type, and a link ID describing the links connected to the node.

[0041] Link data refers to data about links that act as connecting nodes on a road. Link data includes various data such as a link ID (a unique identifier for the link), link shape information (hereinafter referred to as link shape), node coordinates for the start and end points of the link, road attributes, etc. The link shape can be represented by a coordinate sequence indicating the shape of the road edge. The link shape corresponds to the road shape. The link shape can be represented by a three-order spline curve. Road attributes include, for example, road name, road type, road width, lane number information indicating the number of lanes, speed limit values, etc. Link data may also include data indicating the road type, such as whether the road is a highway or a regular road. Here, a highway refers to a road where pedestrians and bicycles are prohibited, such as a toll road like an expressway. Link data may include attribute information indicating whether autonomous driving is permitted on the road. Link data can be subdivided and described for each lane.

[0042] Feature object data includes lane marking data and landmark data. Lane marking data includes a lane marking ID for each lane marking and a set of coordinate points representing the installation portion of the feature object. Lane marking data includes pattern information such as dashed lines, solid lines, and road tracks. Lane marking data is associated with lane information such as lane-level lane IDs or link IDs. Landmarks are three-dimensional structures located along the road. Three-dimensional structures along the road include, for example, guardrails, curbs, trees, utility poles, road signs, traffic lights, etc. Road signs include directional signs such as direction signs and road name signs. Landmark data represents the location and type of each landmark. The shape and location of each feature object are represented by a set of coordinate points. POI (Point of Interest) data indicates the location and type of feature objects that affect vehicle travel plans, such as branch points, intersections, speed limit change points, lane change points, traffic congestion, building sections, intersections, tunnels, toll booths, etc. POI data includes type and location information.

[0043] Map data can be 3D map data including a set of feature points representing the shape and structure of roads. 3D map data corresponds to map data that represents the location of feature objects such as road edges, lane markings, and road signs in 3D coordinates. Note that 3D maps can be generated by REM (Road Experience Management) based on captured images. Furthermore, map data can include driving trajectory models. Driving trajectory models are track data generated by statistically integrating the driving trajectories of multiple vehicles. A driving trajectory model is, for example, an average of the driving trajectory for each lane. A driving trajectory model corresponds to data indicating the driving trajectory used as a reference during autonomous driving.

[0044] The map data is managed by dividing it into multiple smaller blocks. Each block corresponds to map data for a different region. For example... Figure 1 As shown, for example, map data is stored in units of map tiles, where the map recording area is divided into rectangular shapes of 2 square kilometers. Map tiles correspond to the subordinate concept of the aforementioned small blocks. Each map tile is given information indicating the real-world area it corresponds to. This information is represented by, for example, latitude, longitude, and altitude. Furthermore, each map tile is given a unique ID (hereinafter referred to as the tile ID). The map data for each small block or each map tile is a part of the entire map recording area; in other words, partial map data. Map tiles correspond to partial map data. The external server Sv distributes partial map data based on requests from the vehicle control system 1, according to the location of the vehicle control system 1.

[0045] Note that the shape of map tiles is not limited to a rectangle of 2 square kilometers. It can be a rectangle of 1 square kilometer or 4 square kilometers. Furthermore, map tiles can be hexagonal or circular. Each map tile can be configured to partially overlap adjacent map tiles. The map recording area can be the entire country where the vehicle uses the vehicle, or it can be a region, i.e., a portion of the country. For example, the map recording area can be limited to areas where autonomous driving of general vehicles is permitted or areas providing autonomous mobility services (e.g., buses). Furthermore, the way map data is divided can be defined by the data size. In other words, the map recording area can be divided and managed within a range defined by the data size. In this case, each small tile is configured such that the data volume is less than a predetermined value. According to this pattern, the data size in a distribution can be set to a certain value or smaller. Map data is updated in real time by, for example, integrating probe data uploaded from multiple vehicles.

[0046] Configuration of Vehicle Control System 1

[0047] Next, we will refer to Figure 2 The configuration of the vehicle control system 1 according to the first embodiment is described. Figure 2 The vehicle control system 1 shown is used in vehicles capable of performing autonomous driving (hereinafter referred to as autonomous vehicles). Figure 2As shown, the vehicle control system 1 includes a surrounding monitoring sensor 11, a vehicle status sensor 12, a V2X onboard unit 13, a map-keeping unit 14, a locator 15, a driving control actuator 16, an automatic drive unit 20, and a diagnostic device 30. Note that "HMI" in the description is an abbreviation for Human-Machine Interface. Furthermore, V2X is an abbreviation for Vehicle to X (Everything), and refers to the communication technology that connects various things to the vehicle. In V2X, "V" refers to the vehicle itself (Ma), and "X" can refer to various objects other than the vehicle itself (Ma), such as pedestrians, other vehicles, road equipment, networks, servers, etc.

[0048] The level of "autonomous driving" indicated in this disclosure can be, for example, equivalent to Level 3 as defined by the Society of Automotive Engineers (SAE International), or it can be Level 4 or higher. Below, an example of this vehicle Ma performing autonomous driving at at least Level 3 or higher will be described. Note that Level 3 refers to the level where the system performs all driving tasks within its Operational Design Domain (ODD), where, in an emergency, operational authority is transferred from the system to the user. The ODD defines the conditions under which autonomous driving can be performed, such as driving on an expressway. At Level 3, the user is required to respond quickly to requests for changes / switches from the system. The person taking over driving can be an operator located outside the vehicle. Level 3 corresponds to so-called conditional autonomous driving. Level 4 is the level where the system can perform all driving tasks except in specific situations such as driving on roads incompatible with autonomous driving, extreme environments, etc. Level 5 is the level where the system can perform all driving tasks in any environment. Levels 3 through 5 can also be referred to as autonomous driving levels that automatically perform all controls related to vehicle driving.

[0049] The surrounding monitoring sensor 11 is a sensor that monitors the "peripheral" area or surroundings of the vehicle. The surrounding monitoring sensor 11 is configured to detect the presence and location of predetermined targets. Detected targets include, for example, moving objects such as pedestrians or other vehicles. Other vehicles may include bicycles, motorized bicycles, and motorcycles. Furthermore, the surrounding monitoring sensor 11 is configured to detect predetermined feature objects. Feature objects to be detected by the surrounding monitoring sensor 11 include road edges, road markings, and three-dimensional structures installed / built along the road. Road markings refer to paint drawn on the road surface for traffic control and traffic regulations. For example, lane markings indicating lane boundaries, pedestrian crossings, stop lines, merging zones, safety zones, regulatory arrows, etc., are included in road markings. Lane markings are also called lane lines or lane markers. Lane markings also include those implemented by road studs such as peepholes (or vibrating strips) and reflective dots. As mentioned above, three-dimensional structures installed along the road are, for example, guardrails, road signs, traffic lights, etc. That is, preferably, the surrounding monitoring sensor 11 is configured to detect landmarks. Curbs, guardrails, walls, etc., also belong to the aforementioned three-dimensional structures. The surrounding monitoring sensor 11 can be configured to detect fallen objects, i.e., objects left on the road.

[0050] For example, peripheral surveillance cameras, millimeter-wave radar, LiDAR (light detection and ranging / laser imaging detection and ranging), sonar, etc., can be used as peripheral surveillance sensors 11. Perimeter surveillance cameras are vehicle-mounted cameras arranged to image the exterior of the vehicle in a predetermined direction. Perimeter surveillance cameras include front-facing cameras positioned at the upper end of the windshield inside the vehicle, the front grille, etc., to capture / image the front field of the vehicle Ma. Millimeter-wave radar emits millimeter waves or quasi-millimeter waves in a predetermined direction and analyzes the received data of reflected waves from the emitted waves reflected by objects, thereby detecting the relative position and relative velocity of the object relative to the vehicle Ma. Millimeter-wave radar generates, for example, data indicating the received intensity and relative velocity for each detection direction and distance, or data indicating the relative position and received intensity of the detected object, as observation data. LiDAR is a device that generates three-dimensional point cloud data indicating the position of reflection points in each detection direction by illuminating a laser field. LiDAR can also be called lidar. LiDAR can be scanning or flash type. Sonar is a device that detects the relative position and relative velocity of an object with respect to its own vehicle Ma by emitting ultrasonic waves in a predetermined direction and analyzing the data of the received reflected waves, which are reflections of the emitted waves reflected back by the object.

[0051] exist Figure 3In this system, the surrounding surveillance sensor 11 includes a front-facing camera 11A, a millimeter-wave radar 11B, a LiDAR 11C, and a sonar 11D. The front-facing camera 11A, millimeter-wave radar 11B, LiDAR 11C, and sonar 11D are all configured to include the vehicle's direction of travel (e.g., the frontal / forward direction) within the detection range. Note that the sonar 11D can be mounted on the vehicle's front and rear bumpers.

[0052] The front-facing camera 11A, LiDAR 11C, etc., detect the aforementioned target by using a classifier such as a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network). The millimeter-wave radar 11B and sonar 11D detect the aforementioned target by analyzing the intensity of the received reflected waves, the distribution of detection points, etc.

[0053] Each output of the surrounding surveillance sensor 11 is a signal indicating the relative position, type, speed, etc., of each detected object as a detection result. The object type identification result includes the similarity of the identification results, i.e., the probability data indicating the extent to which the identification of the object type is correct. For example, the probability data includes the probability value that the detected object is a vehicle, the probability value that the detected object is a pedestrian, the probability value that the detected object is a cyclist, and the probability value that the detected object is a three-dimensional structure such as a sign. The probability values ​​for each type can be viewed as scores indicating the degree of matching of feature values. Note that "cyclist" here refers to a bicycle ridden by a rider or a rider on a bicycle. The detection results of the surrounding surveillance sensor 11 can be considered as identification results or determination results.

[0054] Furthermore, when an internal malfunction occurs, each of the surrounding monitoring sensors 11 outputs an error signal to the vehicle network Nw. For example, when an anomaly is detected in the image sensor or processing circuitry, the front camera device 11A outputs an error signal. The error signals output by the surrounding monitoring sensors 11 are input to, for example, the automatic drive unit 20 and / or the diagnostic device 30.

[0055] Note that the type of sensor used by the vehicle control system 1 as the ambient monitoring sensor 11 can be appropriately designed and does not necessarily include all of the aforementioned sensors. Furthermore, object recognition processing based on the observation data generated by the ambient monitoring sensor 11 can be performed by an external ECU (electronic control unit), such as the automatic drive unit 20. The automatic drive unit 20 can be configured to include some or all of the object recognition functions incorporated in the ambient monitoring sensor 11, such as the front-facing camera 11A and millimeter-wave radar. In this case, various ambient monitoring sensors 11 can provide the automatic drive unit 20 with observation data such as image data and ranging data as detection result data.

[0056] Briefly return to Figure 2 The vehicle state sensor 12 is a group of sensors that detects state quantities related to the driving control of the vehicle Ma. The vehicle state sensor 12 includes a vehicle speed sensor, a steering sensor, an acceleration sensor, a yaw rate sensor, etc. The vehicle speed sensor detects the vehicle's speed. The steering sensor detects the vehicle's steering angle. The acceleration sensor detects the vehicle's acceleration, such as longitudinal acceleration, lateral acceleration, etc. The acceleration sensor can also detect deceleration, i.e., acceleration in the negative direction / negative value. The yaw rate sensor detects the vehicle's angular velocity. The types of sensors used by the vehicle control system 1 as the vehicle state sensor 12 can be appropriately designed and do not necessarily include all of the aforementioned sensors.

[0057] The V2X onboard unit 13 is a device in the vehicle Ma used to perform wireless communication with other devices. The V2X onboard unit 13 includes a wide-area communication unit and a narrow-area communication unit as communication modules. The wide-area communication unit is a communication module used to perform wireless communication conforming to a predetermined wide-area wireless communication standard. Various standards such as LTE (Long Term Evolution), 4G, and 5G can be used as the wide-area wireless communication standard here. In addition to communication via a wireless base station, the wide-area communication unit can also be configured to perform wireless communication directly with other devices using a method conforming to the wide-area wireless communication standard, i.e., without going through a base station. That is, the wide-area communication unit can be configured to perform cellular V2X. The vehicle Ma becomes a connected car that can connect to the Internet by installing the V2X onboard unit 13. For example, the automatic driving unit 20 downloads the latest partial map data from an external server Sv cooperating with the V2X onboard unit 13 based on the current location of the vehicle Ma. Furthermore, the V2X onboard unit 13 obtains traffic congestion information and weather information from the external server Sv, roadside units, etc. The traffic congestion information includes location information such as the start and end points of the traffic congestion.

[0058] The narrow-range communication unit included in the V2X vehicle-to-everything (V2X) device 13 is a communication module that communicates directly with other devices, namely other mobile objects and roadside units existing around the vehicle Ma, according to the following communication standard (hereinafter referred to as the narrow-range communication standard), in which the communication distance is limited to a range of several hundred meters or less. Other mobile objects are not limited to vehicles, but may also include pedestrians, bicycles, etc. Any standard such as WAVE (Wireless Access in a Vehicle Environment) and DSRC (Dedicated Short Range Communication) standards disclosed in IEEE 1709 can be used as the narrow-range communication standard. For example, the narrow-range communication unit broadcasts vehicle information about its own vehicle Ma to nearby vehicles at predetermined transmission intervals and receives vehicle information sent from other vehicles. Vehicle information includes vehicle ID, current location, direction of travel, speed, operating status of the direction indicator, timestamp, etc.

[0059] Map holding unit 14 is a device for storing partial map data corresponding to the current location of vehicle Ma, which is obtained from an external server Sv by V2X onboard device 13. Map holding unit 14 holds, for example, partial map data related to roads that vehicle Ma plans to traverse within a predetermined time period. Map holding unit 14 is a non-transitory storage medium. Map holding unit 14 can be used, for example, by including data for later reference... Figure 2 This is implemented using a portion of the storage area in the described memory 23 or RAM 22. The map holding unit 14 can be configured as a device built into the autonomous drive unit 20. The process of acquiring map data from an external server Sv cooperating with the V2X vehicle-mounted device 13 and storing the data in the map holding unit 14 can be controlled by the autonomous drive unit 20 or by the locator 15. Note that the map holding unit 14 can be a non-volatile storage device that stores all map data.

[0060] Positioner 15 is a device that generates highly accurate location information of the vehicle Ma by combining multiple pieces of information for combined positioning. Positioner 15 is configured using, for example, a GNSS receiver. A GNSS receiver is a device that repeatedly and / or intermittently detects its current position by receiving navigation signals (hereinafter referred to as positioning signals) transmitted from positioning satellites that constitute a GNSS (Global Navigation Satellite System). For example, if the GNSS receiver can receive positioning signals from four or more positioning satellites, it outputs a positioning result every 100 milliseconds. GPS, GLONASS, Galileo, IRNSS, QZSS, Beidou, etc., can be used as GNSS.

[0061] The locator 15 repeatedly and / or intermittently determines the position of the vehicle Ma by combining the positioning results from the GNSS receiver and the output of the inertial sensor. For example, when the GNSS receiver cannot receive GNSS signals, such as in a tunnel, the locator 15 performs dead reckoning (i.e., autonomous navigation) using the yaw rate and vehicle speed. The yaw rate used for dead reckoning can be calculated by a front-facing camera using SfM technology, or it can be detected by a yaw rate sensor. The locator 15 can use the output of an accelerometer or gyroscope sensor to perform dead reckoning. For example, the vehicle position can be represented in three-dimensional coordinates of latitude, longitude, and altitude. The vehicle position information provided by the locator 15 is output to the vehicle network Nw and used by the automatic drive unit 20, etc.

[0062] The locator 15 can be configured to perform positioning processing. Positioning processing is performed by comparing the coordinates of landmarks determined based on images captured by an in-vehicle camera such as the front-facing camera 11A with the coordinates of landmarks registered in map data to obtain the precise location of the vehicle Ma. Furthermore, the locator 15 can be configured to determine a driving lane ID, which identifies the lane in which the vehicle Ma is traveling, based on the distance to the road edge detected by the front-facing camera or millimeter-wave radar. The driving lane ID indicates, for example, the number of times the vehicle Ma is traveling from the leftmost or rightmost road edge. The automatic drive unit 20 may have some or all of the functions provided by the locator 15.

[0063] The driving control actuator 16 is a driving actuator. The driving control actuator 16 includes, for example, a brake actuator as a braking device, an electronic throttle, a steering actuator, etc. The steering actuator includes an EPS (electric power steering) motor. The driving control actuator 16 is controlled by the automatic drive unit 20. Note that a steering ECU for performing steering control and a power unit control ECU, a brake ECU, etc., for performing acceleration / deceleration control can be interposed between the automatic drive unit 20 and the driving control actuator 16.

[0064] The automatic drive unit 20 is an ECU (Electronic Control Unit) that performs some or all of the driving operations on behalf of the occupant in the driver's seat by controlling the driving control actuator 16 based on the detection results of the surrounding monitoring sensor 11. Here, as an example, the automatic drive unit 20 is configured to be able to operate up to automation level 5 and is configured to be able to switch operating modes corresponding to each automation level. In the following text, for convenience, the operating mode corresponding to automatic operating level N (N = 0 to 5) will also be referred to as level N mode. The following description continues under the assumption of operating in automation level 3 or higher. In operating mode 3 or higher, the automatic drive unit 20 automatically steers, accelerates, and decelerates (in other words, causes braking) the vehicle Ma, causing the vehicle Ma to travel along the road to the destination set by the occupant in the driver's seat or the operator. Note that in addition to user operation, the switching of operating modes is automatically performed due to system limitations, exiting the ODD, etc.

[0065] The automatic drive unit 20 mainly includes a computer having a processing unit 21, RAM 22, a memory 23, a communication interface 24, and a bus connecting them. The processing unit 21 is hardware for arithmetic processing combined with RAM 22. The processing unit 21 is configured to include at least one arithmetic core, such as a CPU (Central Processing Unit). The processing unit 21 performs various processes to implement the functions of each of the functional units by accessing RAM 22, which will be described later. The memory 23 is configured to include a non-volatile storage medium such as flash memory. The memory 23 stores the program executed by the processing unit 21 (hereinafter referred to as the automatic driving program). Execution of the automatic driving program by the processing unit 21 corresponds to executing the method corresponding to the automatic driving program as a vehicle control method. The communication interface 24 is a circuit for communicating with other devices via the in-vehicle network Nw. The communication interface 24 can be implemented using analog circuit elements, ICs, etc.

[0066] Communication interface 24 corresponds to a configuration for acquiring, for example, the detection results (i.e., sensing information) of the output signal of the surrounding monitoring sensor 11 and the detection results of the vehicle state sensor 12. Sensing information includes the position and speed of other moving objects, feature objects, obstacles, etc., existing around the vehicle Ma. For example, the distance between the vehicle in front of the vehicle Ma and the vehicle Ma, as well as the speed of the vehicle in front, are included in the sensing information. The vehicle in front here can include vehicles traveling in the same lane as the vehicle Ma (i.e., vehicles ahead), as well as vehicles traveling in adjacent lanes. That is, "in front" here is not limited to the direction directly in front of the vehicle Ma, but can include obliquely in front. Sensing information includes lateral distance from the road edge, lane ID, offset from the centerline of the lane, etc.

[0067] The automatic drive unit 20 receives various inputs from the vehicle status sensor 12, such as signals indicating the vehicle's speed (Ma), acceleration, yaw rate, etc., the vehicle's position information detected by the locator 15, and map data maintained in the map holding unit 14. Furthermore, traffic information acquired by the V2X onboard unit 13 from roadside units, and other vehicle information acquired through vehicle-to-vehicle communication, are also input to the automatic drive unit 20. Details of the operation of the automatic drive unit 20 based on these input signals will be described later.

[0068] The diagnostic device 30 is configured to detect anomalies in the scene identification system based on output signals from the ambient monitoring sensor 11 and the automatic drive unit 20. The scene identification system here also includes a separate ambient monitoring sensor 11. Furthermore, the scene identification system includes modules such as the fusion unit F2 described later, which performs processing to identify the environment surrounding the vehicle based on the output signals from the ambient monitoring sensor 11 in the automatic drive unit 20. The identification of the driving environment here includes identifying the type and location of objects present around the vehicle.

[0069] The diagnostic device 30 mainly includes a computer having a processing unit 31, RAM 32, a memory 33, a communication interface 34, and a bus connecting them. The memory 33 stores the program (hereinafter referred to as the vehicle control program) executed by the processing unit 31. The execution of the vehicle control program by the processing unit 31 corresponds to the execution of the method corresponding to the vehicle control program as a vehicle control method.

[0070] Note that the diagnostic device 30 receives input from various devices, namely detection results and error signals output by various ambient monitoring sensors 11, as well as data indicating the operation of the fusion unit F2 included in the automatic drive device 20. Data indicating the operation of the fusion unit F2 includes, for example, the fusion result, the utilization rate (in other words, the weighting) of the ambient monitoring sensors 11 used for sensor fusion, etc.

[0071] Furthermore, the diagnostic device 30 outputs a signal indicating the diagnostic results to the automatic drive unit 20. The signal indicating the diagnostic results includes the availability of various ambient monitoring sensors 11. Additionally, when an anomaly is detected in the system, a control signal corresponding to the severity of the anomaly is output to the automatic drive unit 20. Details of the diagnostic device 30 will be described later. The diagnostic device 30 corresponds to the vehicle control unit.

[0072] Configuration of automatic drive unit 20 and diagnostic unit 30 Figure 3 >

[0073] Here, we will refer to Figure 3The configuration of the automatic drive unit 20 and the diagnostic unit 30 is described. The automatic drive unit 20 includes a map linking unit F1, a fusion unit F2, a control planning unit F3, and a safety assessment unit F4, which are processed by the processing unit 21 ( Figure 2 The diagnostic device 30 comprises functional units implemented by the processing unit 31 executing the autonomous driving program. Furthermore, the diagnostic device 30 includes a diagnostic material acquisition unit G1, an identification result holding unit G2, a correspondence determination unit G3, an anomaly detection unit G4, a risk determination unit G5, and a risk response unit G6, which are functional units implemented by the processing unit 31 executing the vehicle control program.

[0074] The map linking unit F1 identifies the environment surrounding the vehicle by combining the recognition results from the front camera device 11A with map data. For example, it corrects the position and type of structures and road markings present in front of the vehicle. Map data is used to increase or correct the reliability and information content of the recognition results from the front camera device 11A. Furthermore, the map linking unit F1 determines the vehicle's position on the map by comparing structures shown as landmarks in the map data with the landmark information recognized by the front camera device 11A; this is referred to as positioning processing.

[0075] The map linking unit F1 outputs recognition result data, determined by complementary / compensatory use of the recognition results from the front camera device 11A and map data, to the diagnostic device 30 and the fusion unit F2. For convenience, the map linking unit F1 is also described as a first recognition unit because it corresponds to the configuration that performs preprocessing on the fusion unit F2, and the recognition result of the map linking unit F1 is also described as a first recognition result. The object recognition result data, which is the first recognition result, includes information indicating the location and type of each detected object. As mentioned above, the type information of the detected object includes correct recognition probability data for each type. For example, the correct recognition probability data includes the probability value that the detected object is a vehicle, the probability value that the detected object is a pedestrian, etc.

[0076] Furthermore, the map linking unit F1 outputs sensor usage data for each detected object to the diagnostic device 30 and the fusion unit F2. The sensor usage data for a given object indicates the weight of the recognition result from the front-facing camera 11A and the weight of the map data used to determine the object's type. The sensor usage rate can be a constant value or can vary depending on the distance from the vehicle to the detected object. Furthermore, the sensor usage rate can change based on the type of object recognized by the front-facing camera 11A and its correct recognition probability. Additionally, the sensor usage rate can change based on the driving scenario. For example, when image recognition performance may deteriorate, such as at night or in rain, the weight of the recognition result from the front-facing camera 11A is set relatively low. Besides the above, driving scenarios where image recognition performance may deteriorate may include sinking, glare / sunlight at sunset, near tunnel entrances / exits, driving on curves, and foggy conditions.

[0077] In this embodiment, the map linking unit F1 is arranged outside the fusion unit F2. However, the fusion unit F2 can have the functions of the map linking unit F1. The output signals of the front camera device 11A and the locator 15, as well as map data, can be directly input to the fusion unit F2. The map linking unit F1 is an arbitrary element. The map linking unit F1 and the fusion unit F2 correspond to the identification unit.

[0078] The fusion unit F2 identifies the driving environment of the vehicle Ma through sensor fusion processing, which integrates the detection results of various surrounding monitoring sensors 11 with predetermined weights (in other words, sensor utilization). The driving environment includes not only the location, type, and speed of objects around the vehicle, but also road curvature, the number of lanes, the driving lane as the vehicle's position on the road, weather, and road surface conditions. Weather and road surface conditions can be determined by combining the identification results from the front-facing camera 11A with weather information acquired through the V2X onboard device 13. Road structure is primarily determined based on information input from the map linking unit F1.

[0079] The data showing the identification results of the fusion unit F2 is also described as fusion result data in order to distinguish it from the identification results of the map linking unit F1. For example, the fusion unit F2 generates fusion result data by integrating the detection results of the map linking unit F1, millimeter-wave radar 11B, LiDAR 11C, and sonar 11D at a predetermined or dynamically set sensor utilization rate. The fusion result data is output to the diagnostic device 30, the control planning unit F3, and the safety assessment unit F4.

[0080] The fusion result data also includes information indicating the location and type of each detected object. For example, the fusion result data includes the type, location, shape, size, relative speed, etc., for each detected object. As mentioned above, the type information includes the correct recognition probability data for each type. Note that fusion unit F2 is configured to perform object recognition processing again / independently using the recognition processing result of map link unit F1, which is used as the first recognition unit, thereby being designated as a second recognition unit. Therefore, the recognition result of fusion unit F2 is also described as a secondary recognition result. In the fusion result data, the recognition result in fusion unit F2 is also described as a secondary recognition result. Hereinafter, the secondary recognition result data can be referred to as fusion result data.

[0081] The fusion unit F2 also outputs sensor usage data for each detected object to the diagnostic device 30, the control planning unit F3, and the safety assessment unit F4. As described above, the sensor usage data for a particular object indicates the weight of each information source used to determine the type of object. These information sources correspond to the surrounding surveillance sensors 11, such as millimeter-wave radar 11B, LiDAR 11C, etc. Furthermore, the map linking unit F1 also corresponds to the information sources for the fusion unit F2. The map linking unit F1 also identifies objects existing around the vehicle based on the recognition results from the front-facing camera 11A. Therefore, the map linking unit F1 can be included within the concept of the surrounding surveillance sensors 11.

[0082] The sensor utilization rate in fusion unit F2 can be a constant value or can vary depending on the distance from the vehicle to the detected object. The sensor utilization rate in fusion unit F2 can be set based on the correct recognition probability output by each surrounding monitoring sensor 11. For example, the weight of a surrounding monitoring sensor 11 with a correct recognition probability of 80% can be set to 0.8 (corresponding to 80%). The utilization rate of a surrounding monitoring sensor 11 in identifying a particular object can be a value obtained by multiplying the correct recognition probability of the surrounding monitoring sensor 11 by a predetermined conversion coefficient. Furthermore, the sensor utilization rate can be changed according to the type of object. Furthermore, the sensor utilization rate can be changed according to the driving scenario. Furthermore, the sensor utilization rate can be changed for each object detection target item. Detection target items include distance, horizontal azimuth angle, vertical azimuth angle, speed, size, type, etc. For example, regarding distance, vertical azimuth angle, and speed, based on the detection results of the front-facing camera 11A, the weight of the millimeter-wave radar 11B or LiDAR 11C may be greater than the weight of the map linking unit F1. Furthermore, regarding type and horizontal azimuth, the weight of LiDAR 11C or map linking unit F1 may be greater than that of millimeter-wave radar 11B.

[0083] The control planning unit F3 uses map data and the recognition results from the fusion unit F2 to generate a driving plan or control plan for the autonomous driving of the vehicle Ma through automated driving. For example, the control planning unit F3 performs route search processing as a medium- to long-term driving plan to generate a recommended route to guide the vehicle from its current location to its destination. Furthermore, based on the medium- to long-term driving plan, the control planning unit F3 generates lane-changing driving plans, driving plans for centering the vehicle in the lane, driving plans for following the vehicle in front, driving plans for following the vehicle in front, and driving plans for avoiding obstacles, etc., as short-term control plans for driving. For example, map data is used to determine the area where the vehicle can drive based on the number of lanes and road width, and the steering amount and target speed are set based on the curvature of the road ahead.

[0084] As a short-term control plan, the control planning unit F3 generates a route that maintains a certain distance from the identified lane markings, or a route that remains in the center of the lane, or a route based on the identified route of the preceding vehicle, or a route along the identified trajectory of the preceding vehicle. When the vehicle's lane corresponds to (i.e., includes) a road with multiple lanes on both sides of traffic, the control planning unit F3 can generate a candidate plan for changing lanes to an adjacent lane in the same direction as the vehicle's lane. When the fusion unit F2 identifies an obstacle in front of the vehicle Ma, the control planning unit F3 can generate a driving plan that passes one side of the obstacle. When the fusion unit F2 has already identified an obstacle in front of the vehicle Ma, the control planning unit F3 can generate a deceleration control plan to stop the vehicle before the obstacle. The control planning unit F3 can be configured to generate a driving plan that has been determined to be optimal through machine learning or the like.

[0085] Control planning unit F3 calculates, for example, one or more plan candidates as candidates for short-term driving plans. Multiple plan candidates have different acceleration / deceleration amounts, bumpy travel (i.e., short-distance travel), steering amounts, and the timing of various control actions. In other words, a short-term driving plan may include acceleration / deceleration schedule information for speed adjustments along the calculated route. Plan candidates can also be referred to as route candidates. Control planning unit F3 outputs data indicating at least one generated plan candidate to safety assessment unit F4.

[0086] Based on the identification results and map data from the fusion unit F2, the safety assessment unit F4 determines the final execution plan from the control plan generated by the control planning unit F3, and outputs control signals to the driving control actuator 16 according to the plan. The map data is used for calculating safe distances based on road structure and traffic rules / regulations, as well as for calculating potential accident liability values. The safety assessment unit F4 includes a liability value calculation unit F41 and an action determination unit F42 as sub-functions.

[0087] The liability value calculation unit F41 corresponds to a configuration used to evaluate the safety of driving plans generated by the control planning unit F3. As an example, the liability value calculation unit F41 assesses safety based on whether the distance between the vehicle and surrounding objects (hereinafter referred to as inter-object distance) is equal to or greater than a safe distance determined using a mathematical formula model that establishes a safe driving concept. For example, for each of the candidate plans developed by the control planning unit F3, the liability value calculation unit F41 determines / calculates a potential accident liability value indicating the degree of responsibility of the vehicle Ma regarding an accident occurring during the vehicle Ma's travel along the candidate plan. The potential accident liability value is determined as one of the determining factors by comparing (i) the inter-vehicle distance between the vehicle Ma and surrounding vehicles with (ii) the safe distance of the vehicle Ma while traveling on the road according to the plan candidate.

[0088] The lower the liability, the smaller the potential liability value for an accident. Therefore, the potential liability value for an accident decreases as the driver (Ma) becomes "more eager / more determined" to drive safely. For example, if sufficient distance between vehicles is maintained, the potential liability value for an accident becomes small. Furthermore, when the vehicle (Ma) suddenly accelerates or decelerates, the potential liability value for an accident may be large.

[0089] Furthermore, when vehicle Ma is traveling in accordance with traffic rules / regulations, the liability value calculation unit F41 can set the potential accident liability value to a low value. In other words, whether the route conforms to the traffic rules at the vehicle's location can also be used as a determining factor affecting the potential accident liability value. To determine whether vehicle Ma is traveling in accordance with traffic rules, the liability value calculation unit F41 can be configured to obtain the traffic rules at the point where vehicle Ma is traveling. The traffic rules at the point where vehicle Ma is traveling can be obtained from a predetermined database, or by analyzing images captured by cameras around vehicle Ma and detecting signs, traffic lights, road markings, etc. Traffic rules can be included in map data.

[0090] The safety distance used by the liability value calculation unit F41 is a parameter used as a reference for assessing the safety between the vehicle and a target vehicle, such as the vehicle in front, and is dynamically determined according to the driving environment. The safety distance is set based at least on behavioral information such as acceleration of the vehicle's Ma. Since various models can be used as methods for calculating the safety distance, detailed descriptions of the calculation methods are omitted here. Furthermore, for example, the RSS (Responsibility Sensitive Safety) model can be used as a mathematical formula model for calculating the safety distance. Additionally, the SFF (Safety Force Field, registered trademark) model can also be used as a mathematical formula model for calculating the safety distance. The safety distance includes the safety distance from the vehicle in front, i.e., the safety distance in the longitudinal direction, and the safety distance in the left-right direction, i.e., the horizontal / lateral direction. The aforementioned mathematical formula models include models for determining these two types of safety distances.

[0091] The mathematical model described above does not guarantee accident-free driving; however, it guarantees that the vehicle will be exempt from liability in an accident if appropriate collision avoidance measures are taken when the distance between vehicles drops below a safe distance. Braking with reasonable force can be considered an example of appropriate action to avoid a collision. Braking with reasonable force includes, for example, braking at the vehicle's maximum possible deceleration. The safe distance calculated by the mathematical model can be rewritten as the minimum distance a vehicle should maintain between itself and an obstacle to avoid close contact with the obstacle.

[0092] Action determination unit F42 is configured to determine a final execution plan based on one of a plurality of control plans, according to the potential accident liability value calculated by liability value calculation unit F41. For example, in action determination unit F42, the final execution plan is selected from the control plans generated by control plan unit F3, either (a) the plan with the minimum potential accident liability value calculated by liability value calculation unit F41, or (b) the plan with the potential accident liability value calculated to an allowable level.

[0093] The safety assessment unit F4 outputs a control signal corresponding to the control plan determined by the action determination unit F42 to the controlled object, that is, to the driving control actuator 16 to control it. For example, when deceleration is planned, the control signal for achieving the planned deceleration is output to the brake actuator and the electronic throttle.

[0094] The diagnostic material acquisition unit G1 acquires information as diagnostic material or clues to determine whether any anomalies have occurred in any of the field identification systems, namely, the surrounding monitoring sensor 11, the map linking unit F1, and the fusion system F2. Signals indicating the detection results of each surrounding monitoring sensor 11, identification result data from the map linking unit F1, and identification result data from the fusion unit F2 can be used as diagnostic material. Furthermore, when an error signal is output from the surrounding monitoring sensor 11, this error signal can also be used as diagnostic material. Note that determining whether an anomaly has occurred corresponds to detecting that an anomaly has occurred. An anomaly here refers to a state where it (i.e., sensor 11) cannot operate normally due to some kind of malfunction. Abnormal states of sensor 11 include states where no identification result is output due to a fault and states where the output signal is stuck (e.g., remains unchanged). The diagnostic material acquisition unit G1 repeatedly and / or intermittently acquires data of a predetermined type that can be used as diagnostic material from each of the surrounding monitoring sensor 11, the map linking unit F1, and the fusion unit F2. The diagnostic material acquisition unit G1 distinguishes the data acquired from each of these configuration components into their respective information types and stores them in the identification result holding unit G2. Furthermore, for example, data of the same type with different acquisition times are sorted and saved chronologically, with the newest data at the top. Various data are configured to determine the acquisition order, for example, by adding timestamps corresponding to the acquisition times. Note that data saved for a certain period may then be discarded sequentially. The diagnostic material acquisition unit G1 corresponds to the identification result acquisition unit.

[0095] The identification result holding unit G2 is implemented using a portion of a storage area, for example, in RAM 32. The identification result holding unit G2 is configured to store, for example, data acquired within the last 10 seconds. The identification result holding unit G2 corresponds to a memory used for temporarily compiling data as diagnostic material.

[0096] The correspondence determination unit G3 associates objects detected at a previous time with objects detected at the next time based on the position and movement speed of the detected objects. In other words, the correspondence determination unit G3 corresponds to a configuration for tracking previously detected objects (i.e., tracking the detected objects). Various methods can be used as object tracking methods. For example, the correspondence determination unit G3 estimates the current position based on the position at a previous time and the movement speed of each detected object, and the object closest to the estimated position in the current observation data can be considered the same object. Furthermore, the correspondence determination unit G3 can also be configured to perform an association (in other words, tracking) between previous and current data regarding whether the detected objects are the same object by using the similarity of their feature values ​​at these (current and previous) times. Items such as color histograms, size, and reflectivity can be used as feature values ​​for the detected objects based on the characteristics of each surrounding monitoring sensor 11.

[0097] Furthermore, the correspondence determination unit G3 also associates objects jointly detected by different surrounding surveillance sensors 11 based on the location information of the detected object. For example, if the millimeter-wave radar 11B also detects another vehicle within a predetermined distance from the location of another vehicle detected by the front-facing camera 11A, then both are considered the same object. Preferably, the same object is assigned the same detection target ID to facilitate tracking over time. The detected object ID is an identifier used to identify the detected object. The determination result of the correspondence determination unit G3 is reflected in the identification result data stored in the identification result holding unit G2.

[0098] The anomaly detection unit G4 is configured to detect anomalies in the field identification system based on identification result data stored in the identification result holding unit G2, etc. Details of the anomaly detection unit G4 will be described separately later. In the following text, the surrounding monitoring sensor 11 determined by the anomaly detection unit G4 to be malfunctioning will be referred to as an anomaly sensor. From another perspective, the anomaly detection unit G4 corresponds to a configuration used to determine the presence or absence of an anomaly sensor.

[0099] The risk assessment unit G5 is configured to determine the degree or level of risk based on the presence or absence of anomalous sensors and their usage in sensor fusion processing. Here, as an example, the degree of risk is represented by three levels from 0 to 2. Risk level 0 corresponds to a non-hazardous state. Risk level 1 corresponds to a state where control is performed more safely than usual. More safe control refers to actions such as increasing the safety distance or transferring operational authority to the user. Risk level 2 indicates a state where the vehicle should stop.

[0100] For example, such as Figure 4As shown, when no abnormal sensors are present, the risk level is determined to be 0. Furthermore, as... Figure 5 As shown, even if an abnormal sensor exists, if the usage level of the abnormal sensor is less than the predetermined risk determination threshold, the risk level is determined to be 1. Figure 6 As shown, when an abnormal sensor is present and the degree of use of the abnormal sensor is equal to or higher than the risk determination threshold, the risk level is determined to be 2. Figures 4 to 6 Sensor A, as shown, corresponds to, for example, a front-facing camera device 11A or a map linking unit F1. Sensor B is, for example, a millimeter-wave radar 11B, and sensor C is, for example, a LiDAR 11C. Sensor D is, for example, a sonar 11D. The risk determination threshold can be, for example, 10%, 25%, etc.

[0101] The utilization level of each surrounding monitoring sensor 11 can be, for example, the weight of the sensor itself in sensor fusion. Furthermore, when the weighting coefficients are different for each object, the utilization level of each surrounding monitoring sensor 11 can be the average or maximum weight for each object. Additionally, the utilization level of the surrounding monitoring sensor 11 can be the ratio of the number of times the identification results of the target surrounding monitoring sensor 11 used for calculation are used in the sensor fusion process within a certain time period. That is, the utilization level of each surrounding monitoring sensor 11 can be a value within a certain time period. The utilization level of each surrounding monitoring sensor 11 in sensor fusion can be, for example, the ratio obtained by dividing the number of objects in the detection results of the target surrounding monitoring sensor 11 used for sensor fusion by the total number of objects detected within a predetermined distance from the vehicle. The surrounding monitoring sensor 11 used in sensor fusion can be, for example, a sensor with a weight of 25% or more among other sensors. If the weight of the target surrounding monitoring sensor 11 used for calculation is not 0, then the sensor can be considered used for sensor fusion and its utilization level can be calculated. Of course, weights can be considered when calculating the utilization level.

[0102] Furthermore, when it is determined in the individual diagnostic process described later that the fusion unit F2 is not operating normally, the risk determination unit G5 sets the risk level to 2. Conversely, when the fusion unit F2 is abnormal, the risk level can be 1. Additionally, if it is determined that the map linking unit F1 is not operating normally, the risk level can be set to 1.

[0103] Risk Response Unit G6 is configured to request Safety Assessment Unit F4 to perform vehicle control based on a non-zero risk level determination. Risk Response Unit G6 requests Safety Assessment Unit F4 to perform a predetermined safety action based on a risk level determination of 1. The safety action could be, for example, vehicle control that brings the vehicle to a slow stop over a period of more than 10 seconds to 1 minute. Furthermore, the safety action could be a handover request processing. Handover request processing corresponds to a request made to the driver's seat occupant or operator to take over driving operations in conjunction with an HMI (Human Machine Interface) system such as a display. The handover request processing can be referred to as a handover request. A safety action could be increasing the safe distance used to calculate potential accident liability values. For example, a safety action could be, for instance, reducing the vehicle's speed from a planned target speed by a predetermined amount. A safety action corresponds to actions used to shift vehicle behavior to a safer level than usual while maintaining driving.

[0104] Note that risk level 1 occurs when the surrounding surveillance sensor 11 used in sensor fusion at a rate equal to or higher than a predetermined value exhibits an anomaly. In other words, the above situation can be rewritten and corresponds to the following configuration: in this configuration, processing for performing safety actions is triggered / invoked based on the anomaly caused by the surrounding surveillance sensor 11 used in sensor fusion at a rate greater than or equal to a predetermined threshold.

[0105] Furthermore, the risk response unit G6 requests the safety assessment unit F4 to execute a predetermined emergency action based on a risk level of 2. The emergency action can be, for example, the MRM (Minimum Risk Maneuver) described below. The specific content of the MRM can be, for example, a process that autonomously drives the vehicle to a safe location and alerts the surrounding area while parking the vehicle. Safe locations include areas with shoulders of equal or greater width than a predetermined value, designated emergency evacuation zones, etc. The content of the MRM can be to bring the vehicle to a gentle deceleration stop in the lane the vehicle is currently traveling in. Preferably, a deceleration of 4 [m / s^2] or less, such as 2 [m / s^2] or 3 [m / s^2], is used for this maneuver. Of course, when it is necessary to avoid a collision with a vehicle ahead, a deceleration exceeding 4 [m / s^2] can be used. Taking into account the vehicle speed at the start of the MRM and the distance between the vehicle and the following vehicle within a range where the vehicle can stop within, for example, 10 seconds, the deceleration at the time of the MRM can be dynamically determined and repeated and / or intermittently updated. Activating the MRM corresponds to initiating deceleration for an emergency stop.

[0106] The above configuration is based on a risk level of 2 to perform emergency vehicle stop procedures. For example, the risk response unit G6 outputs a signal requesting the safety assessment unit F4 to execute MRM. The risk response unit G6 can instruct the control planning unit F3 to generate a plan for implementing MRM as an emergency action. The signal requesting emergency action can be output to the control planning unit F3. The risk response unit G6 corresponds to the vehicle stop processing unit that performs the procedure to stop the vehicle (Ma).

[0107] <Details of the operation of the anomaly detection unit G4>

[0108] Here, will be used Figure 7 The flowchart shown describes a sensor diagnostic process, which determines whether each of a plurality of ambient monitoring sensors 11 is operating normally or in an abnormal state. The sensor diagnostic process is performed repeatedly and / or intermittently at predetermined intervals, for example, when system power for vehicle operation is on or when an autonomous driving function is enabled. The interval for repeating this process can be, for example, 200 milliseconds. Note that the system power mentioned herein refers to the power source for vehicle operation, and when the vehicle is a gasoline engine vehicle, system power refers to the ignition power source. When the vehicle is an electric or hybrid vehicle, system power refers to the system main relay. The sensor diagnostic process of this embodiment includes steps S101 to S105 as examples. Sensor diagnostic processing is performed on each of the ambient monitoring sensors 11. Hereinafter, the ambient monitoring sensor 11 undergoing diagnostic processing may also be designated as the diagnostic target sensor.

[0109] In step S101, the anomaly detection unit G4 sets (i.e. selects) any object detected by the diagnostic target sensor as the target object to be used in subsequent processing, and proceeds to step S102. In step S102, the anomaly detection unit G4 accesses the recognition result holding unit G2 and acquires the history (in other words, time-series data) of the diagnostic target sensor's recognition results for the target object within the most recent predetermined time (e.g., the last few minutes). The time-series data of the recognition results shows the transition of the correct recognition probability data for each type of target object. For example, the time-series data of the recognition results shows the transition of the probability that the target object is a vehicle and the transition of the probability that the target object is a pedestrian. The sampling period that defines the reference range of past recognition results can be, for example, 4 seconds, 6 seconds, etc.

[0110] In step S103, the anomaly detection unit G4 determines whether the identification result for the target object is stable based on the time-series data of the identification result obtained in step S102. For example, when the identification result for the target object is as follows: Figure 8 The stability shown is ( Figure 7If the condition is "yes" in step S103, the process proceeds to step S104. In step S104, it is determined that the target sensor is likely functioning correctly, and the process ends. On the other hand, if... Figure 9 As shown, when the recognition result for the target object fluctuates greatly, that is, when it is unstable (no in step S103), the process proceeds to step S105, and it is determined that the diagnostic target sensor may have an anomaly caused by this. Figure 8 and Figure 9 The vertical axis represents the probability value, and the horizontal axis represents time. A single-dash line represents the transition of the probability value p1, for example, the target object type being a vehicle, while a double-dash line represents the transition of the probability value p2, for example, the target object type being a pedestrian. The larger the amplitude of p1 and p2, the more frequent the switching between high and low values ​​of p1 and p2, and the more unstable the recognition result. The threshold for the amplitude of the probability value for each type and the threshold for the high-low switching frequency used to determine the probability of abnormal operations can be designed appropriately / arbitrarily.

[0111] The anomaly detection unit G4 performs the above-described processing on all objects detected by the diagnostic target sensor and, based on the number of anomaly determinations (which is the number of times an anomaly is likely to have occurred), ultimately / deterministically determines whether the diagnostic target sensor is operating normally. For example, when the number of anomaly determinations is equal to or greater than a predetermined threshold, it is determined that an anomaly has occurred in the diagnostic target sensor. Note that the anomaly detection unit G4 may determine that the diagnostic target sensor has an anomaly caused by this when, for example, the ratio of two counts—the anomaly determination count to the total determination count (i.e., the sum of the anomaly determination count and the normal operation determination count, which counts the number of determinations that the sensor is likely to be operating normally)—is equal to or greater than a predetermined threshold. Furthermore, for example, the anomaly detection unit G4 may be configured to perform the above-described diagnostic processing not on all objects detected by the diagnostic target sensor but on multiple objects randomly or selected based on predetermined rules. Furthermore, the anomaly detection unit G4 may perform the above-described diagnostic processing only on one object randomly or selected based on predetermined rules and determine whether the diagnostic target sensor is operating normally. However, if only one object exists as diagnostic material, there is a risk of incorrectly determining that the surrounding monitoring sensor 11, which is actually operating normally, is not operating normally due to, for example, the characteristics of the object. Therefore, preferably, the anomaly detection unit G4 uses multiple objects to perform statistical processing on the diagnostic results to determine whether the diagnostic target sensor is normal.

[0112] Note that the anomaly detection unit G4 can determine that the surrounding monitoring sensor 11 that outputs an error signal has an anomaly caused by this. In other words, the surrounding monitoring sensor 11 that outputs an error signal can be regarded as an anomaly sensor.

[0113] In addition, the anomaly detection unit G4 also performs operations on the map linking unit F1 and the fusion unit F2. Figure 7 The sensor diagnostic process shown is the same. Therefore, from the viewpoint of the stability of the identification results, both the map linking unit F1 and the fusion unit F2 are subject to diagnostics regarding whether each of these two units is operating normally. The criteria for determining abnormal operation can be the same as those for determining abnormal operation of the surrounding monitoring sensor 11. In the following text, the various components constituting the field identification system, such as the surrounding monitoring sensor 11, map linking unit F1, fusion unit F2, etc., will be referred to as identification modules.

[0114] <Operation of Risk Assessment Unit G5>

[0115] Reference Figure 10 The flowchart shown describes the operation of the diagnostic device 30 based on the operating state of each surrounding monitoring sensor 11. For example, it repeats and / or intermittently at predetermined cycles when power is on for vehicle operation or when autonomous driving functions are enabled. Figure 10 The flowchart shown.

[0116] First, in step S201, diagnostic results are obtained for each of the identification modules, and processing proceeds to step S202. As described above, the identification module includes each of the surrounding monitoring sensor 11, the map linking unit F1, and the fusion unit F2. Note that the above-described sensor diagnostic processing can be performed individually for each of the identification modules as step S201. Step S201 corresponds to the anomaly detection step.

[0117] In step S202, it is determined whether an abnormal sensor exists. If no abnormal sensor exists, a negative determination is made in step S202, and the process proceeds to step S203. If an abnormal sensor exists, a positive determination is made in step S202, and the process proceeds to step S209.

[0118] In step S203, the anomaly detection unit G4 reads whether the fusion unit F2 is determined to be normal. If the anomaly detection unit G4 has determined that the fusion unit F2 is normal, a positive determination is made in step S203, and the process proceeds to step S204. On the other hand, if the anomaly detection unit G4 detects an anomaly in the fusion unit F2, a negative determination is made in step S203, and the process proceeds to step S210.

[0119] In step S204, the anomaly detection unit G4 reads whether the map link unit F1 has been determined to be normal. If the anomaly detection unit G4 has determined that the map link unit F1 is normal, a positive determination is made in step S204, and the process proceeds to step S205. On the other hand, when the anomaly detection unit G4 has detected an anomaly in the map link unit F1, a negative determination is made in step S204, and the process proceeds to step S207.

[0120] In step S205, the risk level is set to 0, and the process proceeds to step S206. In step S206, a signal indicating that no abnormality has occurred in the field identification system is output to the safety assessment unit F4. In this way, the autonomous driving is maintained under normal control.

[0121] In step S207, the risk level is set to 1, and the process proceeds to step S208. In step S208, a signal is output requesting the safety assessment unit F4 to perform a safety action. In this way, a safety action such as increasing the safe distance is performed.

[0122] In step S209, it is determined whether the usage level (in other words, the dependence level) of the abnormal sensor is equal to or higher than a predetermined risk determination threshold. If the usage level of the abnormal sensor is equal to or higher than the risk determination threshold, a positive determination is made in step S209, and the process proceeds to step S210. On the other hand, if the usage level of the abnormal sensor is less than the risk determination threshold, a negative determination is made in step S209, and the process proceeds to step S207.

[0123] In step S210, the risk level is set to 2 and the process proceeds to step S211. In step S211, a signal is output requesting the safety assessment unit F4 to perform an emergency action. In this way, an emergency action such as MRM is performed. Since MRM is a control used to safely stop the vehicle, step S211 corresponds to the stop processing step.

[0124] Based on the above configuration, when an anomaly occurs in the on-site recognition system, safety or emergency actions will be taken. In this way, the likelihood of the vehicle continuing to operate autonomously when the calculation of potential accident liability values ​​may be incorrect can be reduced. This improves safety during autonomous driving.

[0125] Note that when the risk level is 1, the diagnostic device 30 can continue normal control without performing safety actions. This is because, at risk level 1, the impact of abnormal sensor readings on the driving environment is minimal, and control plans based on these observations are generally feasible. Furthermore, on the other hand, when the risk level is 1, the diagnostic device 30 can be configured to perform emergency actions similar to those in a risk level 2 situation.

[0126] <Supplement to Anomaly Detection Unit G4>

[0127] Various methods can be used for anomaly detection, such as watchdog timer methods and task response methods. The watchdog timer method means that if the watchdog pulse input from the monitored device before timeout fails to clear the watchdog timer of the monitoring device, it is determined that the monitored device is not operating normally. Here, the anomaly detection unit G4 corresponds to the configuration on the monitoring side, and each of the surrounding monitoring sensors 11, map linking unit F1, fusion unit F2, etc., corresponds to the monitored device.

[0128] <Variant of Ambient Surveillance Sensor 11>

[0129] In addition to a camera unit located at the upper end of the windshield inside the vehicle for capturing relatively long-range (i.e., distant) images, and separate from this camera unit, the vehicle control system 1 may also have a wide-angle short-range camera unit for on-site monitoring as a front-facing camera unit 11A. That is, the front-facing camera unit 11A can be configured with multiple cameras having different viewing angles. For example, three types of camera units can be configured: a mid-range camera unit, a long-range camera unit, and a wide-angle camera unit. A mid-range camera unit is a camera unit with a viewing angle of approximately 50 degrees and equipped with lenses configured to capture images, for example, up to 150 meters. A long-range camera unit is a camera unit with a relatively narrow viewing angle for capturing distant images farther than those of a mid-range camera unit. For example, a long-range camera unit has a viewing angle of approximately 30 to 40 degrees and is configured to capture distances of 250 meters or greater. A wide-angle camera unit is a camera unit configured to capture wide images of the vehicle's surrounding environment. The wide-angle camera has a field of view of, for example, about 120 to 150 degrees, and is configured to capture images within 50 meters in front of the vehicle.

[0130] Furthermore, the vehicle control system 1 may include a rear-view camera, a right-side camera, a left-side camera, a right rear radar, and a left rear radar as ambient monitoring sensors 11. The rear-view camera is a camera that captures a rear view of the vehicle from a predetermined angle. The rear-view camera is positioned at a predetermined location on the rear of the vehicle body, for example, near the rear license plate or rear window. The right-side and left-side cameras are cameras that capture a side view of the vehicle from a predetermined angle and are positioned at predetermined locations on the left and right sides of the vehicle body or rearview mirrors (e.g., near the base of the A-pillar). The horizontal viewing angle of the right-side and left-side cameras can be set to 180°, and the vertical viewing angle of these cameras is set to 120°. The right rear radar is a millimeter-wave radar whose detection range is a predetermined range on the right rear of the vehicle by emitting a probe wave towards the right rear of the vehicle, and this right rear radar is mounted, for example, at the right corner of the rear bumper. The left rear radar is a millimeter-wave radar, whose detection range is a predetermined range on the left rear of the vehicle by emitting a probe wave toward the left rear of the vehicle, and the left rear radar is installed, for example, at the left corner of the rear bumper.

[0131] Furthermore, the location information of obstacles and other vehicles acquired through road-to-vehicle communication and / or vehicle-to-vehicle communication also corresponds to information indicating the environment surrounding the vehicle. Therefore, the V2X on-board unit 13 can also be regarded as one of the surrounding monitoring sensors 11.

[0132] <Operation of the Scene Identification System>

[0133] The fusion unit F2 determines or detects the position, speed, type, etc. of each of the detected objects by integrating the identification results of each of the surrounding monitoring sensors 11 with the probability values ​​and weights of the surrounding monitoring sensors 11. However, depending on the detection method / principle, the surrounding monitoring sensors 11 may have the following characteristics: some types of objects can only be detected or identified with low accuracy or even not detected at all. In view of this situation, it may be preferable to correct and use the correct identification probability of the identification results based on the characteristics of various surrounding monitoring sensors 11, as well as as well-suited or unsuited objects. For example, the field identification system may be configured as follows.

[0134] like Figure 11 As shown, each ambient monitoring sensor 11 has a sensing unit B1, an identification processing unit B2, a sensor setting storage unit B3, a correction information output unit B4, a correction information acquisition unit B5, a probability value correction unit B6, and a detection result output unit B7. Although each configuration will be described here using a front-facing camera device 11A as an example, ambient monitoring sensors other than the front-facing camera device 11A, such as millimeter-wave radar 11B and LiDAR 11C, also have the same functional units. Figure 11Map link unit F1 and other configurations are not shown in the image.

[0135] Sensing unit B1 is configured to generate observation data for object recognition processing and output it to recognition processing unit B2. Sensing unit B1 includes sensor elements for generating the observation data. For example, the sensor element for the front-facing camera device 11A is an image sensor. For millimeter-wave radar 11B and sonar 11D, a configuration including circuitry for receiving radio waves or ultrasonic waves as probe waves corresponds to sensing unit B1. In LiDAR 11C, a configuration including a light-receiving element for receiving reflected waves from an illuminating laser corresponds to sensing unit B1.

[0136] The recognition processing unit B2 is configured to recognize the position, speed, type, etc. of a detected object based on observation data input from the sensing unit B1. The type is determined based on feature values ​​shown in the observation data. HOG (Histogram of Oriented Gradients) features, etc., can be used as feature values ​​for image data. Furthermore, in configurations using radio waves, ultrasound, radar light, etc., as the detection medium, for example, the correlation between the object's detection distance and received intensity, height, width, contour shape, number of detection points, density of detection points, etc., can be used as feature values. The recognition result of the recognition processing unit B2 is output as a signal indicating the position information, type, etc., for each detected object. As described above, the recognition result for the type of the detected object includes correct recognition probability data indicating the deterministic nature of the recognition result. Furthermore, it includes correct recognition probability data for each type of object. For example, the correct recognition probability data indicates the probability value that the detected object is a vehicle, the probability value that the detected object is a pedestrian, the probability value that the detected object is a cyclist, and the probability value that the detected object is a three-dimensional structure such as a sign.

[0137] The sensor setting storage unit B3 is a storage medium that stores sensor position data indicating the installation position and detection direction of each ambient monitoring sensor 11. Furthermore, the sensor setting storage unit B3 registers sensor characteristic data, which are data indicating the characteristics of each ambient monitoring sensor 11. Such a sensor setting storage unit B3 can also be referred to as a sensor characteristic storage unit. The sensor characteristic data indicates that, due to the characteristics of the detection principle of this sensor, each of the ambient monitoring sensors has certain limitations / constraints, such as (a) objects that this sensor is difficult to detect (i.e., objects that the object sensor is not good at detecting), and (b) situations where the object sensor's detection performance may deteriorate. It should be noted that objects that are not good at detecting include objects that are easily mistaken for (i.e., falsely detected) other types of objects and objects whose detection results are unstable.

[0138] For example, Sonar 11D is not good at detecting objects that do not easily reflect ultrasonic waves, such as sponge-like objects and mesh-like objects. Additionally, floating structures, such as beams, positioned at a predetermined distance above the road surface, and low-profile solid objects, such as curbs and wedges, set on the road surface, may be identified or not, depending on the vehicle's attitude / position. In other words, for Sonar 11D, floating structures and low-profile solid objects correspond to target objects whose detection results are unstable. Note that a floating structure refers to, for example, a three-dimensional object positioned 1m or more above the road surface. A low-profile solid object refers to a three-dimensional object whose height is less than a predetermined threshold (e.g., 0.2m). Furthermore, Sonar 11D may incorrectly identify reflections from the road surface as three-dimensional objects, such as another vehicle or a wall. That is, sponge-like objects, mesh-like objects, floating structures, low-profile solid objects, and road surfaces correspond to objects that Sonar 11D is not good at detecting.

[0139] Furthermore, when an image is overexposed due to backlighting, beams, or trailing, the front-facing camera 11A may be unable to recognize a portion of the image frame. In other words, the situation where a portion of the image is overexposed due to backlighting, beams, or trailing corresponds to a situation where it is not well-suited for imaging.

[0140] LiDAR 11C is not good at detecting black objects. This is because black objects absorb the laser beam. Additionally, LiDAR 11C has difficulty detecting vehicles with a silver body. A silver body has the characteristic of diffuse reflection of light due to the aluminum flakes contained within the coating. Such a vehicle is easily detected when it is directly in front of the camera, but when it is at an angle—that is, diagonally in front or behind—the reflection of light changes, making a silver vehicle at a diagonal angle potentially an object that LiDAR 11C is not adept at detecting.

[0141] Furthermore, the millimeter-wave radar 11B has a lower spatial resolution than the front-facing camera 11A and LiDAR 11C. Therefore, when multiple objects are close to each other or overlap, the millimeter-wave radar 11B has difficulty detecting them as individual objects. In other words, situations where multiple objects are close to or overlap correspond to areas where the millimeter-wave radar 11B is not adept. In some cases, the millimeter-wave radar 11B may identify a single object with a complex shape as two or more objects. That is, objects with complex shapes are not adept at detection by the millimeter-wave radar 11B. Additionally, regarding the millimeter-wave radar 11B, the detection wave passes beneath the vehicle in front, which may result in vehicles in front of the vehicle being either identified or not identified. In other words, vehicles in front of the vehicle in front correspond to objects that the millimeter-wave radar 11B is not adept at detecting. Furthermore, the millimeter-wave radar 11B may incorrectly detect manholes as three-dimensional objects. Therefore, manholes can also be considered objects that the millimeter-wave radar 11B is not adept at detecting.

[0142] Furthermore, in any of the ambient monitoring sensors 11, when an object is present near the sensing unit B1, there may be situations where objects behind the nearby object cannot be identified due to occlusion by the nearby object. In other words, in any of the ambient monitoring sensors 11, the presence of an object near the mounting location of the sensing unit B1 may correspond to a less desirable situation. Note that the term "nearby" in this disclosure refers to a range of, for example, 0.3m or less.

[0143] Based on the output signal of the recognition processing unit B2 and the data stored in the sensor setting storage unit B3, the correction information output unit B4 determines whether it is detecting an object or situation that is not well detected by other sensors of the surrounding monitoring sensor 11. Then, when an object / situation that is not well detected by other sensors is detected, the correction information output unit B4 sends a predetermined probability value correction information to the other sensors, i.e., the sensors that are not well detected.

[0144] Probability value correction information is used to notify other sensors of the presence of an object or situations that other sensors are not good at detecting. Probability value correction information includes, for example, information indicating the type and location of the undetectable object or situation. Note that probability value correction information may include the direction in which the undetectable object or situation exists, but not its location. Alternatively, probability value correction information may include both the location and direction of the undetectable object or situation.

[0145] For example, when the correction information output unit B4 of the front-facing camera device 11A identifies at least one of the following areas through image analysis: a sponge-like object, a mesh-like object, a floating structure, a low-profile solid object, and a road surface area, the correction information output unit B4 outputs probability value correction information to the sonar 11D indicating the presence of these objects or their location or direction. Furthermore, when the correction information output unit B4 of the front-facing camera device 11A detects a black object through image analysis, the correction information output unit B4 outputs probability value correction information to the LiDAR 11C indicating the direction or location of the black object. Additionally, when the correction information output unit B4 of the front-facing camera device 11A detects a vehicle in front of the vehicle ahead through image analysis, the correction information output unit B4 outputs probability value correction information to the millimeter-wave radar 11B indicating the presence of the vehicle in front of the vehicle ahead or its direction or location.

[0146] Furthermore, when an object is detected near another sensor whose installation position differs from that of this sensor, each correction information output unit B4 generates probability value correction information indicating the location of the nearby object and outputs it to the other sensors. For example, when the right-side camera detects a bicycle near the right rear radar, the right-side camera outputs probability value correction information indicating this situation to the right rear radar. Additionally, when a camera is detecting the position of the sun or the position of light with a predetermined brightness or higher, probability value correction information indicating the position or direction of this light source is output to the camera affected by this light source.

[0147] The correction information acquisition unit B5 acquires probability value correction information addressed to this sensor from other sensors and provides it to the probability value correction unit B6. For example, the correction information acquisition unit B5 of the front camera device 11A acquires the direction of the sun, headlights, etc., as probability value correction information from other cameras that are other sensors. Furthermore, for example, the correction information acquisition unit B5 of the sonar 11D acquires probability value correction information indicating the area or direction of the presence of sponge-like objects, mesh objects, floating structures, low-profile solid objects, road surface areas, etc., from the front camera device 11A and LiDAR 11C, which are other sensors.

[0148] The probability value correction unit B6, based on the probability value correction information acquired by the correction information acquisition unit B5, reduces the correct recognition probability of an object relative to the location or orientation of an object or situation where it is not well-suited for recognition. For example, the front-facing camera device 11A reduces the correct recognition probability of areas where backlighting, trailing light, or beams of light may exist. Furthermore, the sonar 11D reduces the correct recognition probability of areas where spongy objects or road surface areas are detected by other sensors. When the front-facing camera device 11A receives a notification of an area containing a black object as probability value correction information, the probability value correction unit B6 of the LiDAR 11C reduces the correct recognition probability of such areas.

[0149] Furthermore, the probability value correction unit B6 of the millimeter-wave radar 11B reduces the correct identification probability of the recognition results in areas where multiple objects detected by the front camera 11A and LiDAR 11C are close to or overlap each other. Additionally, for example, when the front camera 11A, LiDAR 11C, and / or V2X vehicle-mounted device 13 notify that a vehicle exists in front of the preceding vehicle, the probability value correction unit B6 of the millimeter-wave radar 11B reduces the correct identification probability of the recognition results of targets on the extension line of the preceding vehicle. In this way, the risk of fluctuations in the detection results in the area in front of the preceding vehicle can be reduced. On the other hand, the probability value correction unit B6 of the millimeter-wave radar 11B can increase the correct identification probability of the recognition results of targets on the extension line of the preceding vehicle when notified from the front camera 11A, etc., that a vehicle exists in front of the preceding vehicle. This also reduces the risk that the detection results of the millimeter-wave radar 11B will become unstable.

[0150] Note that the decrease or increase in the correct recognition probability via probability value correction unit B6 can be a constant value. For example, the decrease or increase in the correct recognition probability can be 10% or 20%. Furthermore, the decrease in the correct recognition probability can be varied depending on the type of the output source of the probability value correction information. The decrease in the correct recognition probability via probability value correction unit B6 on the receiving side can be varied based on the correct recognition probability of the target object at the output source of the probability value correction information. The decrease can be increased when probability value correction information with the same content is acquired from multiple other sensors. Preferably, the higher the reliability of the probability value correction information, the greater the decrease. If the content notified in the probability value correction information matches the recognition result of this sensor, the correct recognition probability of the target area can be improved. For recognition results of positions or directions for which probability value correction information has not been acquired, probability value correction unit B6 uses the output of recognition processing unit B2 as is.

[0151] The detection result output unit B7 outputs the generated recognition results (i.e., the recognition results of this sensor) and data indicating the correct recognition probability to the map linking unit F1 or the fusion unit F2. The fusion unit F2 integrates the recognition results from the surrounding monitoring sensors 11 to ultimately / determine the type and location, movement speed, acceleration, etc., of the detected object. This is used to calculate the safe distance for each target as a detected object and for tracking processing.

[0152] According to the above configuration, when there is an object / situation where the surrounding monitoring sensor 11 is not well-suited, the result of the identification processing unit B2 of this sensor is not used as is, but is adjusted, that is, based on the probability value correction information from other sensors, the correct identification probability of the identification result is reduced by a predetermined amount. In this way, the correct identification probability of the integrated identification result can be improved. The aforementioned surrounding monitoring sensor 11 corresponds to the following configuration: it communicates with other sensors and reduces and outputs the correct identification probability of its own identification result based on the identification results of other sensors.

[0153] Incidentally, the above content discloses a mode in which the relevant surrounding monitoring sensors 11 communicate with each other, correct the correct identification probability through the surrounding monitoring sensors 11, and output the corrected probability to the fusion unit F2, etc. The concept of this disclosure is not limited to this form.

[0154] like Figure 12 As shown, each ambient monitoring sensor 11 need not necessarily include a correction information acquisition unit F5 and a probability value correction unit B6. Instead, the fusion unit F2 includes the correction information acquisition unit F5 and the probability value correction unit B6. Figure 12 In the configuration example shown, each ambient monitoring sensor 11 notifies the fusion unit F2 of the identification result and the correct identification probability of its own sensor. Furthermore, when each ambient monitoring sensor 11 identifies an object or situation where other sensors are not well-suited, it outputs probability value correction information, including information from the other sensors, to the fusion unit F2.

[0155] The probability value correction information here includes identification information from other target sensors (e.g., sensor ID), the location or orientation of objects or situations where the sensor is not well-suited, and their type. The probability value correction unit B6 included in the fusion unit F2 corrects the correct identification probability of each sensor's identification result using the probability value correction information. According to this configuration, after correcting the correct identification probability of each sensor's identification result using the probability value correction information, the fusion unit F2 integrates the detection results from each surrounding monitoring sensor 11, thereby increasing the correct identification probability of the integrated identification result.

[0156] For example, when the fusion unit F2 integrates the identification results of sonar 11D and the identification results of the rear camera device, if the rear camera device detects a short (i.e., low-profile) object, such as a curb or wheel wedge, the sonar 11D gives a lower weight to the identification results of the corresponding, i.e., same location, in the information integration. In this way, when the sonar 11D identifies that there is no three-dimensional object in the corresponding area, the risk of degradation / tampering of the integrated identification results due to the influence of the sonar 11D's identification results can be reduced. Furthermore, for example, when another sensor with a different installation position detects an object in front of a certain surrounding monitoring sensor, the probability of correct identification by the surrounding monitoring sensor 11 near the object is reduced. In this way, the number of incorrect misidentifications in the integrated identification results can be reduced. In addition, when the fusion unit F2 determines that a predetermined amount or more of snowfall has been detected by the image sensor or LiDAR, it can be assumed that there is a possibility of snow adhering to the sonar, and the weight of the identification results through the sonar can be reduced to integrate the information.

[0157] The above configuration corresponds to the following configuration: when the first sensor, acting as an arbitrary ambient monitoring sensor, detects an object / situation that the second sensor, acting as another sensor, is not good at detecting, the identification result of the second sensor is determined to have low reliability based on the detection result. Furthermore, the fusion unit F2, which integrates the sensing information, corresponds to a configuration that reduces the weight of the identification result of the second sensor to integrate the sensing information. That is, the fusion unit F2 corresponds to a configuration that integrates the identification results of multiple ambient monitoring sensors 11, wherein information integration is performed based on the identification result of one sensor by reducing / adjusting the weights of the identification results of other sensors. In the configuration that determines the weights for each sensor in sensor fusion based on the correct identification probability of the identification result, correcting the correct identification probability corresponds to correcting the weights for each sensor when performing sensor fusion. The second sensor corresponds to the sensor that is not good at detection.

[0158] The above configuration corresponds to the following configuration: when a first sensor, acting as any ambient monitoring sensor, detects an object / situation that a second sensor, acting as another sensor, is not good at detecting, the correct identification probability of the second sensor's identification result is reduced based on the detection result. However, while there are objects / situations that the second sensor is not good at detecting, there are also objects / situations that the second sensor is good at detecting. An object that a certain ambient monitoring sensor is good at detecting is an easily detectable object. A situation that a certain ambient monitoring sensor is good at detecting means that, in this case, the detection accuracy of the detected object will not deteriorate. When the first sensor detects an object / situation that the second sensor is good at detecting, it can be configured to increase the correct identification probability of the second sensor's identification result based on the detection result. Even with this configuration, the correct identification probability of the identification result after integration by the fusion unit F2 can be improved. When the correct identification probability of the second sensor's identification result is increased based on the identification result of the first sensor, probability value correction information can be sent and received between sensors or between the sensors and the fusion unit F2. It should be noted that those skilled in the art will understand the method for implementing the configuration based on the identification result of the first sensor to increase the correct identification probability of the second sensor's identification result based on the above disclosure, and thus the details of this configuration are omitted.

[0159] <Example of system configuration modification>

[0160] like Figure 13 As shown, a safety evaluator 40 corresponding to the safety evaluation unit F4 can be set independently of the automatic drive unit 20. The safety evaluator 40 is a module with the same functions as the safety evaluation unit F4 and is configured to determine the final control content based on the potential accident liability value of the control plan generated by the automatic drive unit 20. The aforementioned safety evaluation unit F4 and the aforementioned safety evaluator 40 can be switched to achieve the same function. The safety evaluator 40 is configured as a computer including a processing unit 41 and memory 42. The processing unit 41 is configured using a processor. The memory 42 includes RAM and flash memory.

[0161] (Second Implementation)

[0162] In addition to the scene identification system used by the control planning unit F3 to generate a control plan, the vehicle control system 1 may also include a scene identification system used by the safety evaluator 40 to calculate potential accident liability values. That is, the scene identification system can be configured as a dual-system / duplex system. In the following, reference will be made to the second embodiment. Figure 14 and Figure 15This describes the configuration corresponding to this technical concept. For convenience, the field identification system used to generate control plans will be referred to as the identification system for control plans. On the other hand, the field identification system used to generate identification results for calculating potential accident liability values ​​is described as the identification system for safety assessment (SE: safety assessment hereinafter). SE can also be described as SA (safety evaluation).

[0163] like Figure 14 As shown, the vehicle control system 1 of the second embodiment includes a second ambient monitoring sensor 17 in addition to the first ambient monitoring sensor 11 corresponding to the aforementioned ambient monitoring sensor 11. The first ambient monitoring sensor 11 is the aforementioned front-facing camera 11A, millimeter-wave radar 11B, LiDAR 11C, sonar 11D, etc. The first ambient monitoring sensor 11 is a sensor group used by the automatic drive unit 20, which includes a control planning unit F3. The map linking unit F1 and the fusion unit F2 included in the automatic drive unit 20 identify the driving environment based on the output signal of the first ambient monitoring sensor 11. The first ambient monitoring sensor 11, the map linking unit F1, and the fusion unit F2 correspond to the identification module constituting the identification system for control planning. The map linking unit F1 and the fusion unit F2 correspond to the first identification unit.

[0164] The second surrounding surveillance sensor 17 is, for example, a front-facing camera 17A, a millimeter-wave radar 17B, a LiDAR 17C, a sonar 17D, etc. The front-facing camera 17A is a sensor that is physically different from / separated from the front-facing camera 11A. This also applies to millimeter-wave radar 17B, LiDAR 17C, sonar 17D, etc.

[0165] For example, while the various first ambient monitoring sensors 11 are sensors mounted on the vehicle body, the second ambient monitoring sensor 17 can be a sensor assembly retrofitted to the roof of the vehicle. For example, the second ambient monitoring sensor 17 is configured as an assembly for autonomous driving that can be attached to and detached from the vehicle body. The positions of the first ambient monitoring sensor 11 and the second ambient monitoring sensor 17 can be interchanged. That is, the first ambient monitoring sensor 11 can be a detachable sensor attached to the roof of the vehicle, etc.

[0166] Furthermore, the vehicle control system 1 of the second embodiment includes an SE identification device 50 that identifies the driving environment based on the output signal of a second ambient monitoring sensor 17. The SE identification device 50 is configured as a computer including a processing unit 51 and memory 52. ​​The processing unit 51 is configured to utilize a processor such as a GPU, and the memory 52 includes RAM and flash memory.

[0167] like Figure 15As shown, similar to the automatic drive unit 20, the SE identification device 50 includes a map linking unit H1 and a fusion unit H2. The map linking unit H1 is configured to perform object identification based on the identification results from the front-facing camera 17A and map data. The map linking unit H1 can be configured to perform the same processing as the map linking unit F1 included in the automatic drive unit 20. The fusion unit H2 is configured to perform object identification based on all the identification results from the map linking unit H1, the millimeter-wave radar 17B, the LiDAR 17C, and the sonar 17D. The fusion unit H2 can also be configured to perform the same processing as the fusion unit F2 included in the automatic drive unit 20. The second ambient monitoring sensor 17, the map linking unit H1, and the fusion unit H2 correspond to the identification modules constituting the identification system for the SE. The map linking unit H1 and the fusion unit H2 correspond to the identification unit and the second identification unit.

[0168] In addition to the output signal of the first ambient monitoring sensor 11, the identification result of the map linking unit F1, and the identification result of the fusion unit F2, the diagnostic device 30 also receives inputs from various devices, such as the output signal of the second ambient monitoring sensor 17, the identification result of the map linking unit H1, and the identification result of the fusion unit H2. The anomaly detection unit G4 of the diagnostic device 30 also performs the aforementioned sensor diagnostic processing on each of the various second ambient monitoring sensors 17, map linking unit H1, and fusion unit H2. That is, sensor diagnostic processing is also performed on the identification modules constituting the identification system for the SE.

[0169] Furthermore, the anomaly detection unit G4 detects anomalies in either the fusion unit F2 for control planning or the fusion unit H2 for SE by comparing the identification results of the fusion unit F2 for control planning with the identification results of the fusion unit H2 for SE. For example, as Figure 16 As shown, when the identification results of fusion unit F2 used for control planning and fusion unit H2 used for SE are similar, both are determined to be normal. Figure 16 In the diagram, the vertical axis represents the probability value, and the horizontal axis represents time. Single-dash and double-dash lines indicate the identification results of fusion unit F2 for control planning, while dashed and dotted lines indicate the identification results of fusion unit H2 for SE. Specifically, single-dash lines indicate the transition of probability value p1f in fusion unit F2 for control planning, where the target object type is a vehicle, and double-dash lines indicate the transition of probability value p2f in fusion unit F2, where the target object type is a pedestrian. Dashed lines indicate the transition of probability value p1h in fusion unit H2 for SE, where the target object type is a vehicle, and dotted lines indicate the transition of probability value p2h in fusion unit H2, where the target object type is a pedestrian.

[0170] For example, the anomaly detection unit G4 calculates the similarity between the recognition results of a predetermined target object by comparing historical data of the probability values ​​of the recognition results using pattern matching, etc. Then, if the similarity of the recognition results is equal to or higher than a predetermined threshold (e.g., 75%), it is determined that both are operating normally. On the other hand, when both recognition results are lower than the predetermined threshold or differ from each other, the anomaly detection unit G4 determines that an anomaly has occurred in either the fusion unit F2 for control planning or the fusion unit H2 for SE. In this case, the result of sensor diagnostic processing is used to determine which one is abnormal. Determining that the recognition results are similar corresponds to determining that similar recognition results are consistent / matched, or interpreting similarity as consistency and normality.

[0171] Now, in Figure 17 In the figure, although the target object type is consistent / matched in the identification results from fusion unit F2 and fusion unit H2, their probability values ​​are significantly different from each other. Therefore, this pattern is identified as an anomaly of one of the fusion units F2 (for control planning) and H2 (for SE). Specifically, although both fusion unit F2 (for control planning) and H2 (for SE) determine the target object type as a vehicle, the pattern in the figure shows that their probability values ​​differ from each other by a predetermined threshold (e.g., 30%) or more. Furthermore, Figure 18 Another pattern is illustrated in the figure, which is used to identify anomalies in one of the fusion units F2 for control planning and H2 for SE, where the types of target objects in the identification results are different. Specifically, in the above, fusion unit F2 for control planning determines that the target object is a vehicle, while fusion unit H2 for SE determines that the target object is a pedestrian. Figure 17 and Figure 18 Items indicated by symbols and line types in the middle Figure 16 Those are the same as those in the text.

[0172] When the identification result of the fusion unit F2 used for control planning does not match the identification result of the fusion unit H2 used for SE, that is, when one of them is abnormal, the anomaly detection unit G4 determines which one is abnormal based on the result of sensor diagnostic processing.

[0173] The above discussion describes a process for detecting anomalies in either the fusion unit F2 for control planning or the fusion unit H2 for SE by comparing the identification results of the fusion unit F2 for control planning with the identification results of the fusion unit H2 for SE. Furthermore, the anomaly detection unit G4 performs the same comparison process on map linking units F1 and H1. That is, the anomaly detection unit G4 detects anomalies in either map linking unit F1 or H1 by comparing the identification results of the map linking unit F1 for control planning with the identification results of the map linking unit H1 for SE. In this way, it becomes possible to more accurately determine whether an anomaly has occurred in either map linking unit F1 or H1. Moreover, when the identification results of map linking unit F1 and map linking unit H1 for SE are inconsistent / mismatched, that is, when either of them is abnormal, the anomaly detection unit G4 determines which one is abnormal based on the results of the sensor diagnostic processing.

[0174] If an anomaly occurs in the identification system used for SE, the diagnostic device 30 can notify the safety evaluator 40 of this situation, while simultaneously performing a calculation of the potential accident liability value using the identification results from the fusion unit F2 used for the control plan. According to this configuration, the calculation of the potential accident liability value can continue until the vehicle comes to a stop. In this way, a control plan can be selected for stopping the vehicle in a mode other than full braking. By enabling the use of deceleration control modes other than full braking, the risk of tire slippage or lock-up can be reduced.

[0175] Furthermore, when an anomaly occurs in the identification system used for the control plan, the diagnostic device 30 notifies the automatic drive unit 20 and the safety evaluator 40 of this situation. Additionally, when an anomaly occurs in the identification system used for the control plan, the diagnostic device 30 can control the control plan unit F3 to generate a control plan for stopping the vehicle using the identification results from the fusion unit H2 used for the SE. With this configuration, the vehicle can be stopped more safely.

[0176] For example, if it is determined that the fusion unit H2 used for SE is not operating normally, the risk determination unit G5 of the second embodiment can determine the risk level as 1. If an anomaly occurs in the fusion unit H2, the risk level can be 2. Furthermore, even if it is determined that the fusion unit H2 used for SE is not operating normally, as long as the fusion unit F2 used for control planning is operating normally, the risk level can be considered 0. This is because, as long as the fusion unit F2 used for control planning is operating normally, the safety evaluator 40 can continue to calculate the potential accident liability value by utilizing the identification results of the fusion unit F2 used for control planning.

[0177] Furthermore, the risk determination unit G5 can be configured to determine a risk level of 2 if an anomaly occurs in both the fusion unit F2 for control planning and the fusion unit H2 for SE, and to determine a risk level of 1 or less if (i.e., provided) at least one of them is operating normally. Alternatively, the risk level can be determined to be 1 based on the determination that the map linking unit H1 is not operating normally. Alternatively, if the second ambient monitoring sensor 17 has an abnormal sensor and the usage level of the abnormal sensor is equal to or higher than a predetermined risk determination threshold, the risk level can be determined to be 1.

[0178] Based on the above configuration, the identification results of the two identification systems are compared, and then anomaly diagnosis is performed for each system. By performing anomaly diagnosis in each of the identification systems in this way, even if anomalies occur simultaneously in both, it becomes possible to determine that anomalies have occurred in both. Furthermore, when anomalies in one of the two identification systems are detected by comparing their identification results, it is possible to determine which one is anomalous based on the results of the sensor diagnostic processing. When the diagnostic device 30 detects an anomaly in the identification system used for control planning or for SE (Self-Controlled Detection), it is preferable that the diagnostic device 30 at least performs a safety action, such as deceleration or a handover request.

[0179] In addition, such as Figure 19 As shown, the vehicle control system 1 may have a third fusion unit X2, which identifies the surrounding environment by integrating sensing information from a portion of the first ambient monitoring sensor 11 and a portion of the second ambient monitoring sensor 17. For example, the third fusion unit X2 may be configured to integrate the identification results from the front-facing camera 11A and millimeter-wave radar 11B, which are the first ambient monitoring sensor 11, and the identification results from the LiDAR 17C and sonar 17D, which are the second ambient monitoring sensor 17. The third fusion unit X2 may be set as part of the automatic drive unit 20 or as part of the SE identification device 50. The third fusion unit X2 may be set by other ECUs, such as a driver support ECU, or may be built into the front-facing camera 11A. The identification results of the third fusion unit X2 are also input to the diagnostic device 30.

[0180] According to the above configuration, the diagnostic device 30 can detect abnormal operation of one of the three fusion units F2, H2, and X2 by comparing the identification results of the three fusion units. For example, the anomaly detection unit G4 can determine the fusion unit of the abnormal operation by using a concept such as majority voting. The third fusion unit X2 can be implemented by logically combining the first ambient monitoring sensor 11 and the second ambient monitoring sensor 17. Therefore, "triple" redundancy can be provided without adding a new ambient monitoring sensor 11 in terms of hardware.

[0181] <Additional Notes>

[0182] The control unit and methods described in this disclosure can also be implemented by a dedicated computer, which constitutes a processor programmed to perform one or more functions implemented by a computer program. Furthermore, the apparatus and methods described in this disclosure can also be implemented by dedicated hardware logic circuitry. Additionally, the apparatus and methods described in this disclosure can also be implemented by one or more dedicated computers, which consist of a processor for executing a computer program and a combination of one or more hardware logic circuitry. The computer program can be stored as instructions to be executed by a computer in a tangible, non-transitory, computer-readable storage medium. That is, the means and / or functions provided by the automatic drive device 20 and the diagnostic device 30, etc., can be provided as software, software only, hardware only, or a combination thereof, recorded in a physical storage device and a computer. For example, some or all of the functions included in the diagnostic device 30 can be implemented as hardware. A configuration for implementing a function as hardware includes a configuration for implementing that function using one or more ICs, etc. Processing units 21, 31, 41, 51 can be implemented by using an MPU or GPU instead of a CPU. Processing units 21, 31, 41, and 51 can be implemented by combining multiple types of arithmetic processing devices such as CPUs, MPUs, and GPUs. Processing units 21, 31, 41, and 51 can be implemented as a System-on-a-Chip (SoC). Furthermore, various processing units can be implemented using FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Various programs can be stored in non-transitional tangible storage media. Various storage media such as HDDs (Hard Disk Drives), SSDs (Solid State Drives), EPROMs (Erasable Programmable ROMs), flash memory, USB storage, and SD (Secure Digital Storage) cards can be used as storage media for programs.

Claims

1. A vehicle control device for generating a control plan for autonomous driving of a vehicle, the vehicle control device comprising: An anomaly detection unit detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: detecting the output signal of the at least one ambient monitoring sensor of an object within a predetermined detection range, and the identification unit identifying an object near the vehicle based on the output signal of the at least one ambient monitoring sensor. as well as The vehicle stop processing unit performs processing to stop the vehicle based on the anomaly detection of the scene recognition system by the anomaly detection unit. The vehicle has: A control planning unit is used to generate a control plan for the autonomous driving of the vehicle. as well as The security assessment unit is used to assess the security of the control plan generated by the control planning unit. The anomaly detection unit is configured to compare the processing of the field identification system used in the control planning unit with the processing of the field identification system used in the safety assessment unit. If the results are different, it is determined that an anomaly has occurred in the field identification system. The processing of the field identification system used in the safety assessment unit is the same as the processing of the field identification system used in the control planning unit. The processing of the field identification system used in the control planning unit and the processing of the field identification system used in the safety assessment unit are processes that fuse the identification results of multiple surrounding monitoring sensors.

2. A vehicle control device for generating a control plan for autonomous driving of a vehicle, the vehicle control device comprising: An anomaly detection unit detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: detecting the output signal of the at least one ambient monitoring sensor of an object within a predetermined detection range, and the identification unit identifying an object near the vehicle based on the output signal of the at least one ambient monitoring sensor. as well as The vehicle stop processing unit performs processing to stop the vehicle based on the anomaly detection of the scene recognition system by the anomaly detection unit. The vehicle has: A control planning unit is used to generate a control plan for the autonomous driving of the vehicle. as well as The security assessment unit is used to assess the security of the control plan generated by the control planning unit. The anomaly detection unit is configured to compare the processing of the field identification system used in the control planning unit with the processing of the field identification system used in the safety assessment unit. If the results are different, it is determined that an anomaly has occurred in the field identification system, wherein the processing of the field identification system used in the safety assessment unit is the same as the processing of the field identification system used in the control planning unit. The processing of the on-site identification system used in the control planning unit and the processing of the on-site identification system used in the safety assessment unit are processes for determining the location of the vehicle on the map.

3. A vehicle control device for generating a control plan for autonomous driving of a vehicle, the vehicle control device comprising: An anomaly detection unit detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: detecting the output signal of the at least one ambient monitoring sensor of an object within a predetermined detection range, and the identification unit identifying an object near the vehicle based on the output signal of the at least one ambient monitoring sensor. as well as The vehicle stop processing unit performs processing to stop the vehicle based on the anomaly detection of the scene recognition system by the anomaly detection unit. The vehicle has: A control planning unit is used to generate a control plan for the autonomous driving of the vehicle. as well as The security assessment unit is used to assess the security of the control plan generated by the control planning unit. The anomaly detection unit is configured to compare the processing of the field identification system used in the control planning unit with the processing of the field identification system used in the safety assessment unit. If the results are different, it is determined that an anomaly has occurred in the field identification system, wherein the processing of the field identification system used in the safety assessment unit is the same as the processing of the field identification system used in the control planning unit. The processing of the field identification system used in the control planning unit and the processing of the field identification system used in the safety assessment unit are processes that identify the environment surrounding the vehicle by combining the identification results of the surrounding monitoring sensors and map data.

4. The vehicle control device according to any one of claims 1 to 3, further comprising: A recognition result acquisition unit, which repeatedly and / or intermittently acquires the recognition results of the recognition unit; as well as The recognition result holding unit temporarily holds the recognition results acquired by the recognition result acquisition unit at multiple time points in association with the associated time, wherein... The anomaly detection unit By referring to the time-series data of the identification results of the identification unit for any of the objects detected by the surrounding monitoring sensors, which is held by the identification result holding unit, it is determined whether the identification results are stable, and Based on the identification result being determined to be an unstable object, it is determined that an anomaly has occurred in the on-site identification system.

5. The vehicle control device according to any one of claims 1 to 3, wherein, Multiple surrounding monitoring sensors are installed. The surrounding monitoring sensor outputs a signal indicating the identification results regarding the location and type of the detected object as the output signal, and The anomaly detection unit diagnoses the operational status of each of the multiple surrounding monitoring sensors based on the output signals of the multiple surrounding monitoring sensors.

6. The vehicle control device according to claim 5, wherein, The identification unit determines the type of detected object by weighting and integrating the identification results from multiple surrounding monitoring sensors. The vehicle stop processing unit performs the following process: stopping the vehicle based on the weight of the surrounding monitoring sensors that are determined to be abnormal by the anomaly detection unit being equal to or greater than a predetermined threshold.

7. The vehicle control device according to claim 5, wherein, The identification unit determines the type of the detected object by weighting and integrating the identification results from multiple surrounding monitoring sensors. A risk determination unit is configured to determine the risk level based on the weights of surrounding monitoring sensors identified as anomalous by the anomaly detection unit. The vehicle stop processing unit performs the following process: stops the vehicle based on the risk level being equal to or higher than a predetermined threshold.

8. The vehicle control device according to claim 5, wherein, The identification unit determines the type of the detected object by weighting and integrating the identification results from multiple surrounding monitoring sensors. The output of each of the plurality of surrounding monitoring sensors indicates the probability of correct identification of the type of the detected object, and The recognition unit changes the weight of each surrounding monitoring sensor when integrating the recognition results of multiple surrounding monitoring sensors based on the correct recognition probability and at least one of the driving scenarios.

9. The vehicle control device according to claim 5, wherein, While retaining information about objects that are difficult for other sensors to detect, each of the plurality of surrounding surveillance sensors, upon detecting an object that is difficult for the other sensors to detect, sends predetermined probability value correction information to the other sensors that are not good at detecting related objects. The other sensors are one of the surrounding surveillance sensors other than the current surrounding surveillance sensor. The probability value correction information includes information indicating the location or direction of the object that is difficult to detect. Upon receiving the probability value correction information, the surrounding monitoring sensor corrects the probability of correctly identifying the possible location or orientation of the unsuitable object based on the received probability value correction information.

10. The vehicle control device according to claim 9, further comprising: Camera devices and sonar, which are configured as ambient monitoring sensors, wherein, When the captured image is analyzed and at least one of the following is detected: a sponge-like object, a mesh-like structure, a floating structure floating at a predetermined distance above the road surface, or a low-profile solid object with a height less than a predetermined threshold, the camera device notifies the sonar of probability value correction information including the location or orientation of the detected object. The sonar reduces the probability of correctly identifying the location or direction indicated by the probability value correction information.

11. The vehicle control device according to claim 9, further comprising: Camera devices and lidar are configured as the surrounding surveillance sensors, wherein, When a black or silver object is detected by analyzing the captured image, the camera device notifies the lidar of probability value correction information, including the location or orientation of the detected object. The lidar reduces the probability of correctly identifying the position or direction indicated by the probability value correction information.

12. The vehicle control device according to claim 5, wherein, While maintaining information about the poor detection performance of other sensors on objects, each of the plurality of surrounding monitoring sensors, upon detecting a poor detection performance of the other sensor, sends predetermined probability value correction information to the other sensor that is the poor sensor in the relevant poor detection situation, wherein the other sensor is one of the surrounding monitoring sensors other than the current surrounding monitoring sensor, and the probability value correction information includes information indicating: (i) the type of the poor detection situation, and (ii) the direction in which the detection performance of the other sensor may be degraded due to the poor detection situation. Upon receiving the probability value correction information, the surrounding monitoring sensor corrects the correct recognition probability of the direction recognition result affected by the unfavorable situation based on the received probability value correction information.

13. The vehicle control device according to claim 5, wherein, While retaining information about objects that are difficult for other sensors to detect, each of the plurality of surrounding surveillance sensors sends information as predetermined probability value correction information to the identification unit upon detecting an object that is difficult for the other sensors to detect. Here, the other sensors are those other than the current surrounding surveillance sensor. The probability value correction information includes information indicating the type of the object being poorly trained and the location or direction in which the object exists, as well as... The identification unit uses the probability value correction information to correct the correct identification probability of the identification results of each surrounding monitoring sensor, and integrates the identification results of multiple surrounding monitoring sensors.

14. The vehicle control device according to any one of claims 1 to 3, comprising: At least one first ambient monitoring sensor corresponding to the ambient monitoring sensor; as well as At least one second ambient monitoring sensor is mounted at a different location than the first ambient monitoring sensor, and wherein, As the identification unit, a first identification unit is provided that performs driving environment identification processing based on the output signal of the first ambient monitoring sensor, and a second identification unit that performs driving environment identification processing based on the output signal of the second ambient monitoring sensor. The anomaly detection unit determines that an anomaly has occurred in the on-site identification system based on the mismatch between the identification results of the first identification unit and the identification results of the second identification unit.

15. The vehicle control device according to claim 14, wherein, The first ambient monitoring sensor is an ambient monitoring sensor mounted on the vehicle body, and The second ambient monitoring sensor is a movable ambient monitoring sensor that is retrofitted to the vehicle.

16. The vehicle control device according to any one of claims 1 to 3, wherein, The surrounding monitoring sensor is configured to include at least one of a camera device, millimeter-wave radar, lidar, and sonar.

17. A vehicle control method, wherein the vehicle control method is executed by at least one processor to generate a control plan for autonomous driving of a vehicle, the method comprising the steps of: Anomaly detection, which detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: (a) detecting the output signal of the at least one ambient monitoring sensor of an object within a predetermined detection range, and (b) the identification unit identifying the identification result of an object near the vehicle based on the output signal of the at least one ambient monitoring sensor; as well as The vehicle processing is stopped, which executes actions to stop the vehicle processing based on the detection of anomalies by the scene recognition system in the anomaly detection step. The vehicle has: A control planning unit is used to generate a control plan for the autonomous driving of the vehicle. as well as The security assessment unit is used to assess the security of the control plan generated by the control planning unit. In the anomaly detection step, the processing of the field identification system used in the control planning unit is compared with the processing of the field identification system used in the safety assessment unit. If the results are different, it is determined that an anomaly has occurred in the field identification system. The processing of the field identification system used in the safety assessment unit is the same as the processing of the field identification system used in the control planning unit. The processing of the field identification system used in the control planning unit and the processing of the field identification system used in the safety assessment unit are processes that fuse the identification results of multiple surrounding monitoring sensors.

18. A vehicle control method, the vehicle control method being executed by at least one processor to generate a control plan for autonomous driving of a vehicle, the method comprising the following steps: Anomaly detection, which detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: (a) detecting the output signal of the at least one ambient monitoring sensor of an object within a predetermined detection range, and (b) the identification unit identifying the identification result of an object near the vehicle based on the output signal of the at least one ambient monitoring sensor; as well as The vehicle processing is stopped, which executes actions to stop the vehicle processing based on the detection of anomalies by the scene recognition system in the anomaly detection step. The vehicle has: A control planning unit is used to generate a control plan for the autonomous driving of the vehicle. as well as The security assessment unit is used to assess the security of the control plan generated by the control planning unit. In the anomaly detection step, the processing of the field identification system used in the control planning unit is compared with the processing of the field identification system used in the safety assessment unit. If the results are different, it is determined that an anomaly has occurred in the field identification system. The processing of the field identification system used in the safety assessment unit is the same as the processing of the field identification system used in the control planning unit. The processing of the on-site identification system used in the control planning unit and the processing of the on-site identification system used in the safety assessment unit are processes for determining the location of the vehicle on the map.

19. A vehicle control method, said vehicle control method being executed by at least one processor to generate a control plan for autonomous driving of a vehicle, said method comprising the following steps: Anomaly detection, which detects anomalies in a field identification system comprising at least one ambient monitoring sensor and an identification unit based on at least one of the following: (a) detecting the output signal of the at least one ambient monitoring sensor of an object within a predetermined detection range, and (b) the identification unit identifying the identification result of an object near the vehicle based on the output signal of the at least one ambient monitoring sensor; as well as The vehicle processing is halted by executing a procedure to stop the vehicle processing based on the detection of anomalies in the scene identification system by the anomaly detection step. The vehicle has: A control planning unit is used to generate a control plan for the autonomous driving of the vehicle. as well as The security assessment unit is used to assess the security of the control plan generated by the control planning unit. The anomaly detection step compares the processing of the field identification system used in the control planning unit with the processing of the field identification system used in the safety assessment unit. If the results are different, it is determined that an anomaly has occurred in the field identification system. The processing of the field identification system used in the safety assessment unit is the same as the processing of the field identification system used in the control planning unit. The processing of the field identification system used in the control planning unit and the processing of the field identification system used in the safety assessment unit are processes that identify the environment surrounding the vehicle by combining the identification results of the surrounding monitoring sensors and map data.

20. A vehicle control device, comprising: processor; Computer-readable storage medium; Fusion unit; Map link unit; First sensor; as well as Second sensor, The vehicle control device is configured to: Obtain the recognition results for each sensor; Determine if any sensor is malfunctioning; When the following conditions are met, the risk level should be set to low and normal controls should be maintained: (i) No abnormal sensors are present. (ii) The fusion unit is normal, and (iii) The map link unit is normal; If the following conditions are met, the risk level will be set to intermediate and a safety action will be requested: (i) No abnormal sensors are present. (ii) The fusion unit is normal, and (iii) The map link unit is malfunctioning; When the following conditions are met, the risk level is set to the intermediate level and the safety action is requested: (i) There is at least one abnormal sensor, and (ii) The at least one abnormal sensor has not been heavily used; When the following conditions are met, the risk level is set to high and emergency action is requested: (i) There is at least one abnormal sensor, and (ii) The at least one anomaly sensor is used extensively; When the following conditions are met, the risk level is set to the high level and the emergency action is requested: (i) There are no abnormal sensors, and (ii) The fusion unit is malfunctioning.

Images