Vehicle recording device, driving status recording program

Abstract

To provide a vehicle recording device and an information recording method that enable analyticity of an accident cause to be increased while reducing a data storage amount.SOLUTION: An operation recording unit 30 includes an information acquisition unit G1, a recording target setting unit G2, and a recording processing unit G3. The information acquisition unit G1 acquires a plurality of types of information that may be a recording target from sensors mounted on a vehicle. The recording target setting unit G2 changes the recording target in accordance with a driving mode of an automatic driving ECU 20. The recording processing unit G3 sequentially stores, in a data storage unit Mm, data about an item relevant to the recording target set by the recording target setting unit G2. Since the item to be the recording target is changed in accordance with the driving mode, less useful data can be eliminated from the recording target in a cause analysis of an accident or the like. As a result, a data amount to be stored can be reduced. In addition, an item necessary for the cause analysis can be left as a recording target. Thus, analyticity of the accident cause can be improved.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a vehicle recording device and an information recording method for recording data indicating at least one of the conditions inside and outside a vehicle while the vehicle is traveling. [Background technology]

[0002] So far, as vehicle recording devices, devices that record images captured by an in-vehicle camera for a predetermined time before and after a predetermined event such as a collision or sudden braking have been proposed, as well as drive recorders that constantly record images from an in-vehicle camera. For example, Patent Document 1 discloses a configuration for a constant-recording drive recorder that associates and saves video data from a front camera, in-vehicle audio data, and driving mode information. The driving mode information in Patent Document 1 indicates whether the driving mode is automatic or manual. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-124823 A Summary of the Invention [Problem to be solved by the invention]

[0004] Vehicles equipped with autonomous driving functions are required to be configured so that the causes of accidents occurring during autonomous driving, and whether the responsibility for the accident lies with the system or the driver, can be objectively analyzed later. This has created a need to record sensor / ECU data that was not included in the recording targets with conventional technology. In addition, it is expected that vehicles equipped with autonomous driving functions will be equipped with even more sensors / ECUs. In other words, it is necessary to record more data than ever before. If conventional technology is directly diverted to meet such demand, the amount of data to be stored will be enormous.

[0005] The present disclosure has been made based on the above circumstances, and an object of the present disclosure is to provide a vehicle recording device that can reduce the amount of data stored while improving the analysis of the cause of an accident. , Operation status record program The aim is to provide 。 [Means for solving the problem]

[0006] A vehicle recording device for achieving this object is a computer that provides an automatic driving function, and includes an automatic driving processing unit (20) that has an occupant involvement mode corresponding to any one of automatic driving levels 0 to 2 as an operation mode, and an autonomous driving mode in which the vehicle is driven autonomously. The vehicle recording device records the operation state of an automatic driving system, and includes an information acquisition unit (G1) that acquires multiple types of information that can be recorded from sensors mounted on the vehicle, a communication unit (34) that acquires a signal indicating an operation mode of the automatic driving processing unit, and a processing unit (31) that executes a process of changing the recording target according to the operation mode, and a process of saving information corresponding to the recording target in a data storage unit (Mm) realized using a predetermined storage medium, and the processing unit is further configured to execute, in the case of an operation mode in which the recognition result of the automatic driving system is included in the recording target, setting information of an application that was operating in the automatic driving system and version information of the software that was being used as the recording target, and recording information of the application that was operating in the automatic driving system at the time when a predetermined event occurred and version information of the software that was being used in the data storage unit, and the event is a decrease in the automatic driving level based on the operation of an occupant in the driver's seat. ,luck A decrease in the autonomous driving level due to the exit of the row design area, execution of the handover request processing by the autonomous driving processing unit, and MRM (Minimum Risk Maneuver) Row At least one of them is included.

[0007] In the above configuration, By the recording target setting section Depending on the driving mode Note record pair Elephant will be changed. Therefore, the amount of data to be stored can be reduced. In addition, when in a specific operation mode, information on the applications that were running in the autonomous driving system and version information of the software that was being used are also stored. Because it is recorded ,caseThis can improve the analysis of the cause of the problem.

[0008] In addition, a driving status recording program for achieving the above object includes instructions to cause a computer used in an autonomous driving system having an occupant involvement mode corresponding to any one of autonomous driving levels 0 to 2 as an operation mode and an autonomous driving mode in which the vehicle is driven autonomously to execute the following: acquiring information on multiple types of elements that can be recorded from sensors mounted on the vehicle, acquiring a signal indicating the operation mode of the autonomous driving system, changing the recording target according to the operation mode, saving information on the elements set as the recording target in a data storage unit (Mm) realized using a predetermined storage medium, and recording information on an application that was operating in the autonomous driving system at the time a predetermined event occurred and version information of the software being used in the data storage unit, wherein the event is a decrease in the autonomous driving level based on the operation of an occupant in the driver's seat. ,luck A decrease in the autonomous driving level due to the exit of the row design area, execution of handover request processing by the autonomous driving system, and MRM (Minimum Risk Maneuver) Row At least one of them is included.

[0009] the above Through the program, This makes it possible to improve the analysis of the cause of accidents while reducing the amount of data stored.

[0010] In addition, the symbols in parentheses in the claims indicate a correspondence with the specific means described in the embodiments described below as one aspect, and do not limit the technical scope of the present disclosure. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of an automatic driving system 1. [Diagram 2] FIG. 2 is a diagram showing an example of a perimeter monitoring sensor 11. [Diagram 3] FIG. 2 is a diagram illustrating an example of an occupant status sensor 12. [Figure 4] FIG. 2 is a block diagram showing the configuration of a locator 14. [Diagram 5] FIG. 2 is a block diagram showing the configuration of an HMI system 15. [Figure 6] FIG. 13 is a diagram showing an example of a driving mode notification image 80. [Figure 7] FIG. 2 is a block diagram showing the configuration of an automatic driving ECU 20. [Figure 8] FIG. 2 is a block diagram showing the configuration of an operation recording unit 30. [Figure 9] FIG. 2 is a conceptual diagram for explaining the configuration of a data storage unit Mm. [Figure 10] FIG. 4 is a diagram showing an example of a correspondence relationship between an operation mode and items to be recorded. [Figure 11] 13 is a diagram showing a recording target period of the event recording process by the recording processing unit G3. FIG. [Figure 12] 4 is a flowchart showing the operation of the operation recording unit 30. [Figure 13] 13 is a diagram for explaining the mutual operation between the ODD inside / outside determination unit F51 and the operation recording unit 30. FIG. [Figure 14] FIG. 2 is a block diagram showing a modified example of the automatic driving ECU 20. [Figure 15] 11 is a diagram showing a modified example of the number of perimeter monitoring sensors 11 mounted. [Figure 16] 1 is a diagram showing an example of a correspondence relationship between a scheduled action in an autonomous driving mode and a perimeter monitoring sensor 11 that records observed data. FIG. [Figure 17] FIG. 2 is a diagram showing a modified example of the configuration of the autonomous driving system 1. [Figure 18] 13 is a block diagram showing a modified example of the configuration of the operation recording unit 30. FIG. [Figure 19] 4 is a diagram showing an example of a manner in which data is recorded by the operation recording unit 30. FIG. [Figure 20] 13 is a diagram showing another example of the manner in which data is recorded by the operation recording unit 30. FIG. [Figure 21] FIG. 13 is a diagram showing a modified example of the system configuration. [Figure 22] FIG. 13 is a diagram showing a modified example of the system configuration. [Diagram 23] FIG. 13 is a diagram showing a modified example of the system configuration. [Figure 24] FIG. 13 is a diagram showing a modified example of the system configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a schematic configuration of an autonomous driving system 1 to which the vehicle recording device of the present disclosure is applied. As shown in FIG. 1, the autonomous driving system 1 includes a perimeter monitoring sensor 11, an occupant status sensor 12, a vehicle status sensor 13, a locator 14, an HMI system 15, a V2X vehicle-mounted device 16, a body ECU 17, an autonomous driving ECU 20, and an operation recording unit 30. In addition, ECU in the names of the components is an abbreviation for Electronic Control Unit, which means an electronic control device. In addition, HMI is an abbreviation for Human Machine Interface. V2X is an abbreviation for Vehicle to X (Everything), which means a communication technology that connects a vehicle to various things. When simply described as a system, it refers to the autonomous driving system 1.

[0013] The various devices or sensors constituting the autonomous driving system 1 are connected as nodes to a communication network (hereinafter, LAN: Local Area Network) 99 constructed in the vehicle. These nodes connected to the LAN 99 can communicate with each other. Note that specific devices may be configured to be able to communicate directly with each other without going through the LAN 99. For example, the autonomous driving ECU 20 and the operation recording unit 30 may be directly electrically connected by a dedicated line. Also, in FIG. 1, the LAN 99 is configured as a bus type, but is not limited to this. The network topology may be a mesh type, a star type, a ring type, or the like. The network shape can be changed as appropriate. As the standard of the LAN 99, various standards can be adopted, such as Controller Area Network (hereinafter, CAN: registered trademark), Ethernet (Ethernet is a registered trademark), and FlexRay (registered trademark).

[0014] Hereinafter, the vehicle equipped with the autonomous driving system 1 will also be referred to as the vehicle itself, and the occupant sitting in the driver's seat of the vehicle (i.e. the driver's seat occupant) will also be referred to as the user. Note that the front-rear, left-right, and up-down directions in the following description are defined based on the vehicle itself. Specifically, the front-rear direction corresponds to the longitudinal direction of the vehicle itself. The left-right direction corresponds to the width direction of the vehicle itself. The up-down direction corresponds to the vehicle height direction. From another perspective, the up-down direction is defined along the vertical direction of a horizontal plane that defines the front-rear and left-right directions.

[0015] <Overview of each component> The perimeter monitoring sensor 11 is a device that collects information about the external environment around the vehicle. For example, a camera, a millimeter wave radar, LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a sonar, etc. can be adopted as the perimeter monitoring sensor 11. The millimeter wave radar is a device that transmits millimeter waves or quasi-millimeter waves in a predetermined direction and detects the relative position and relative speed of an object with respect to the vehicle by analyzing the received data of the reflected waves that are reflected by the object and returned from the transmitted waves. The millimeter wave radar generates, as a detection result, data indicating, for example, the reception strength and relative speed for each detection direction and distance, or data indicating the relative position and reception strength of the detected object. The LiDAR is a device that generates three-dimensional point cloud data indicating the position of a reflection point for each detection direction by irradiating a laser light.

[0016] 2, the autonomous driving system 1 includes a front camera 11a, a rear camera 11b, a right side camera 11c, a left side camera 11d, a front radar 11e, a right rear radar 11f, and a left rear radar 11g as the periphery monitoring sensors 11. The front camera 11a, the rear camera 11b, the right side camera 11c, and the left side camera 11d can also be referred to as periphery monitoring cameras.

[0017] The front camera 11a is a camera that captures the front of the vehicle at a predetermined angle of view. The front camera 11a is arranged, for example, at the upper end of the windshield on the interior side of the vehicle, or at the front grill. The front camera 11a may be a camera for monitoring the surroundings that is configured to capture a relatively wide angle and a short distance, in addition to the camera arranged at the upper end of the windshield on the interior side of the vehicle and configured to capture a relatively long distance. The rear camera 11b is a camera that captures the rear of the vehicle at a predetermined angle of view. The rear camera 11b is arranged at a predetermined position on the rear part of the body, for example, near the rear license plate or near the rear window. In the description of the mounting position of the surrounding monitoring sensor 11, the "near" of a certain member refers to a range within 30 cm from the member. For example, the near the license plate refers to a range within 30 cm from the license plate. The right side camera 11c and the left side camera 11d are cameras that capture the sides of the vehicle at a predetermined angle of view, and are arranged, for example, at a side mirror or at a predetermined position on the left and right sides of the body (for example, near the base of the A-pillar).

[0018] The front radar 11e is a millimeter wave radar that detects a predetermined range in front of the vehicle by transmitting a search wave toward the front of the vehicle, and is installed, for example, on the front grill or the front bumper. The right rear radar 11f is a millimeter wave radar that detects a predetermined range in the right rear of the vehicle by transmitting a search wave toward the right rear of the vehicle, and is installed, for example, on the right corner of the rear bumper. The left rear radar 11g is a millimeter wave radar that detects a predetermined range in the left rear of the vehicle by transmitting a search wave toward the left rear of the vehicle, and is installed, for example, on the left corner of the rear bumper.

[0019] The various perimeter monitoring sensors 11 detect, for example, a predetermined detection object and identify the relative position of the detection object with respect to the vehicle. Then, detection result data indicating the position, type, etc. of the detected object is sequentially provided to the automatic driving ECU 20 and the operation recording unit 30 (hereinafter, automatic driving ECU 20, etc.). The detection object here is, for example, a pedestrian, an animal other than a human, another vehicle, a structure installed along the road, etc. The other vehicle also includes a bicycle, a motorbike, and a motorcycle. The structure installed along the road is, for example, a guard rail, a curb, a tree, a utility pole, a road sign, a traffic light, etc. In addition, in this embodiment, as a more preferable aspect, road markings such as lane markings and objects fallen on the road are also registered as detection objects of the perimeter monitoring camera.

[0020] The perimeter monitoring sensor 11 also provides observation data used for object recognition, such as image data, to the autonomous driving ECU 20 via the LAN 99. For example, the front camera 11a outputs image data as observation data of the area in front of the vehicle, and analysis results of the image data. Moreover, the observation data of the millimeter wave radar refers to data indicating the reception strength and relative speed for each detection direction and distance, or data indicating the relative position and reception strength of the detected object. The observation data of the ultrasonic sensor refers to the distance measurement results. The observation data of the LiDAR is three-dimensional point cloud data. The observation data corresponds to raw data observed by the sensor, or data before the recognition process is performed.

[0021] The object recognition process based on the observation data may be executed by an ECU outside the sensor, such as the autonomous driving ECU 20. In that case, the camera or millimeter wave radar serving as the periphery monitoring sensor 11 may provide the observation data, such as image data or distance measurement data, as detection result data to the autonomous driving ECU 20. When an internal failure or the like occurs, the various periphery monitoring sensors 11 output an error signal to the LAN 99. For example, the front camera 11a outputs an error signal when it detects an abnormality in the imaging element or processing circuit.

[0022] The occupant status sensor 12 is a sensor that sequentially detects index information indicating the status of the driver's seat occupant. As the index information indicating the status of the driver's seat occupant, a face image, an upper body image, a pressure distribution acting on the seating surface of the driver's seat, a gripping state of the steering wheel, and the like can be adopted. In addition, biological information such as a heart rate, a breathing interval, and an electroencephalogram can also be adopted as the index information indicating the status of the driver's seat occupant. For example, the autonomous driving system 1 includes a driver status monitor (hereinafter, DSM: Driver Status Monitor) 12a, a seat sensor 12b, and a steering wheel sensor 12c as the occupant status sensor 12, as shown in FIG. 3. Note that the autonomous driving system 1 does not need to include all of these sensors. In addition, sensors other than these may be included as the occupant status sensor 12. For example, a wristwatch-type wearable device that measures body temperature and pulse may be included as the occupant status sensor 12.

[0023] The DSM 12a sequentially detects the state of the driver's seat occupant based on the facial image of the driver's seat occupant. Specifically, the DSM 12a uses a near-infrared camera to capture the face of the driver's seat occupant, and performs image recognition processing on the captured image to sequentially detect the face direction, line of sight, eyelid opening degree, etc. of the driver's seat occupant. The DSM 12a is arranged on the top surface of the steering column cover, the top surface of the instrument panel, the rearview mirror, etc., with the near-infrared camera facing the headrest of the driver's seat so that the face of the occupant sitting in the driver's seat can be captured.

[0024] The image captured by the near-infrared camera is analyzed by a control unit (not shown). The control unit of the DSM 12a estimates the degree of wakefulness of the driver's seat occupant by comprehensively considering the face direction, gaze direction, eyelid opening degree, blinking frequency and interval, and the like. The DSM 12a sequentially outputs information indicating the face direction, gaze direction, eyelid opening degree, wakefulness degree, and the like of the driver's seat occupant identified from the captured image to the LAN 99 as occupant state data. In addition, as a more preferable aspect in this embodiment, the DSM 12 provides the face image data of the driver's seat occupant to the automatic driving ECU 20. The face image data of the driver's seat occupant is a judging material for the wakefulness degree of the driver's seat occupant, and corresponds to the observation data of the near-infrared camera constituting the DSM 12a. The automatic driving ECU 20 may be provided with a function of analyzing the image captured by the near-infrared camera. In that case, the DSM 12a may be configured to output the image captured by the near-infrared camera. The image data may be output in the form of a video signal.

[0025] The seat sensor 12b sequentially detects the state of the driver's seat occupant based on the distribution of pressure acting on the backrest and seating surface of the driver's seat. The output signal of the seat sensor 12b can be used as a material for estimating whether the driver's seat occupant is seated in a position suitable for driving. The seat sensor 12b is embedded inside the surface of the backrest and seating surface of the driver's seat. The steering wheel sensor 12c is provided on the steering wheel and detects the pressure of the driver gripping the steering wheel. The steering wheel sensor 12c may be a capacitance type sensor that detects that the occupant is gripping the steering wheel based on a change in capacitance. The steering wheel sensor 12c may be a sensor that detects the torque acting on the steering wheel (for example, a torsion bar). The torque acting on the steering wheel indicates that the user is gripping the steering wheel. The output signal of the steering wheel sensor 12c can be used as a signal indicating whether the driver is gripping the steering wheel with both hands. The detection results of the seat sensor 12b and the steering wheel sensor 12c are provided to the automatic driving ECU 20, for example, via the LAN 99.

[0026] The vehicle state sensor 13 is a sensor that detects state quantities related to the driving control of the host vehicle. The vehicle state sensor 13 includes a brake sensor, an accelerator sensor, a shift position sensor, a steering angle sensor, a vehicle speed sensor, a steering angle sensor, an acceleration sensor, and the like. The brake sensor is a sensor that detects the position of the brake pedal, in other words, the amount by which the brake pedal is depressed by the driver's seat occupant (hereinafter, brake depression amount). The accelerator sensor is a sensor that detects the position of the accelerator pedal, in other words, the amount by which the accelerator pedal is depressed by the driver's seat occupant (hereinafter, accelerator depression amount). The shift position sensor is a sensor that detects the position of the shift lever. The steering angle sensor is a sensor that detects the rotation angle of the steering wheel (so-called steering angle). The brake sensor, accelerator sensor, shift position sensor, and steering angle sensor correspond to sensors that detect physical state quantities that indicate the content of the driving operation by the driver's seat occupant.

[0027] The vehicle speed sensor is a sensor that detects the traveling speed of the host vehicle. The steering angle sensor is a sensor that detects the angle of the tire relative to the vehicle axle (so-called steering angle). The acceleration sensor is a sensor that detects the acceleration in the vehicle aft direction and the acceleration in the lateral direction acting on the host vehicle. The vehicle speed sensor, the steering angle sensor, and the acceleration sensor correspond to sensors that detect physical state quantities that indicate the behavior of the vehicle that occurs as a result of the driving operation of the driver's seat occupant or the control by the autonomous driving ECU 20.

[0028] Each sensor outputs data indicating the current value (i.e., detection result) of the physical state quantity to be detected to the LAN 99. The output data of each sensor is acquired by the automatic driving ECU 20 or the like via the LAN 99. The type of sensor used by the automatic driving system 1 as the vehicle state sensor 13 may be appropriately designed, and it is not necessary to include all of the sensors described above. The automatic driving system 1 may also include sensors other than those described above, such as a yaw rate sensor, an orientation sensor, and an engine rotation speed sensor, as the vehicle state sensor 13. The yaw rate sensor is a sensor that detects the rotational angular velocity (i.e., yaw rate) of the host vehicle around a vertical axis. The orientation sensor is a sensor that detects the orientation angle to which the host vehicle is facing. The engine rotation speed sensor is a sensor that detects the rotation speed of the engine. In addition, if the host vehicle is a vehicle equipped with a motor as a driving source, a sensor that detects the rotation speed / output torque of the motor may be included.

[0029] The locator 14 is a device that generates highly accurate position information of the vehicle by composite positioning that combines a plurality of pieces of information. The locator 14 is realized, for example, as shown in FIG. 4, using a GNSS receiver 141, an inertial sensor 142, a high-accuracy map database (hereinafter, the database is referred to as DB) 143, and a position calculation unit 144. The GNSS receiver 141 is a device that sequentially detects the current position of the GNSS receiver 141 (for example, every 100 milliseconds) by receiving a navigation signal transmitted from a positioning satellite that constitutes a GNSS (Global Navigation Satellite System). As the GNSS, GPS, GLONASS, Galileo, IRNSS, QZSS, Beidou, etc. can be adopted. The inertial sensor 142 is, for example, a three-axis gyro sensor and a three-axis acceleration sensor.

[0030] The high-precision map DB 143 is a non-volatile memory that stores high-precision map data. The high-precision map data here corresponds to map data that indicates the road structure and the position coordinates of features arranged along the road with an accuracy that can be used for autonomous driving. The high-precision map data includes, for example, three-dimensional shape data of the road, lane data, feature data, and the like. The above three-dimensional shape data of the road includes node data related to points (hereinafter, nodes) where multiple roads intersect, merge, or branch, and link data related to roads (hereinafter, links) connecting the points. The link data also includes data indicating the road type, such as whether it is a motorway or a general road. Here, a motorway is a road on which pedestrians and bicycles are prohibited from entering, and refers to, for example, a toll road such as an expressway. The road type may include attribute information indicating whether or not the road is a road on which autonomous driving is permitted. The lane data indicates the number of lanes, the coordinates of the installation positions of lane dividing lines (so-called lane markers), the traveling direction of each lane, and branching / merging points at the lane level. Lane markers include road studs, etc., as well as dashed or continuous lines painted with yellow or white paint. The feature data includes information on the position and type of road markings such as stop lines, and information on the position, shape, and type of landmarks. Landmarks include three-dimensional structures installed along roads, such as traffic signs, traffic lights, poles, and commercial signs.

[0031] The position calculation unit 144 sequentially determines the position of the vehicle by combining the positioning result of the GNSS receiver 141 and the measurement result of the inertial sensor 142. The determined vehicle position is output to the LAN 99 and used by the autonomous driving ECU 20 and the like. The locator 14 also reads out map data of a predetermined range determined based on the current position from the high-precision map DB 143, and provides it to the autonomous driving ECU 20 and the like via the LAN 99. The map data may be obtained from an external server or the like via the V2X vehicle-mounted device 16. Furthermore, the locator 14 (mainly the position calculation unit 144) may be configured to be able to perform localization processing. The localization processing refers to processing for determining the detailed position of the vehicle by matching the coordinates of a landmark specified based on an image captured by a surrounding monitoring camera such as the forward camera 11a with the coordinates of a landmark registered in the high-precision map data. Some or all of the functions of the locator 14 may be included in the autonomous driving ECU 20.

[0032] The HMI system 15 is a system that provides an input interface function for accepting user operations and an output interface function for presenting information to the user. As shown in FIG. 5, the HMI system 15 includes an HCU (HMI Control Unit) 151, an input device 152, and a display 153.

[0033] The HCU 151 is configured to comprehensively control the acquisition of operation information input by the user via the input device 152 and the presentation of information to the user using the display 153. Note that the presentation of information to the user may be realized using a speaker, a vibrator, a lighting device (e.g., an LED), or the like. The HCU 151 is realized using a processor such as a CPU or a GPU, a RAM, a flash memory, and the like. Details of the HCU 151 will be described after the explanation of the input device 152 and the display 153.

[0034] The input device 152 is a device for receiving user's instruction operations for various electronic devices (hereinafter, in-vehicle devices) mounted on the vehicle, such as a navigation device. The in-vehicle devices here include the automatic driving ECU 20 and the operation recording unit 30. In the present embodiment, as an example, the HMI system 15 includes a mechanical switch (i.e., a steering switch) provided on the steering wheel or the like as the input device 152. The HMI system 15 also includes the input device 152 that functions as a driving mode switch. The driving mode switch is a switch for activating or deactivating the automatic driving function. Such a driving mode switch corresponds to a switch for the driver's seat occupant to switch the driving mode. The input device 152 may be a touch panel laminated on the surface of the display 153, or may be a voice input device realized using a well-known voice recognition technology. It may also be a haptic device arranged in the center console. The vehicle control system 100 may include the above-mentioned multiple types of devices as the input device 152.

[0035] The display 153 is a device that displays an image input from the HCU 151. In the present embodiment, as an example, the display 153 is a display (so-called center display) provided at the top of the central part (hereinafter, central area) of the instrument panel in the vehicle width direction. The display 153 is capable of full-color display, and can be realized by using a liquid crystal display, an OLED (Organic Light Emitting Diode) display, a plasma display, or the like. Note that the HMI system 15 may include, as the display 153, a head-up display that projects a virtual image on a portion of the windshield in front of the driver's seat. The display 153 may also be a display (so-called meter display) arranged in an area of ​​the instrument panel located in front of the driver's seat.

[0036] The HCU 151 displays an image (hereinafter, a driving mode notification image 80) indicating the operation state of the automatic driving function on the display 153 based on information provided by the automatic driving ECU 20. The driving mode notification image 80 may include, for example, a mode icon image 81, a road surface recognition state image 82, and a moving object recognition state image 83, as shown in FIG. 6. The mode icon image 81 is an image that clearly indicates the driving mode. The road surface recognition state image 82 is an image that indicates the recognition state of lane markings or road edges in the automatic driving ECU 20. The moving object recognition state image 83 is an image that indicates the recognition state of other moving objects in the automatic driving ECU 20. The planned action notification image 84 is an image that indicates the next action that the automatic driving ECU 20 plans. The HCU 151 may express whether or not the lane markings or road edges are recognized by, for example, changing the display mode (for example, color) of the line image that constitutes the road surface recognition state image 82. The same applies to the moving object recognition state image 83. In addition, when the HCU 151 transfers driving operation authority (so-called handover request) from the autonomous driving system 1 to the driver's seat occupant, it requests the driver's seat occupant to take over driving operations by using a specified notification method using images, light, and sound.

[0037] The HCU 151 provides a screenshot of the display screen of the display 153 based on a request from the operation recording unit 30. The HCU 151 also sequentially provides data indicating a notification state to the occupant regarding the driving mode to the operation recording unit 30. Elements constituting the notification state to the occupant regarding the driving mode can include the content (e.g., image ID) displayed on the display 153 as an image indicating the driving mode, the data ID and volume of the message or warning sound output from the speaker, and the like.

[0038] The V2X vehicle-mounted device 16 is a device for the vehicle to perform wireless communication with other devices. The "V" in V2X refers to an automobile as the vehicle, and the "X" can refer to various entities other than the vehicle, such as pedestrians, other vehicles, road facilities, networks, and servers. The V2X vehicle-mounted device 16 includes a wide-area communication unit and a narrow-area communication unit as communication modules. The wide-area communication unit is a communication module for performing wireless communication conforming to a predetermined wide-area wireless communication standard. Various standards such as LTE (Long Term Evolution), 4G, and 5G can be adopted as the wide-area wireless communication standard. The wide-area communication unit may be configured to perform wireless communication directly with other devices (in other words, without going through a base station) by a method conforming to the wide-area wireless communication standard, in addition to communication via a wireless base station. In other words, the wide-area communication unit may be configured to perform cellular V2X. The vehicle becomes a connected car that can connect to the Internet by being equipped with the V2X vehicle-mounted device 16. For example, the locator 14 can download the latest map data from a specified server in cooperation with the V2X vehicle-mounted device 16 and update the map data stored in the high-precision map DB 143.

[0039] The short-range communication unit included in the V2X vehicle-mounted device 16 is a communication module for directly performing wireless communication with other moving objects and roadside devices present around the vehicle using a communication standard (hereinafter, short-range communication standard) in which the communication distance is limited to approximately several tens to several hundreds of meters. The other moving objects are not limited to vehicles, but may include pedestrians, bicycles, and the like. As the short-range communication standard, any standard may be adopted, such as the Wireless Access in Vehicular Environment (WAVE) standard disclosed in IEEE1609 or the Dedicated Short Range Communications (DSRC) standard.

[0040] The body ECU 17 is an ECU that comprehensively controls the body-related in-vehicle devices mounted on the vehicle. The body-related in-vehicle devices include lighting devices such as headlights and direction indicators, a silo mirror motor that changes the angle of the side mirrors, a window motor that opens and closes the windows, a door lock motor, a wiper blade motor, and the like. The body ECU 17 controls the lighting state of the headlights based on, for example, the operation of a light switch by a driver, a detection value of an illuminance sensor, or an instruction from the autonomous driving ECU 20. The body ECU 17 also outputs various signals indicating the operating state of lighting devices such as headlights and direction indicators, the operating state of wipers, and the opening / closing and locking states of doors to the LAN 99. Note that, in the case where the vehicle is equipped with a driving mode notification light, which is a lighting device that notifies the outside, such as pedestrians, of the driving mode (for example, that the vehicle is autonomously driving), the driving mode notification light can also be included in the body-related in-vehicle devices.

[0041] The autonomous driving ECU 20 is an ECU that controls actuators for driving (hereinafter, driving actuators 18) based on the detection results of the surrounding monitoring sensors 11 and the like, thereby performing some or all of the driving operations on behalf of the driver. The autonomous driving ECU 20 corresponds to an autonomous driving device that provides an autonomous driving function. The driving actuators 18 include, for example, a braking device (a so-called brake actuator), an electronic throttle, a steering actuator, and the like. The driving actuators 18 are hardware elements related to at least one of the acceleration, deceleration, and steering of the vehicle.

[0042] The concept of autonomous driving here is not limited to a mode in which all operations related to the driving of a vehicle are performed automatically, but can also include a mode in which some driving operations are performed automatically. In other words, it can also include autonomous driving level 1 to level 2 control defined by the Society of Automotive Engineers (SAE International). Note that autonomous driving level 1 refers to a level in which the system supports either one of steering operation and acceleration / deceleration operation, and autonomous driving level 2 refers to a level in which the system supports two or more of steering operation and acceleration / deceleration operation. Autonomous driving level 3 refers to a level in which the system performs all driving operations within an operational design domain (ODD) that defines the conditions under which autonomous driving can be performed (i.e., autonomous driving permission conditions), while the system transfers operational authority to the driver in an emergency. Level 4 refers to a level in which the system performs all driving operations within the ODD. Level 5 refers to a level in which the system performs all driving operations without location restrictions. Level 3 and above correspond to an autonomous driving level in which all control related to the driving of a vehicle is automatically performed.

[0043] Here, as an example, the autonomous driving ECU 20 is configured to be capable of executing autonomous driving up to level 5, and is configured to be able to switch between operation modes corresponding to each automation level. Hereinafter, for convenience, an operation mode corresponding to autonomous driving level N (N=0 to 5) is referred to as a level N mode. For example, level 3 mode refers to an operation mode in which the vehicle is autonomously driven at autonomous driving level 3. In addition, in this specification, the levels 3 to 5 modes are also referred to as autonomous driving modes, and the levels 1 to 2 modes are referred to as driving assistance modes. The levels 0 to 2 modes correspond to occupant involvement modes in which at least a part of the driving operation is left to the occupant in the driver's seat. The operation modes of the autonomous driving ECU 20 correspond to driving modes from the occupant's perspective.

[0044] The switching of the operation mode is performed automatically due to a user operation, a system limit, exiting from ODD, etc. In the autonomous driving mode, the autonomous driving ECU 20 automatically performs steering, acceleration, deceleration (in other words, braking), etc. of the vehicle so that the vehicle travels along the road to the destination set by the driver's seat occupant or the operator.

[0045] The automatic driving ECU 20 is mainly composed of a computer including a processing unit 21, a RAM 22, a storage 23, a communication interface 24, and a bus connecting these. The processing unit 21 is hardware for arithmetic processing coupled to the RAM 22. The processing unit 21 includes at least one arithmetic core such as a CPU (Central Processing Unit). The processing unit 21 executes various processes for realizing the functions of each functional unit described later by accessing the RAM 22. The storage 23 includes a non-volatile storage medium such as a flash memory. The storage 23 stores a program executed by the processing unit 21 (hereinafter, an automatic driving program). The processing unit 21 executing the automatic driving program corresponds to the execution of a method corresponding to the automatic driving program. The communication interface 24 is a circuit for communicating with other devices via a LAN 99. The communication interface 24 may be realized using analog circuit elements, ICs, etc. Details of the automatic driving ECU 20 will be described separately later.

[0046] The operation recording unit 30 is a device that records data indicating at least one of the conditions inside the vehicle while the vehicle is traveling and the conditions outside the vehicle cabin. The conditions inside the vehicle while the vehicle is traveling may include the operating state of the automatic driving ECU 20 and the state of the occupant in the driver's seat. Furthermore, the data indicating the operating state of the automatic driving ECU 20 also includes the recognition result of the surrounding environment in the automatic driving ECU 20, the driving plan, the calculation result of the target control amount of each driving actuator, and the like. The data to be recorded is acquired from ECUs and sensors mounted on the vehicle, such as the automatic driving ECU 20 and the surroundings monitoring sensor 11, via a LAN 99 or the like.

[0047] The operation recording unit 30 is also mainly composed of a computer including a processing unit 31, a RAM 32, a storage 33, a communication interface 34, and a bus connecting these. The processing unit 31 is hardware for arithmetic processing coupled with the RAM 32. The processing unit 31 includes at least one processor such as a CPU. The processing unit 31 executes various processes for realizing the functions of each functional unit described later by accessing the RAM 32. The storage 33 includes a non-volatile storage medium such as a flash memory. The storage 33 stores a program executed by the processing unit 21 (hereinafter, an operation status recording program). The processing unit 31 executing the operation status recording program corresponds to the execution of a method (corresponding to an information recording method) corresponding to the operation status recording program. The communication interface 34 is a circuit for connecting to a LAN 99 and communicating with other devices via the LAN 99. The communication interface 34 is realized by using analog circuit elements, ICs, etc. Details of the operation recording unit 30 will be described separately later.

[0048] <About Autonomous Driving ECU 20> Here, the functions and operations of the autonomous driving ECU 20 will be described with reference to Fig. 7. The autonomous driving ECU 20 provides functions corresponding to various functional blocks shown in Fig. 7 by executing an autonomous driving program stored in the storage 23. That is, the autonomous driving ECU 20 includes, as functional blocks, a surrounding environment recognition unit F1, an occupant state recognition unit F2, a host vehicle state acquisition unit F3, an operation content acquisition unit F4, a mode control unit F5, an action planning unit F6, a control execution unit F7, an abnormality detection unit F8, and an operation state output unit F9.

[0049] The surrounding environment recognition unit F1 recognizes the surrounding environment around the vehicle based on the detection results of the surrounding monitoring sensor 11. The surrounding environment here includes not only static environmental factors such as the current position, driving lane, road type, speed limit, and relative positions of features, but also dynamic environmental factors such as the positions and moving speeds of other moving objects, weather, brightness, and temperature. The other moving objects include other vehicles such as automobiles, pedestrians, and bicycles.

[0050] For example, the surrounding environment recognition unit F1 acquires the latest vehicle position and traveling direction, and high-precision map data of the surrounding area of ​​the vehicle A from the locator 14. The surrounding environment recognition unit F1 also acquires detection results from each of the multiple surrounding monitoring sensors 11, and recognizes the position and type of objects present around the vehicle A by complementarily combining (i.e., fusing) them. That is, the environment recognition result of the autonomous driving ECU 20 also includes the recognition result of objects present around the vehicle A. Objects to be recognized include features on the ground and moving objects. The recognition result of an object may include the position, type, and moving speed of the detected object. The position and speed may be a relative position and relative speed based on the vehicle A, or an absolute position and absolute speed based on the ground.

[0051] Additionally, the surrounding environment recognition unit F1 evaluates the possibility of collision with an object present around the vehicle. As an index indicating the possibility of collision, the remaining time to collision (so-called TTC: Time-To-Collision), the margin to collision (MTC: Margin-To-Collision), the time to headway (THW: Time-Headway), and the like can be adopted. The concept of TTC also includes the use of the acceleration or jerk of at least one of the vehicle and the other vehicle. Furthermore, the surrounding environment recognition unit F1 recognizes the positions and types of road markings and signs around the vehicle and the lighting status of traffic lights based on the detection results of the surrounding monitoring sensor 11 and high-precision map data.

[0052] In addition, the surrounding environment recognition unit F1 uses at least one of the detection results of the perimeter monitoring sensors 11 and the high-precision map data to identify the relative positions and shapes of the left and right dividing lines or road edges of the lane on which the vehicle is currently traveling as boundary information related to the boundary of the lane. Note that the data acquired by the surrounding environment recognition unit F1 from each perimeter monitoring sensor 11 may be observation data such as image data instead of analysis results. In that case, the surrounding environment recognition unit F1 may identify the surrounding environment including the positions and shapes of the left and right dividing lines or road edges based on the observation data of the various perimeter monitoring sensors 11.

[0053] In addition, the surrounding environment recognition unit F1 may identify the surrounding environment using other vehicle information received by the V2X vehicle-mounted device 16 from other vehicles, traffic information received from a roadside device via road-to-vehicle communication, weather information received from a server, etc.

[0054] The occupant state recognition unit F2 judges the state of the driver's seat occupant based on the information provided by the occupant state sensor 12. For example, the occupant state recognition unit F2 judges the degree of wakefulness of the driver's seat occupant based on the data provided by the DSM 12a. The degree of wakefulness may be evaluated in two stages of wakefulness or not, or may be evaluated in three or more stages. The occupant state recognition unit F2 judges whether the driver's seat occupant is in a forward gaze state in which the driver is looking ahead of the vehicle based on the data provided by the DSM 12a. For example, the occupant state recognition unit F2 judges the driver's seat occupant to be in a forward gaze state when the eyelid opening degree is equal to or greater than a predetermined threshold value and the line of sight of the driver's seat occupant is directed toward the front of the vehicle. In this specification, the forward gaze state is also referred to as eyes on, and the non-forward gaze state is also referred to as eyes off. The degree of wakefulness and eyes on / off may be determined by combining sensing information from multiple types of sensors such as the DSM 12a and the seat sensor 12b.

[0055] Furthermore, the occupant state recognition unit F2 judges whether or not the occupant in the driver's seat is in a steering wheel gripping state, based on the output signal of the steering wheel sensor 12c. For example, when the steering wheel sensor 12c detects a pressure or capacitance change equal to or greater than a predetermined threshold, it judges that the occupant is in a steering wheel gripping state. In this specification, the steering wheel gripping state is also referred to as "hands on," and the non-steering wheel gripping state is also referred to as "hands off."

[0056] Alternatively, the occupant state recognition unit F2 may be configured to determine whether or not the vehicle is in a deadman state based on at least the output signals of the DSM 12a and the seat sensor 12b. The deadman state refers to a state in which it is difficult to return to a state in which the driver can normally drive due to a sudden change in physical condition such as a stroke. The deadman state can be determined based on the posture and pulse of the driver's seat occupant, the degree of pupil opening, vehicle behavior, etc.

[0057] The vehicle state acquisition unit F3 recognizes the state of the vehicle based on the output data of the vehicle state sensor 13 and the body ECU 17. The state of the vehicle includes the vehicle speed, acceleration, yaw rate, shift position, engine rotation speed, steering angle, etc. The vehicle state acquisition unit F3 also acquires the operating state of the body-related in-vehicle devices based on the output signal of the body ECU 17. For example, the vehicle state acquisition unit F3 communicates with the body ECU 17 to acquire data indicating the operating state of the turn signal, the illumination state of the headlights, the operating state of the wipers, etc.

[0058] The operation content acquisition unit F4 detects the presence or absence / amount of operation of the driver's seat occupant to these operating members based on output signals (in other words, operation signals) from pedals, levers, switches, and input devices 152 provided in the vehicle. For example, the operation content acquisition unit F4 acquires the operation content of the occupant to the automatic driving switch, the amount of brake depression, etc.

[0059] The mode control unit F5 switches the driving mode based on at least one of the surrounding environment, the passenger state, the vehicle state, and the user operation. For example, the mode control unit F5 switches the driving mode based on the user operation. For example, when the mode control unit F5 operates in a mode of level 2 or lower, if the mode control unit F5 receives a user operation to switch to the level 3 mode via the input device 152, the mode control unit F5 switches to the level 3 mode on the condition that the surrounding environment is capable of executing the automatic driving level 3. The same applies to the transition from the mode of level 2 or lower to the level 4 mode. Whether or not the surrounding environment is capable of executing the automatic driving levels 3 to 4 may be determined based on the recognition result of the surrounding environment recognition unit F1. For example, if the recognition result of the surrounding environment satisfies the ODD specified for the vehicle, it can be determined that the surrounding environment is capable of executing the automatic driving levels 3 to 4. In addition, when the mode control unit F5 receives a user operation to switch to the level 3 mode via the input device 152 under a situation where the mode control unit F5 operates in a mode of level 2 or lower, the mode control unit F5 switches the operation mode to the level 5 mode.

[0060] On the other hand, if it becomes difficult to maintain the autonomous driving mode while driving in the autonomous driving mode, the authority transfer process is inserted and the mode is switched to level 2. Circumstances in which the autonomous driving mode cannot be maintained include when the ODD is no longer satisfied due to a sensor failure or dynamic environmental factors, or when the vehicle leaves a road section designed as an ODD.

[0061] The authority transfer process is a process for transferring driving authority from the system to the driver's seat occupant, and is executed mainly under the planning of the action planning unit F6 in cooperation with the automatic driving ECU 20 and the HMI system 15. The driver's seat occupant is requested to take over driving operations for a predetermined period of time in the authority transfer mode, and when the occupant performs a predetermined operation such as holding the steering wheel during the process, the transfer is completed. The state in which the authority transfer process is being executed is called the transfer mode. The transfer mode can also include an operation mode in which the driving authority is transferred from the driver's seat occupant to the system.

[0062] In addition, the mode control unit F5 may gradually lower the autonomous driving level in the order of 5 → 4 → 3 → 2 → 1 → 0 due to a decrease in the number of surrounding monitoring sensors 11 that are operating normally or a deterioration of the environment. For example, when the recognizable distance of the front camera 11a caused by fog or rain becomes equal to or less than a predetermined value, the mode control unit F5 may switch the operation mode from the level 4 mode to the level 3 mode. The deterioration of the environment refers to, for example, an increase in the amount of rainfall, backlight, etc. The concept of rainfall also includes the amount of snowfall.

[0063] In addition, when the driver's seat occupant becomes a dead man during the autonomous driving mode or does not respond to the authority transfer process, the autonomous driving ECU 20 executes MRM (Minimum Risk Maneuver) processing to safely stop the vehicle. The content of the MRM processing can be, for example, processing to autonomously drive the vehicle to a safe place while issuing an alarm to the surroundings and park it. The safe place can be a road shoulder having a width of a predetermined value or more, or a place specified as an emergency evacuation area. The content of the MRM can also be stopping in the lane in which the vehicle is traveling by slow deceleration. As the deceleration in this case, it is preferable to adopt a value of 4 [m / s^2] or less, such as 2 [m / s^2] or 3 [m / s^2]. Of course, when it is necessary to avoid a collision with a preceding vehicle, a deceleration exceeding 4 [m / s^2] can also be adopted. The deceleration during the MRM may be dynamically determined and sequentially updated in consideration of the traveling speed at the time of starting the MRM and the distance between the vehicle and the following vehicle, within a range in which the vehicle can be stopped within 10 seconds, for example.

[0064] In addition, when the mode control unit F5 detects, for example, an override by a driver's seat occupant based on the information acquired by the operation content acquisition unit F4, the mode control unit F5 transitions to one of the operation modes of levels 0 to 2 based on the user settings. The mode control unit F5 outputs status information indicating the operation mode of the autonomous driving ECU 20 to the communication network. From another perspective, the status information corresponds to data indicating whether the driving mode of the host vehicle is an autonomous driving mode or a manual driving mode.

[0065] When the automatic driving ECU 20 is in the autonomous driving mode, the action planning unit F6 creates a driving plan (in other words, an action plan) for the vehicle based on the surrounding environment recognized by the surrounding environment recognition unit F1. The driving plan includes a long-term plan and a short-term plan. The long-term plan includes the selection of a driving route to a destination on a road-by-road or lane-by-lane basis. The short-term plan includes the determination of the content of an action to be performed within a predetermined time (e.g., 10 seconds) from the current time point and specific control target values ​​(e.g., target vehicle speed, target acceleration, target steering amount, etc.) for realizing the planned action based on the long-term plan. The short-term plan includes the selection of an action to be performed next from the current time point (hereinafter, a planned action). Options for the planned action include maintaining vehicle speed, accelerating, decelerating, stopping, starting, driving on a straight road, turning on a curve, changing lanes, going straight through an intersection, turning left, turning right, etc. In addition, a composite action consisting of a combination of multiple actions such as overtaking and parking may also be set as the planned action. In addition, when the autonomous driving ECU 20 is in the driving assistance mode, the action planning unit F6 calculates a control target value corresponding to an operation item (for example, speed adjustment) that the system should assist with.

[0066] The action planning unit F6 includes an ACC (Adaptive Cruise Control) function, an LTC (Lane Trace Control) function, an AEB (Automatic Emergency Braking) function, and the like as element functions (applications) for providing an automatic driving function. The ACC function is a function for driving the host vehicle at a constant speed at a target vehicle speed, or for driving the host vehicle while maintaining a distance from a vehicle ahead. The LTC function is a function for driving the host vehicle within the host vehicle's driving lane along the host vehicle's driving lane, and generates a planned driving line along the host vehicle's driving lane. The AEB function (so-called collision damage mitigation brake) automatically applies the brakes when a collision with an object in front of the vehicle is predicted.

[0067] The control execution unit F7 determines specific control amounts of the various traveling actuators 18 based on the traveling plan formulated by the action planning unit F6. Then, it outputs control signals corresponding to the control amounts to each traveling actuator 18. The control execution unit F7 also controls turning on / off of turn signals, headlights, hazard lights, etc., according to the traveling plan and the external environment. When the traveling mode of the host vehicle is an operation mode of level 2 or lower, the control execution unit F7 comprehensively operates the traveling actuators 18 based on operation information corresponding to the driving operation to be performed by the driver's seat occupant, thereby controlling the behavior of the host vehicle.

[0068] The abnormality detection unit F8 is configured to detect abnormalities in components necessary for automatic driving, such as the perimeter monitoring sensor 11, the locator 14, and the operation recording unit 30. For example, the abnormality detection unit F8 detects an abnormality in the front camera 11a based on an error signal input from the front camera 11a. The abnormality detection unit F8 also diagnoses the operating state of the operation recording unit 30 by communicating with the operation recording unit 30. As a method for detecting an abnormality, well-known methods such as a watchdog timer method and a homework answer method can be used. The watchdog timer method is a method in which, when a watchdog timer provided in a monitoring device expires without being cleared by a watchdog pulse input from a monitored device, it is determined that the monitored device is not operating normally. Here, the automatic driving ECU 20 (specifically, the abnormality detection unit F8) corresponds to the monitoring device, and the operation recording unit 30 corresponds to the monitored device.

[0069] The homework answer method is a method in which the monitoring device sends a predetermined monitoring signal to the monitored device, and judges whether the monitoring device is operating normally based on whether the answer returned from the monitored device is correct. In the homework answer method, the monitored device generates a response signal corresponding to the monitoring signal input from the monitoring device and returns it to the monitoring device. If the content of the response signal received from the monitored device is different from the data corresponding to the transmitted monitoring signal, or if the response signal is not returned within a predetermined time limit, the monitoring device judges that the monitored device is operating abnormally. Note that the autonomous driving ECU 20 may reset the operation recording unit 30 when it detects abnormal operation of the operation recording unit 30. Also, if it detects that the operation recording unit 30 is not operating normally, it may downgrade the driving mode to level 2 or lower (i.e., driving assistance mode). According to a configuration in which the operation mode is limited to level 2 or lower based on the fact that the operation recording unit 30 is not operating normally, it is possible to reduce the risk of autonomous driving being performed under circumstances in which various data may not be recorded.

[0070] The abnormality detection unit F8 may be configured to detect an internal abnormality in the autonomous driving ECU 20. For example, when the autonomous driving ECU 20 includes a plurality of processing units 21, the plurality of processing units 21 may be configured to diagnose each other's operating status by performing bidirectional communication.

[0071] The operating state output unit F9 sequentially outputs the results of the recognition of the surrounding environment by the surrounding environment recognition unit F1, the results of the recognition of the driver's seat occupant by the occupant state recognition unit F2, the action plan data, and the control data of the traveling actuators 18 to the operation recording unit 30. The action plan data includes the next action to be performed selected by the action planning unit F6, and control target values ​​such as the target vehicle speed, the target acceleration, and the target traveling position. The control data of the traveling actuators 18 indicates the contents of the control signals output to each traveling actuator 18. The control data indicates, for example, the output voltage / current values ​​to the various traveling actuators 18, which correspond to the control amounts of the various traveling actuators 18.

[0072] <About the Operation Recording Unit 30> Here, the function and operation of the operation recording unit 30 will be described with reference to Fig. 8. The operation recording unit 30 provides functions corresponding to the various functional blocks shown in Fig. 8 by executing programs stored in the storage 33. That is, the operation recording unit 30 has, as functional blocks, an information acquisition unit G1, a recording target setting unit G2, a recording processing unit G3, and an abnormality detection unit G4. The information acquisition unit G1 has a mode acquisition unit G11. The operation recording unit 30 also has a recording target setting memory unit Md, and a data storage unit Mm.

[0073] The recording target setting memory unit Md is a storage medium that stores data indicating the recording target for each driving mode (hereinafter, recording target setting data D1). The data storage unit Mm is a storage destination for data by the recording processing unit G3. The recording target setting memory unit Md and the data storage unit Mm are each realized using a portion of the storage area of ​​the storage 33. Note that the recording target setting memory unit Md and the data storage unit Mm may each be realized using a physically independent storage medium. Also, a portion or all of the data storage unit Mm may be provided outside the vehicle, such as in a server.

[0074] As shown in FIG. 9, the data storage unit Mm includes an overwriteable area M1 and an overwrite-prohibited area M2. The overwriteable area M1 is an area in which the stored data can be overwritten with other data, in other words, the stored data can be deleted. The overwrite-prohibited area M2 is an area configured such that overwriting of the stored data is not possible / prohibited. The overwriteable area M1 and the overwrite-prohibited area M2 may be logically separated using a partition technique, or may be physically independent. The overwrite-prohibited area M2 may be realized using a non-rewritable storage medium, or may be configured such that deletion of data is prohibited by software. Here, as an example, the overwrite-prohibited area M2 is a storage area configured such that automatic overwriting of data by software cannot be performed.

[0075] It should be noted that the overwrite-prohibited area M2 is not an essential element. The overwrite-allowed area M1 and the overwrite-prohibited area M2 do not need to be clearly separated in the data storage unit Mm. The overwrite-allowed area M1 and the overwrite-prohibited area M2 may be separated by the concept of folders. The overwrite-prohibited area M2 may be configured so that stored data can be deleted by an operation by a user or a specialized company, or by an instruction signal from a specified server.

[0076] As an example, it is preferable that each of the overwriteable area M1 and the overwrite-prohibited area M2 has a capacity capable of recording 2500 sets of data recorded by the event recording process described later. As an example, the overwrite-prohibited area M2 has a capacity capable of recording 2500 sets of 35 seconds of video signals from the front camera 11a. The number of times recording can be performed may be 2000 or more. With this capacity, data for approximately six months can be stored.

[0077] The information acquisition unit G1 sequentially acquires multiple types of information that can be recorded from various in-vehicle devices such as the autonomous driving ECU 20 and the surrounding monitoring sensor 11, assigns information indicating the time the information was acquired (a so-called timestamp), and provides the information to the recording processing unit G3.

[0078] As shown in Fig. 10, major categories of information that can be recorded include driving mode, time information, driver's seat occupant status, occupant operation content, body signal, mode notification status, driving location, observation data of the surrounding monitoring sensor 11, and operation status information of the autonomous driving ECU 20. The occupant status data includes items such as whether the driver's seat occupant is awake or not, eyes on / off, hands on / off, and captured image data of the face of the driver's seat occupant. The occupant status data can include pressure distribution detected by the seat sensor 12b, output value of the steering wheel sensor 12c (e.g. steering wheel torque), and the like.

[0079] The mode notification state refers to a notification status to an occupant regarding the operating state of the autonomous driving ECU 20. For example, it refers to the display content as the driving mode notification image 80. The information acquisition unit G1 causes the HMI system 15 to acquire a screenshot of the display 153 displaying the driving mode notification image 80 as the mode notification state. Note that the mode notification state may include an output signal from the autonomous driving ECU 20 to the HMI system 15 and an output signal from the HCU 151 to various notification devices (for example, the display 153).

[0080] The driving location includes items such as location coordinates and road type. The operating state of the autonomous driving ECU 20 includes items such as recognition result information of other vehicles and lane lines by the surrounding environment recognition unit F1, planned action information by the action planning unit F6, and control values ​​determined by the control execution unit F7. The mode acquisition unit G11 acquires operation mode information from the autonomous driving ECU 20 and provides it to the recording target setting unit G2.

[0081] The interval at which the information acquisition unit G1 acquires various data such as the driving mode, wakefulness state, eyes on / off, hands on / off, operation content, body signal, mode notification state, driving location, and operation state of the autonomous driving ECU 20 may be, for example, 100 milliseconds. The specific value of the data acquisition interval can be changed as appropriate. The data acquisition interval by the information acquisition unit G1 corresponds to the data storage interval by the recording processing unit G3 described later. The data acquisition interval may be dynamically changed depending on the situation. Image data captured by the surrounding monitoring camera or the DSM 12a may be acquired as a continuous video signal.

[0082] Other major categories of information that may be recorded include vehicle status. The vehicle status includes items such as vehicle speed, acceleration, yaw rate, shift position, engine speed, steering angle, and operating status of body-related in-vehicle devices. These items of information can be obtained from the vehicle status sensor 13 and the body ECU 17.

[0083] The record target setting unit G2 sets information items to be recorded according to the current operation mode of the autonomous driving ECU 20. Here, as an example, the record target is changed for each operation mode in the manner shown in Fig. 10. Specifically, in levels 0 to 2, the drive mode, time, operation details of the driver's seat occupant, and notification status of the HMI are set as the information to be recorded.

[0084] In the authority transfer mode, the following are set as the recording targets: driving mode, time, occupant alertness level, eyes on / off, hands on / off, face image, occupant operation content, HMI notification state, driving position coordinates, road type, recognition results of white lines, etc., and actuator control values. In the level 3 mode, the following are set as the recording targets: driving mode, time, occupant alertness state, face image, body signal, mode notification state, driving position coordinates, road type, front camera image, recognition results of white lines, etc., scheduled actions, and actuator control values. In the level 4 mode, the following are set as the recording targets: driving mode, time, body signal, mode notification state, driving position coordinates, road type, observation data of multiple periphery monitoring sensors 11 including the front camera 11a, recognition results of white lines, etc., scheduled actions, and actuator control values. In the level 5 mode, the following are set as the recording targets: driving mode, time, body signal, mode notification state, driving position coordinates, observation data of multiple periphery monitoring sensors 11 including the front camera 11a, recognition results of white lines, etc., scheduled actions, and actuator control values.

[0085] The recording processing unit G3 is configured to store data of items set as recording targets by the recording target setting unit G2 in the data storage unit Mm. In the present embodiment, as an example, the recording processing unit G3 is configured to operate in two modes: a continuous recording mode and an event recording mode. That is, the recording processing unit G3 sequentially stores the recording target data acquired by the information acquisition unit G1 in the overwritable area M1 as the continuous recording mode. The sequentially acquired data is stored in chronological order, for example, with the most recent data at the top. When the capacity of the overwritable area is full, the recording processing unit G3 can delete old data and store new data.

[0086] Furthermore, the recording processing unit G3 executes an event recording process as an event recording method when a predetermined recording event occurs. The event recording process is a process of storing data to be recorded within a predetermined time before and after the event occurrence in the data storage unit Mm in an overwrite-prohibited format. For example, as the event recording process, the recording processing unit G3 stores data to be recorded within a predetermined time before and after the event occurrence in an overwrite-prohibited area M2. Note that the recording destination of data accompanying the occurrence of a recording event does not necessarily have to be the overwrite-prohibited area M2. It may be the overwrite-enabled area M1. The storage format of data accompanying the occurrence of a recording event may also be a format that allows overwriting with new data.

[0087] Examples of the recorded events include lowering the autonomous driving level due to user operation, system limit, or exit from ODD, or starting MRM when AEB is activated. The lowering of the autonomous driving level also includes transitioning from an autonomous driving mode such as level 3 mode to an occupant-involved mode such as level 2 mode. In addition, the recorded events may include a case where a collision sensor detects an impact equal to or greater than a predetermined threshold, a case where an output value of an acceleration sensor is equal to or greater than a predetermined threshold, or a case where an output value of a yaw rate sensor is equal to or greater than a predetermined threshold. In addition, the recorded events may include the start of execution of an application such as ACC or LTC. In addition, the recorded events may include switching of the driving mode.

[0088] In addition, the TTC or MTC calculated by the automatic driving ECU 20 may be set to a value that is equal to or lower than a predetermined recording threshold as a recording event. The recording threshold is preferably set to a value that indicates a state in which a collision is highly likely. For example, a value at which a collision damage mitigation brake is activated can be used. More specifically, the recording threshold for the TTC can be set to 1.4 to 1.6 seconds. The recording threshold for the MTC can be set to a value of 1.0 or less (for example, 0.8 seconds). By using a decrease in the TTC or MTC as a recording event, recording can be performed before a collision actually occurs, and the risk of recording failure due to the impact of the collision can be reduced.

[0089] The recording target period is, for example, as shown in FIG. 11, a period from a pre-event time Tα before the event occurrence time until a post-event time Tβ has elapsed. The pre-event time Tα is set to, for example, 30 seconds, and the post-event time Tβ is set to 5 seconds. With this configuration, data for 35 seconds before and after the event occurrence time can be saved. The pre-event time Tα may of course be 15 seconds, 30 seconds, 45 seconds, 60 seconds, etc. Also, the post-event time Tβ can be 3 seconds, 10 seconds, or 15 seconds. The longer the pre-event time Tα or the post-event time Tβ is, the more the analytic ability of the cause of an event such as an accident can be improved, but the greater the risk of insufficient storage capacity. Also, the shorter the pre-event time Tα or the post-event time Tβ is, the less the risk of insufficient storage capacity can be, but the greater the risk of missing information necessary for investigating the cause of an accident. With the above-mentioned settings of 30 seconds before the event and 5 seconds after the event, it is expected that the analytic ability of the cause of the event occurrence can be sufficiently ensured while reducing the shortage of storage capacity. According to such an embodiment, when a recording event occurs in the autonomous driving mode, the observation data of the periphery monitoring sensor 11 for 35 seconds before and after the event is stored in an overwrite-prohibited manner, so that the cause of the accident can be analyzed based on objective facts. For data before the event occurs, for example, various data stored in the overwrite-permitted area M1 can be copied to the overwrite-prohibited area M2. For data after the event occurs, the data can be added and stored in the overwrite-prohibited area M2 in sequence. The data stored in the data storage unit Mm may be encrypted and stored so that the contents cannot be viewed without using a specific software used by the data analysis company.

[0090] The length of the recording period may be changed according to the type and characteristics of the recording event. The recording period during AEB operation may be set longer than the recording period during ACC start in order to reliably capture the events before and after the accident. The recording period during ACC start may be set to include the events before and after the transition, for example, 5 seconds before the transition and 3 seconds after the transition. The recording period during the transition from level 5 mode to level 4 mode may be set shorter than the recording period during the transition from level 3 mode to level 2 mode. The transition from level 4 mode to level 5 mode is completed within the system (in other words, the occupant is not involved), so problems are unlikely to occur even if the recording period is set relatively short. On the other hand, by setting the recording period during the transition from level 3 mode to level 2 mode relatively longer, the events before and after the transfer of authority can be reliably captured.

[0091] That is, the length of the recording period may be prepared in a plurality of patterns, and may be selected according to the seriousness of the recorded event and the degree of involvement of the occupant. It is preferable to set the recording period longer as the seriousness or the degree of involvement of the occupant increases. Recording events with a high degree of seriousness include, for example, the occurrence of a collision, the operation of the AEB, and the execution of the MRM. Recording events with a high degree of involvement of the occupant include, for example, a transition from the level 3 mode to the level 2 mode. The degree of involvement of the occupant is relatively high not only in the level 3 mode but also in the transition from the autonomous driving mode to the occupant involvement mode by the occupant override. The sudden transition from the level 3 mode to the occupant involvement mode due to the system limit is relatively more serious than the transition from the level 3 mode to the occupant involvement mode that was planned in advance and notified to the occupant. The length of the recording period for each event may be set in consideration of the seriousness and the degree of involvement of the occupant as described above. According to such a configuration in which the recording period is changed according to the characteristics of the event, it is possible to save only the amount of data required for post-mortem analysis. For example, the recording period for events with a relatively low level of severity or occupant involvement can be set to be short, reducing the risk of storing unnecessary data.

[0092] In this embodiment, the recording processing unit G3 is configured to operate in two modes, a continuous recording mode and an event recording mode, but is not limited thereto. The recording processing unit G3 may be configured to execute data recording only when a predetermined recording event occurs. A configuration that executes only event recording processing can reduce the amount of data stored per month, for example, compared to a configuration that employs the continuous recording mode. The recording processing unit G3 may also be configured to operate only in the continuous recording mode.

[0093] The abnormality detection unit G4 is configured to determine whether the operation recording unit 30 itself and the autonomous driving ECU 20 are operating normally. As a method for detecting an abnormality, the above-mentioned methods such as a watchdog timer method and a homework answer method can be used. For example, the abnormality detection unit G4 may determine that the autonomous driving ECU 20 is operating abnormally when the watchdog timer included in the abnormality detection unit G4 expires without being cleared by a watchdog pulse input from the autonomous driving ECU 20.

[0094] In addition, when the abnormality detection unit G4 detects an internal abnormality of the operation recording unit 30, it notifies the autonomous driving ECU 20 of the abnormality. The occurrence of an internal abnormality may be detected based on the deviation of the calculation result from the normal range, the occurrence of an abnormal voltage, the output of an abnormality code, or the like. Whether or not the operation recording unit 30 is operating normally may be diagnosed by an on-board diagnostic device (OBD: On-Board Diagnostics). The abnormality detection unit G4 may be realized using the OBD. According to the configuration in which the processing unit 21 that provides the autonomous driving function and the processing unit 31 that is responsible for the data recording process are independent as in this embodiment, the operating state can be monitored by each other, so that if a malfunction occurs in either one, it can be detected immediately. As a result, if a malfunction occurs in either the processing unit 21 or the processing unit 31, it is possible to quickly interrupt the autonomous driving mode, etc.

[0095] <Detailed explanation of the operation of the operation recording unit 30> Next, the flow of operation of the operation recording unit 30 will be described with reference to the flowchart shown in Fig. 12. The flowchart shown in Fig. 12 may be started, for example, when the vehicle's running power source is turned on. The running power source is, for example, the ignition power source in an engine vehicle. In an electric vehicle, the system main relay corresponds to the running power source. Although not shown in Fig. 12, the information acquisition unit G1 successively acquires various data and video signals. The acquisition of various information by the information acquisition unit G1 may be included in step S1.

[0096] First, in step S1, the mode acquisition unit G11 acquires driving mode information from the autonomous driving ECU 20 and proceeds to step S2. Step S1 corresponds to a mode acquisition step. In step S2, the record target setting unit G2 determines the type of data to be recorded based on the driving mode information acquired in step S1 and the record target setting data D1 stored in the record target setting memory unit Md. Step S2 corresponds to a record target setting step. Note that if there has been no change in the driving mode since the previous time step S2 was performed, step S2 can be omitted.

[0097] In step S3, the recording processing unit G3 stores multiple types of data set as the data to be recorded in the data storage unit Mm in an overwritable format together with a timestamp indicating the acquisition time. For example, the data to be recorded is stored in the overwritable area M1 together with the timestamp. Step S3 corresponds to a recording step. When step S3 is completed, step S4 is executed.

[0098] In step S4, the recording processing unit G3 judges whether or not a recording event has occurred based on various information (e.g., acceleration, etc.) acquired by the information acquisition unit G1. If it is determined that a recording event has occurred, step S5 is executed. On the other hand, if it is determined that a recording event has not occurred, step S6 is executed. In step S5, the recording processing unit G3 stores the data to be recorded for a predetermined time period before and after the present in the data storage unit Mm in an overwrite-prohibited format. For example, the above data set is stored in the overwrite-prohibited area M2. When the processing of step S5 is completed, the process proceeds to step S8. Note that step S5 also corresponds to an example of a recording step.

[0099] In step S6, the abnormality detection unit G4 judges whether or not an abnormality occurs in the operation recording unit 30. If an abnormality occurs in the operation recording unit 30, step S7 is executed. On the other hand, if an abnormality does not occur in the operation recording unit 30, step S8 is executed. In step S7, the abnormality detection unit G4 outputs a signal indicating that the operation recording unit 30 is not operating normally to the LAN 99. Note that, when the automatic driving ECU 20 receives a signal indicating that the operation recording unit 30 is not operating normally, for example, via the LAN 99 during the autonomous driving mode, it executes an authority transfer process and transitions to an operation mode of level 2 or lower. When the HMI system 15 receives a signal indicating that the operation recording unit 30 is not operating normally, it displays and outputs the signal on the display 153 or the like. When the operation recording unit 30 is not operating normally, the HMI system 15 may notify the occupant that the autonomous driving mode is unavailable due to the event.

[0100] In step S6, it is determined whether the running power supply has been turned off. If the running power supply is turned off, this flow ends. On the other hand, if the running power supply is not turned off, that is, if the running power supply is still on, the process returns to step S1, and the processes from step S1 onwards are repeated. With this configuration, while the running power supply is on, various information according to the operating mode can be recorded in both the continuous recording method and the event recording method.

[0101] <Effects of the above configuration> In the above configuration, the items to be recorded are changed according to the driving mode. For example, whether or not data indicating the state of the driver's seat occupant, such as an image of the occupant's face, is recorded is changed according to the driving mode. With this configuration, the state of the driver's seat occupant is not always recorded. For example, in level 4 to 5 modes in which the driver's seat occupant does not incur driving responsibility, it is possible to exclude data indicating the state of the driver's seat occupant from the data to be recorded. Therefore, the amount of data to be saved can be reduced. Also, according to the above configuration, in level 3 mode, the face image of the driver's seat occupant captured by the DSM 12a is set as the data indicating the state of the driver's seat occupant to be recorded. This makes it possible to verify whether the driver's seat occupant was in a state where he or she could drive in an emergency. The level 3 mode corresponds to an example of a driving mode in which the driver's seat occupant incurs driving responsibility.

[0102] In addition, in the case of an accident involving a vehicle equipped with an autonomous driving function, it is necessary to be able to objectively analyze later whether the system or the driver was responsible for the driving at the time of the accident, and if it was the system, what kind of judgment was used to control it. In the above configuration, in the autonomous driving mode, the object recognition state and the contents of the scheduled action of the autonomous driving ECU 20 are also recorded. Therefore, it is possible to analyze how the autonomous driving ECU 20 recognized the outside world and what it was trying to do. Therefore, it is possible to improve the analytical ability of the cause of the accident during autonomous driving and the location of responsibility.

[0103] In level 3 mode, the image signal of the front camera 11a is also recorded. In addition, in level 4 mode, in addition to the image of the front camera 11a, observation data of other surrounding monitoring sensors 11, such as millimeter wave radar, is also recorded. With this configuration, the actual situation of the outside world can be identified. In addition, the recognition results and judgment contents (e.g. planned actions) of the autonomous driving ECU 20 are also recorded together with the observation data of the surrounding monitoring sensor 11, so that the actual situation of the outside world and the behavior of the autonomous driving ECU 20 can be compared and verified.

[0104] Furthermore, in levels 3 to 5 modes, the control amounts of the various driving actuators 18 by the autonomous driving ECU 20 are also set as targets for recording, along with the surrounding environment recognition state and judgment results. By sequentially recording the recognition, judgment, and control results of the autonomous driving ECU 20 as a set, it becomes easier to analyze the cause of recorded events such as accidents when they occur in autonomous driving mode.

[0105] In the above configuration, the observation data of the perimeter monitoring sensor 11 is not always stored. There are driving modes in which the observation data of the perimeter monitoring sensor 11 is included in the recording target and driving modes in which it is not included. Specifically, the image of the front camera 11a is recorded in the levels 3 to 5 modes and the authority transfer mode, but is not included in the recording target in the operation modes of the autonomous driving level 2 or lower. In addition, the image of the rear camera 11b, the image of the right side camera 11c, the image of the left side camera 11d, and the sensing data of the millimeter wave radar are included in the recording target in the level 4 to 5 modes, but are not recorded in the mode of level 3 or lower. With such a configuration, the amount of data stored as an operation recording device of the autonomous driving device can be further reduced.

[0106] In addition, in the authority transfer mode, the system's recognition result, such as eyes on / off, and observation data, such as facial images, are recorded as a set as the driver's seat occupant's status. With this configuration, the system's recognition result and the actual state of the driver's seat occupant can be compared and verified. Therefore, even if an accident occurs during authority transfer, it becomes easier to verify who is responsible. Even if the system's recognition result of the occupant's status and the driver's own claim are at odds, it is possible to confirm the facts based on facial image data.

[0107] Furthermore, the record target setting unit G2 includes the mode notification state in the record target in each operation mode. In particular, as the mode notification state, a screenshot of the display 153 set as the display destination of the driving mode notification image 80 is included in the record target. With this configuration, when the testimony of the driver's seat occupant differs from the system record regarding the driving mode at the time of the accident, it is possible to verify the credibility of the testimony of the driver's seat occupant.

[0108] Although the embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-mentioned embodiment, and various modified examples described below are also included in the technical scope of the present disclosure. Furthermore, in addition to the following, various modifications can be implemented within the scope of the gist. For example, the following various modified examples can be implemented in appropriate combinations within the scope of no technical contradiction. Note that the same reference numerals are used for components having the same functions as those described in the above-mentioned embodiment, and their description is omitted. In addition, when only a part of the configuration is mentioned, the configuration of the embodiment described above can be applied to the other parts.

[0109] <Additional information on the functions and configuration of the autonomous driving ECU 20> Although the above describes an example of an embodiment configured to be executable up to level 5, this is not limiting. The autonomous driving ECU 20 may execute autonomous driving up to level 3. In this case, the operation recording unit 30 may have a data set of recording targets up to level 3. The autonomous driving ECU 20 may have only modes corresponding to autonomous driving levels 0, 2, and 4. The operation mode of the autonomous driving ECU 20 may be three stages: a fully manual driving mode, a driving assistance mode, and an autonomous driving mode. The recording target setting memory unit Md may store recording target setting data for each driving mode that the autonomous driving ECU 20 can provide.

[0110] In addition, the autonomous driving ECU 20 (for example, the mode control unit F5) may include an ODD inside / outside determination unit F51 that sequentially determines whether the surrounding environment of the vehicle satisfies the ODD as shown in FIG. 13. Examples of the ODD include (a) that the road is an expressway or a motor vehicle-only road with two or more lanes in each direction and a central reservation strip and guardrails, and (b) that there are no traffic signals or pedestrians within the detection range of the surrounding monitoring sensor. Other examples of the ODD include (c) that the amount of rainfall is equal to or less than a predetermined threshold, (d) that there are no fallen objects or parked vehicles within a predetermined distance of the vehicle, and (e) that the surrounding monitoring sensor 11 including the on-board camera is operating normally. The conditions for determining whether autonomous driving is possible / impossible, in other words, the detailed conditions that define the ODD, can be changed as appropriate. In addition to the ODD for autonomous driving level 3, an ODD for autonomous driving level 4 may be set. For example, in terms of the number of surrounding monitoring sensors operating normally (in other words, redundancy) and the allowable position estimation error, an ODD for level 4 autonomous driving may have stricter conditions than an ODD for level 3 autonomous driving.

[0111] 13, when the autonomous driving ECU 20 includes an ODD inside / outside determination unit F51, the recording target setting unit G2 may set the determination result by the ODD inside / outside determination unit F51 and the sensor data that is the basis for the determination as recording targets in level 3 to 4 mode. The criteria for determining whether the ODD is satisfied are determined according to the contents of the ODD. For example, the criteria for determining whether the ODD is satisfied can be road type, road structure (e.g., number of lanes), weather information, sensing information from the perimeter monitoring sensor 11, the presence or absence of fallen objects, and the like.

[0112] Furthermore, if the ODD inside / outside determination unit F51 can calculate the remaining time until the ODD is exited (hereinafter, ODD exit remaining time), the recording target setting unit G2 may include the calculated value of the ODD exit remaining time in the recording target. For example, if a road section capable of autonomous driving is defined as the ODD, the remaining time until the vehicle exits the road section corresponds to the ODD exit remaining time. More specifically, if the ODD is limited to an expressway, the time until the vehicle exits the expressway corresponds to the ODD exit remaining time.

[0113] In addition, the recording target setting unit G2 always includes both the eyes on / off recognition result and the face image data in the recording target in the level 3 mode, while in the level 4 mode, it may set both to be recorded from the timing when the remaining time to exit the ODD becomes equal to or less than a predetermined threshold. In other words, if there is enough time before exiting the ODD, the face image data is not saved. With such a configuration, the amount of data saved can be reduced.

[0114] Furthermore, the recording processing unit G3 may adopt as a recording event that the automatic driving ECU 20 (specifically, the ODD inside / outside determination unit F51) has determined that the vehicle has left the ODD or will leave within a predetermined time. Specifically, in the level 3 / 4 mode, when the ODD inside / outside determination unit F51 has determined that the vehicle has left the ODD or will leave within a predetermined time, the recording processing unit G3 may record the determination basis data, which is the material used for the determination, in the data storage unit Mm. The determination basis data may be acquired by the automatic driving ECU 20. It is preferable that the determination basis data is saved in the overwrite-prohibited area M2. It is preferable that the determination basis data is saved as a set with the determination result, the determination time, and the position information.

[0115] Alternatively, the operation recording unit 30 may be included in the automatic driving ECU 20. In other words, the automatic driving ECU 20 may include an operation recording section F30 equivalent to the operation recording unit 30, as shown in FIG.

[0116] <Additional information about perimeter monitoring sensor 11> The mounting positions of the perimeter monitoring camera are not limited to four positions, front, rear, left and right. As shown in FIG. 15, the perimeter monitoring camera may be mounted at six positions, namely, the front end, rear end, left and right side mirrors, the upper end of the interior side of the windshield, and the interior surface of the rear glass. Four perimeter monitoring cameras with optical axes directed to the right diagonally forward, left diagonally forward, right diagonally rear, and left diagonally rear may also be mounted on the roof. Similarly, a large number of millimeter wave radars may also be provided. In addition, the autonomous driving system 1 may include, as the perimeter monitoring sensor 11, a LiDAR whose detection range includes the front of the vehicle and a LiDAR whose detection range includes the rear of the vehicle. The LiDAR may be mounted on the roof, for example. The autonomous driving system 1 may also include, as the perimeter monitoring sensor 11, a plurality of sonars each having a different detection range and installation position. The sonars may be mounted on the bumper, side sill, etc.

[0117] <Additional information on the recording method of observation data from the perimeter monitoring sensor 11> The combination of perimeter monitoring sensors 11 that record observation data in the autonomous driving mode may be changed depending on the driving scene, the action being performed, and the planned action. Observation data of the perimeter monitoring sensors 11 whose detection range includes a direction that requires attention in the action being performed / planned by the autonomous driving ECU 20 may be included in the recording target, while observation data of the perimeter monitoring sensors 11 whose detection direction is a direction that requires less attention may be excluded from the recording target. Such a configuration is particularly suitable for a vehicle equipped with a large number of perimeter monitoring cameras (e.g., eight or more), as shown in FIG. 15. This is because the more perimeter monitoring sensors 11 there are, the larger the amount of data becomes.

[0118] In the following description, the forward sensor refers to a perimeter monitoring sensor 11 whose detection range includes the front. The forward sensor may include a sensor whose detection range is diagonally forward to the left or right. The perimeter monitoring sensor 11 whose main detection range is the rear is also referred to as a rear sensor. The perimeter monitoring sensor 11 whose main detection range is the rear may be a sensor whose central axis of the detection range is oriented in a direction within 45 degrees from the rear of the vehicle. The rear sensor may include a sensor whose detection range is diagonally backward to the left or right (e.g., right rear radar 11f). The perimeter monitoring sensor 11 whose main detection range is the left / right side is also referred to as a side sensor. The perimeter monitoring sensor 11 whose main detection range is the side may be a sensor whose central axis of the detection range is oriented in a direction within 45 degrees from the left / right direction.

[0119] FIG. 16 illustrates an example of a combination of the perimeter monitoring sensors 11 that record observation data for each scheduled behavior. As shown in FIG. 16, for example, while the vehicle is autonomously traveling along a road, the observation data of the front sensors may be recorded, while the observation data of the rear sensors may be excluded from the recording. The observation data of the side sensors may be recorded at a lower resolution or sampling rate than the front sensors. The reason for including the observation data of the side sensors in the recording is that there is a demand to record actions such as cutting in from an adjacent lane and pulling over to the side. Based on the same technical idea, the observation data of the rear sensors may be recorded at a lower resolution or sampling rate than the observation data of the front sensors even when the vehicle is autonomously traveling along a road.

[0120] When the autonomous driving ECU 20 plans to automatically change lanes to the right, it may include in the recording target the observation data of the periphery monitoring sensor 11 whose detection range includes a range from the right diagonal front to the right diagonal rear, while excluding from the recording target the observation data of the other periphery monitoring sensors 11. When the autonomous driving ECU 20 plans to automatically change lanes to the right, it may include in the recording target the observation data of the periphery monitoring sensor 11 whose detection range includes a range from the left diagonal front to the left diagonal rear, while excluding from the recording target the observation data of the other periphery monitoring sensors 11.

[0121] In addition, when the autonomous driving ECU 20 executes deceleration for stopping or plans it as a scheduled action, the observation data of the front and rear sensors may be recorded, while the observation data of the side sensors may be excluded from the recording. When a right turn / left turn at an intersection is executed or planned as a scheduled action, the observation data of the front and side sensors may be recorded, while the observation data of the rear sensors may be excluded from the recording. When driving, the observation data of the perimeter monitoring sensors 11 that is not related to the moving direction may be excluded from the recording, while when automatic parking or automatic leaving is executed or planned as a scheduled action, the observation data of all the perimeter monitoring sensors 11 may be included in the recording. Note that when MRM is being implemented, the observation data of all the perimeter monitoring sensors 11 may be included in the recording regardless of the moving direction, etc.

[0122] According to the above configuration, the observation data of the perimeter monitoring sensors 11 whose detection range includes a direction requiring attention in an action being performed or planned is included in the recording target, while the observation data of the other perimeter monitoring sensors 11 is excluded from the recording target or is stored with a reduced amount of information. By excluding the observation data of some of the perimeter monitoring sensors 11 from the recording target in this way, the amount of data stored can be reduced. The reason that the observation data of the front system sensors is included in the recording target when decelerating is that the reason for deceleration may be ahead.

[0123] In addition, according to the above configuration, since the observation data for the direction requiring attention is included in the recording target, data showing objective facts in the direction can be left. Therefore, it is possible to ensure the ability to perform post-analysis while suppressing the amount of data storage. The direction requiring attention corresponds to a direction with a high risk of collision or a direction in which the vehicle is about to move.

[0124] In the above configuration, the recording target setting unit G2 may include the recognition results of the autonomous driving ECU 20 in the direction in which the observation data is not included in the recording target. For example, when traveling along a road, the observation data of the rear system sensors may not be recorded, but the recognition results of objects present behind the vehicle based on the data may be included in the recording target. The recognition results of the object may include, for example, the tagging status, type, and tracking status of the object in addition to the position information, speed, and moving direction. Even if the observation data of a certain surrounding monitoring sensor 11 is excluded from the recording target, the recognition status of the autonomous driving ECU 20 based on the surrounding monitoring sensor 11 is included in the recording target, so that the general situation can be grasped.

[0125] Furthermore, in the autonomous driving mode, it is preferable that the recording target setting unit G2 sets the observation data of the periphery monitoring sensor 11 in the direction related to the action being performed (hereinafter, current action) and the planned action, and the object recognition result (tagging state) in the direction as a set of recording targets. For example, in the case of a lane change, the direction related to the current action and the planned action refers to the diagonal front to diagonal rear on the lane change side. Also, the direction related to deceleration refers to the rear. The direction related to parking can be, for example, all directions. The direction related to right and left turns can be the front and the side. With this configuration, it is possible to verify how the autonomous driving ECU 20 recognized the direction based on the sensing information of the direction to be careful of.

[0126] The above configuration corresponds to a configuration in which the autonomous driving ECU 20, in the autonomous driving mode, includes data indicating the behavior of other vehicles present in the direction corresponding to the planned behavior of the vehicle in the recording target. The data indicating the behavior of the surrounding vehicles refers to, for example, the observation data of the surrounding monitoring sensor 11 and the recognition result of the autonomous driving ECU 20.

[0127] <Additional information regarding recording subjects> As shown in FIG. 17, if the vehicle is equipped with a lamp (hereinafter, driving mode notification lamp 51) for notifying the outside of the driving mode as a lighting device, the record target setting unit G2 may set the lighting state of the driving mode notification lamp 51 as a recording target in the autonomous driving mode. With such a configuration, it becomes possible to verify whether the autonomous driving mode has been properly notified to the outside. The driving mode notification lamp 51 may be a device that indicates whether the autonomous driving mode is in the autonomous driving mode or the occupant participation mode by the presence or absence of lighting and the lighting color. The driving mode notification lamp may be controlled by the body ECU 17 based on the status information output by the autonomous driving ECU 20.

[0128] In addition, when the vehicle is equipped with an external communication device 52, the record target setting unit G2 may include the operating state of the external communication device 52 in the record target. The external communication device 52 is a projector that projects an image for communicating with pedestrians on the road surface or a window glass. The external communication device 52 is installed, for example, in a headlamp or on a ceiling part (for example, near the upper end of a window frame part) in a position where the irradiated light hits the road surface or the window glass. The operating state of the external communication device 52 may include an image displayed on the road surface or the window glass. Assumed images for communicating with pedestrians include an image for the purpose of giving way to pedestrians, an image showing the moving direction of the vehicle, an image showing a target parking position, an image showing that the vehicle will be transmitting soon, an image showing gratitude to the presence that has given way, and the like. According to the above configuration, it is possible to verify whether communication between the autonomous vehicle and pedestrians has been properly performed.

[0129] In the autonomous driving mode, the recording target setting unit G2 may set the identification information of other vehicles traveling around the vehicle as the recording target. The identification information of the surrounding vehicles may be, for example, the vehicle ID of the other vehicles that are performing vehicle-to-vehicle communication with the vehicle. The identification information of the surrounding vehicles may also be the vehicle number written on the license plate. The vehicle number can be identified by image recognition of the license plate. When a recording event such as an accident occurs in the vehicle, information indicating the behavior of the vehicle may be recorded in the vehicle recording device of the surrounding vehicle. By including the identification information of the surrounding vehicles in the recording target, the vehicles that were in the vicinity at the time of the accident can be identified. In addition, it is expected that objective data indicating the cause of the accident can be collected from the recording devices of the other vehicles that were in the vicinity of the vehicle.

[0130] The recording target setting unit G2 may also include data indicating the behavior of surrounding vehicles in the recording target. Examples of the behavior of surrounding vehicles include acceleration, overtaking, lane changing, stopping, and deceleration. In addition, dangerous actions such as sideways driving and tailgating may also be recorded. Regarding the configuration in which the information of surrounding vehicles is included in the recording target, the direction to be recorded may be limited to the direction related to the current action and the planned action. For example, when a lane change to the right is planned, information of other vehicles traveling in the lane in which the vehicle is traveling and the lane to the right of the vehicle may be included in the recording target, while information of other vehicles traveling in the lane to the left of the vehicle may be excluded from the recording target. With such a configuration, it is possible to reduce the amount of data to be stored while recording information that is useful in the event of an accident.

[0131] In addition, when the operation recording unit 30 detects an accident / traffic violation of another vehicle during autonomous driving, the operation recording unit 30 may report the observation data of the surrounding monitoring sensor 11 for a predetermined time before and after the accident to a predetermined center. Whether or not another vehicle traveling in the vicinity is an autonomous vehicle can be determined based on vehicle-to-vehicle communication or the lighting state of the driving mode notification light 51. In addition, whether or not another vehicle is an autonomous vehicle may be determined by image recognition of a sticker indicating that the vehicle is an autonomous vehicle. The center to which traffic violations and the like are reported is, for example, a center managed by the vehicle manufacturer. According to the above configuration, information on accidents and traffic violations caused by autonomous vehicles is collected by the vehicle manufacturer, so that the vehicle manufacturer can understand under what circumstances the autonomous vehicle has committed an accident or traffic violation. As a result, the improvement of the autonomous driving system 1 can be promoted. In particular, ignoring stop lines and traffic lights is due to a misrecognition of the surrounding environment, and it is assumed that the autonomous vehicle itself that has committed a traffic violation cannot recognize the fact. In light of such circumstances, it is highly useful to report the traffic violation of an autonomous vehicle by another vehicle to the center.

[0132] Furthermore, the recording processing unit G3 may adopt, as a recorded event, the fact that the autonomous driving ECU 20 has requested the driver to take over the driving operation. Specifically, when the autonomous driving ECU 20 has requested the driver to take over the driving operation, the recording processing unit G3 may record handover factor data indicating the reason for the handover request in the data storage unit Mm. The handover factor data indicating the reason for the handover request is acquired from the autonomous driving ECU 20. The handover factor data is preferably saved in the overwrite-prohibited area M2. The handover factor data is preferably saved as a set with data indicating the determination time, position information, and the driver's seat occupant state.

[0133] Furthermore, as described above, the recording processing unit G3 may adopt the execution of MRM by the autonomous driving ECU 20 as a recording event. Specifically, when the autonomous driving ECU 20 executes MRM, the recording processing unit G3 may store data indicating the reason for the execution of MRM and the observation data of the periphery monitoring sensor 11 during the execution of MRM in the overwrite-prohibited area M2. In addition, it is preferable to include path plan data such as the stopping target position and deceleration set during MRM driving as the recording target. The above various data are preferably stored as a set with time information, position information, and data indicating the driver's seat occupant state.

[0134] In the autonomous driving mode, the recording target setting unit G2 may set at least one of the positional relationship with a traffic light, weather information, and road surface conditions as the recording target. The positional relationship with a traffic light may be specified based on high-precision map data. In addition, when a recording event occurs in or near an intersection, the recording processing unit G3 may store a set of the video signal of the front camera 11a for a predetermined time before and after the event occurrence and data indicating the recognition result of the lighted state of the traffic light in the overwrite-prohibited area M2. With this configuration, it is possible to verify whether the cause of the accident is the erroneous recognition of the lighted state of the traffic light by the autonomous driving ECU 20. Note that the vicinity of the intersection refers to within a predetermined distance (for example, 10 m) from the intersection.

[0135] The recording processing unit G3 may store data indicating whether the in-vehicle software is the latest in the overwrite-prohibited area M2 at a predetermined timing. Verification of whether the software related to the autonomous driving is the latest may be realized, for example, by the autonomous driving ECU 20 communicating with the center via the V2X in-vehicle device 16. Verification of whether the software is the latest may be performed when the driving power source is turned on or when the driving power source is turned off in response to an instruction from the occupant. Recording processing by the recording processing unit G3 on the verification result of whether the software is the latest may be performed, for example, when the above verification processing is completed. In order to use the autonomous driving mode, the driver is required to maintain the vehicle in a state in which it can perform its original functions. According to the above configuration, it is possible to verify whether the driver has properly updated the in-vehicle software.

[0136] When a signal indicating that the object detection capability is reduced due to dirt such as mud or bird droppings adhering to the sensor surface is input from the periphery monitoring sensor 11 during the autonomous driving mode, the autonomous driving ECU 20 may cooperate with the HMI system 15 and notify the occupant of the reduction. In such a configuration, the record target setting unit G2 may record whether the occupant responded appropriately to the above notification. An appropriate response by the occupant to the above notification refers to, for example, switching to an operation mode that can be used without using the periphery monitoring sensor with reduced detection capability, or temporarily stopping the vehicle to wipe off the sensor dirt. Switching to an operation mode that can be used without using the periphery monitoring sensor 11 with reduced detection capability refers to, for example, switching to an operation mode with a lower automation level. The HMI system 15 may present the occupant with an operation mode / automation level that can be used without using the periphery monitoring sensor 11 with reduced detection capability. According to the above configuration, it is possible to verify whether the autonomous driving mode was used in an unstable state of the system.

[0137] The interval at which data is saved by the recording processing unit G3 may be adjusted according to the vehicle speed. For example, the interval may be every 50 milliseconds during high-speed driving, every 100 milliseconds during normal driving, and every 200 milliseconds during low-speed driving. Low-speed driving can be defined as a state in which the vehicle speed is equal to or lower than a predetermined first speed (e.g., 30 km / h). Normal driving can be defined as a state in which the vehicle speed is higher than the first speed and equal to or lower than a predetermined second speed (e.g., 60 km / h). High-speed driving can be defined as a state in which the vehicle speed is higher than the second speed. Since the travel distance per unit time is smaller as the vehicle speed is lower, there is little risk of the post-analysis performance decreasing even if the saving interval is set sparse. Also, by providing a mode in which the saving interval is set sparse, the amount of data saved can be suppressed. The first speed may be 10 km / h, 20 km / h, or the like. The second speed may be 80 km / h, 100 km / h, or the like.

[0138] The storage quality of the observation data (mainly image data) of the perimeter monitoring sensor 11 may also be changed according to the vehicle speed, based on the same technical idea as above. For example, the higher the vehicle speed, the higher the quality of the data stored. Also, the lower the vehicle speed, the lower the resolution or frame rate may be reduced before the data is stored. The degree of reduction in resolution may be determined according to the vehicle speed. The degree of reduction in frame rate may also be set to be greater the lower the vehicle speed.

[0139] The interval at which data is saved by the recording processing unit G3 in the autonomous driving mode may be adjusted according to the difficulty of the action being performed or planned. For example, when following the road, the saving interval may be set to be sparser than when performing complex control such as lane change. However, when turning right or left at an intersection, the saving interval may be set to be sparser than when going straight at the intersection. The difficulty of the action may be set higher in the order of following the road, changing lanes, going straight at an intersection, and turning right or left at an intersection in a road section without an intersection, for example. When following the road, the section may be divided into a sharp curve with a curvature equal to or greater than a predetermined threshold, or a straight section. The difficulty of the action may be changed depending on whether or not a preceding vehicle exists. According to the above configuration, when performing an action (in other words, a control) with a high degree of difficulty, various data are saved relatively densely. Therefore, it is possible to improve the post-mortem analysis in the event of an accident or the like. According to the above configuration, when performing a control with a low degree of difficulty, various data are saved relatively sparsely, so the amount of data saved can be reduced.

[0140] Regarding the recording of driver's seat occupant status data, in an operation mode in which the system is responsible for driving, only the system recognition result such as eyes on / off is recorded, whereas in an operation mode in which the driver's seat occupant is responsible for driving, facial image data may also be included in the recording target. With the above configuration, the amount of data stored can be reduced while maintaining the ability to analyze the cause of an accident. Note that an operation mode in which the system is responsible for driving refers to, for example, level 4 / 5 mode. An operation mode in which the driver's seat occupant is responsible for driving refers to level 0 to 3 mode. The driver's seat occupant may be responsible for driving during MRM. While the handover request process is being performed, facial image data may also be included in the recording target regardless of the operation mode.

[0141] When storing data in the data storage unit Mm, the recording processing unit G3 may add verification information to prevent tampering and store the data. For example, the verification information may be a hash value of the data to be stored. The hash value may be generated using a predetermined hash function such as SHA-256.

[0142] The conditions under which the recording processing unit G3 executes the event recording process may be changed according to the surrounding environment and the vehicle type. For example, the recording conditions in fine weather may be set to be looser than the recording conditions in rainy weather. In addition, the recording thresholds for acceleration, TTC, MTC, etc. may be different between a case where the system-equipped vehicle as the vehicle is a passenger car and a case where the system-equipped vehicle is a large vehicle such as a truck or trailer. For example, since a large vehicle such as a truck or trailer requires a relatively long time to brake, it is preferable that the recording thresholds for TTC and MTC are set to larger values. By changing the recording conditions according to the surrounding environment and the vehicle type, it is possible to prevent unnecessary data from being stored.

[0143] In addition, in the operation modes of level 3 or higher, the recording processing unit G3 may change the density of the information to be stored (e.g., storage interval, data compression rate, etc.) according to the system stability. The system stability here is the control margin relative to the limit of the autonomous driving system 1. The autonomous driving ECU 20 may evaluate the system stability based on the output of various sensors and the recognition results of the surrounding environment. For example, the lower the system stability calculated by the autonomous driving ECU 20, the denser the storage interval is. Also, the higher the system stability calculated by the autonomous driving ECU 20, the higher the compression rate of the image data output by the surrounding monitoring camera may be. According to such a configuration, for example, the amount of information around the moment when the mode falls from level 4 to level 3 or around the moment when the mode shifts from autonomous driving to manual driving becomes large, improving the analyticity of accidents, etc.

[0144] The operation record unit 30 may include a recorded data management section G5 that manages data stored in the data storage section Mm as shown in FIG. 18. The recorded data management section G5 preferentially deletes other old data when the storage capacity becomes full, so that data related to the captured images of the front camera 11a remains for, for example, six months. The captured images of the front camera 11a for the most recent month may be stored in a format equivalent to the observation data, while data older than that may be thinned out and stored by lowering the frame rate or resolution, for example. For example, a FullHD image as the observation data generated by the front camera 11a may be converted into a captured image of VGA (Video Graphics Array) with a lower resolution when the storage period has elapsed for one month. This configuration reduces the risk of the storage capacity of the data storage section Mm becoming full. The format equivalent to the observation data includes a storage format that maintains a predetermined quality. The storage format of the observation data of the perimeter monitoring sensor 11 may not be completely the same as the output format of the front camera 11a, provided that the quality and data restorability are guaranteed. The output format of the observation data from the perimeter monitoring sensor 11 may be different from the storage format of the observation data by the recording processing unit G3. For example, the original data may be compressed and stored in a format that allows the data to be restored. Note that various formats such as HEVC (H.264) and HEVC (H.265) can be used as the storage format for the video data generated by the perimeter monitoring camera. HEVC is an abbreviation for High Efficiency Video Coding.

[0145] When the recording target setting unit G2 includes the recognition result of the system in the recording target (for example, in the autonomous driving mode), the recording target may include information on the application that was running in the system and information on the version of the software that was used. In other words, the operation recording unit 30 may be configured to record information on the software operating in the autonomous driving ECU 20 or the like sequentially or at a predetermined timing in the autonomous driving mode. This is because the recognition logic and the judgment logic may differ depending on the software version. Note that the software that was running in the system includes software that was running inside the devices that configure the autonomous driving system 1, such as the autonomous driving ECU 20 and the surroundings monitoring sensor 11.

[0146] It is preferable that the type information of the application running in the system is recorded sequentially or when the operating state of the application changes. The version information of the software used by the system, such as recognition, may be recorded when a software update is implemented or when a predetermined recording event occurs. The software version information may be recorded for each trip or for each predetermined period (e.g., 24 hours). A trip here refers to the driving from when the driving power source is turned on to when it is turned off.

[0147] According to the above configuration, the recognition result and the control instruction contents of the autonomous driving ECU 20 are recorded in association with the version information of the recognition software used at that time. Therefore, when a recording event occurs, for example, it is possible to perform a cause analysis including what software was operating / used. Note that the above information may be included in the recording target when a recording event such as an accident occurs.

[0148] As shown in FIG. 19, multiple pieces of record target data can be classified and stored for each device that serves as the information source. Each piece of data may be assigned an individual time stamp. However, if multiple elements occur simultaneously, various pieces of data may be recorded collectively with one time stamp. For example, as shown in FIG. 20, the operating states of multiple types of devices at each time may be grouped and stored by time. By assigning the same time stamp to data that occurs / acquired at the same time and storing them in a group, it becomes easier to analyze the interrelationships / interactions between elements.

[0149] As described above, the management mode of the stored data may be for each device or for each time. The recording processing unit G3 may also store various types of data to be recorded by assigning an event ID for each recording event. For example, various types of stored data may be classified for each recording event, and data having a common occurrence / acquisition time may be grouped with the same time stamp. For convenience, time information corresponding to a time stamp is expressed in terms of year / month / day / hour / minute / second in Figs. 19 and 20, but is not limited to this. The time stamp may be expressed in epoch seconds. The time stamp may be expressed in Universal Time Coordinated (UTC). The time stamp may also include data of a time finer than one second (for example, several milliseconds).

[0150] In addition, the contents of the record target setting data D1 stored in the record target setting storage unit Md may be changed based on, for example, an instruction from a server. The record target setting data D1 may also be configured as part of software so as to be updatable by wireless communication with the server. Furthermore, the contents of the record target setting data D1 may be configured so as to be changeable by an operation of a driver's seat occupant within a predetermined allowable range.

[0151] The operation recording unit 30 may upload data from a certain time ago or more to a server and delete the data from the data storage unit Mm. Also, a storage area of ​​an external server may be used as part or all of the data storage unit Mm. For example, the overwrite-prohibited area M2 may be realized using a cloud storage. The process of wirelessly transmitting the data to be recorded to a cloud server / cloud storage serving as the data storage unit Mm in cooperation with the V2X vehicle-mounted device 16 is also included in the process of sequentially saving the data to be recorded in the data storage unit Mm.

[0152] If the vehicle is a remote-controlled vehicle, the remote operator corresponds to the driver's seat occupant. For example, if the vehicle equipped with the system is a robot taxi / unmanned bus equivalent to level 3, the operator corresponds to the driver's seat occupant.

[0153] <Supplementary information on system configuration example> Various configurations can be adopted as the configuration of the autonomous driving system 1. For example, as shown in FIG. 21, the autonomous driving system 1 may be configured such that a plurality of ECUs 41-46 execute recognition processing based on observation data from the surrounding monitoring sensor 11 associated with the autonomous driving system and output the same to a LAN 99. For example, the autonomous driving system 1 may be configured to include a front camera ECU 41, a surrounding monitoring camera ECU 42, a front radar ECU 43, a rear side radar ECU 44, a sonar ECU 45, and a LiDAR-ECU 46, and each of the ECUs 41-46 executes recognition processing based on observation data from the surrounding monitoring sensor 11 associated with the autonomous driving system. In this case, the autonomous driving ECU 20 can be configured to acquire recognition results from each ECU, integrate (so-called fusion), and execute control judgment.

[0154] The front camera ECU 41 is an ECU that performs recognition processing on the image captured by the front camera 11a. The periphery monitoring camera ECU 42 is an ECU that performs recognition processing on the image captured by the periphery monitoring camera 11h such as the rear camera 11b. The front radar ECU 43 is an ECU that recognizes the position, relative speed, type, etc. of an object present in front of the vehicle by analyzing the distance measurement result of the front radar 11e, that is, the reception strength for each detection direction and distance. The rear side radar ECU 44 is an ECU that recognizes the position, relative speed, type, etc. of an object present in the rear side by analyzing the distance measurement result of the rear side radar 11i. The rear side radar 11i includes the right rear radar 11f and the left rear radar 11g. The sonar ECU 45 is an ECU that estimates the position of an object, and more preferably, its height and type, by analyzing the distance measurement result of the sonar 11j. The LiDAR-ECU 46 is an ECU that identifies the position, type, etc. of an object present within a detection range by analyzing the three-dimensional point cloud data (in other words, range image data) generated by the LiDAR 11k. Each of the ECUs 41 to 46 may be configured integrally with a sensor.

[0155] Furthermore, the autonomous driving system 1 may be configured such that the autonomous driving ECU 20 acquires observation data from each surrounding monitoring sensor 11 and executes recognition processing, fusion processing, and control decisions based on each observation data, as shown in Fig. 22. Alternatively, the configuration for performing the recognition processing may be separated into an image system ECU 47 that collectively performs recognition of image data such as cameras and LiDAR, and a distance measurement system ECU 48 that generally performs recognition based on distance measurement data, as shown in Fig. 23. The image system ECU 47 and the distance measurement system ECU 48 may be built into the autonomous driving ECU 20.

[0156] In addition, the configuration of the operation recording unit 30 can be changed as appropriate, and for example, it may be divided into multiple units. For example, as shown in FIG. 24, the operation recording unit 30 may be arranged such that the video recording unit 30a for recording camera images such as the front camera 11a and the recognition results based on the images is arranged independently from the units for recording data other than the above. For convenience, a unit for recording data that is not recorded by the video recording unit 30a is described as a non-video recording unit 30b. The video recording unit 30a may be integrated with, for example, a navigation device or the front camera 11a. The video recording unit 30a and the non-video recording unit 30b are configured to distribute the functions of the operation recording unit 30 described above, and the specific configuration and operation can be referred to the description of the operation recording unit 30. Note that, in a configuration in which the operation recording unit 30 is distributed and arranged in multiple units, it is preferable that the time information of each unit is synchronized by transmitting and receiving a synchronization signal (for example, a pulse signal, etc.). For example, it is preferable that the video recording unit 30a and the non-video recording unit 30b are configured to synchronize with each other by transmitting and receiving a synchronization signal. This is because the visual recording unit 30a and the non-visual recording unit 30b can add time stamps using common time information.

[0157] <Additional remarks (part 1)> The control unit and the method thereof described in the present disclosure may be realized by a dedicated computer constituting a processor programmed to execute one or more functions embodied in a computer program. The above-mentioned processing units 21 and 31 may be realized by combining multiple types of processors such as a CPU, an MPU, a GPU (Graphics Processing Unit), a DFP, and an NPU. Furthermore, for example, some of the functions to be provided by the processing units 21 and 31 may be realized using an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like.

[0158] The apparatus and method described in the present disclosure may be realized by a dedicated hardware logic circuit. The apparatus and method described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. The computer program is stored in a computer-readable non-transitive tangible storage medium as instructions to be executed by the computer. Various storage media such as a hard-disk drive (HDD), a solid-state drive (SSD), an erasable programmable ROM (EPROM), a flash memory, a USB memory, and a secure digital (SD) memory card can be used as the storage medium for the program.

[0159] <Additional remarks (part 2)> The present disclosure also includes the following technical ideas. Note that the autonomous driving processing unit described below corresponds to the autonomous driving ECU 20.

[0160] [Technical philosophy (1)] A vehicle recording device for recording the operating state of an automatic driving processing unit (20), which is a computer that provides the automatic driving function and is used in a vehicle equipped with an automatic driving function, comprising: An information acquisition unit (G1) that acquires multiple types of information that can be recorded from sensors mounted on a vehicle; An operation mode acquisition unit (G11) that acquires a signal indicating an operation mode corresponding to an operation state of the automatic operation processing unit; A recording target setting unit (G2) that changes the recording target depending on the operation mode; A recording processing unit (G3) that executes a process of storing information corresponding to the recording target set by the recording target setting unit in a data storage unit (Mm) realized using a predetermined storage medium, The setting of the record target for each driving mode is configured to be changeable by wireless communication with a predetermined server. With this configuration, the record target for each driving mode can be changed at any time. Therefore, it is possible to flexibly respond to amendments to laws and regulations.

[0161] [Technical philosophy (2)] When the autonomous driving processing unit is in autonomous driving mode and deceleration is planned as a scheduled action, the recording target setting unit sets the observation data of the surrounding monitoring sensor with the rear as the detection direction and the environmental recognition result of the autonomous driving processing unit for the rear as the recording target. According to this configuration, while information in the forward / backward direction where there is a possibility of a collision is recorded as raw data, the observation data in the left / right direction where there is a low risk is excluded from the recording target. Therefore, it is possible to reduce the amount of data stored while maintaining the ability to perform post-mortem analysis.

[0162] [Technical philosophy (3)] When the autonomous driving processing unit is in autonomous driving mode and a right or left turn at an intersection is planned as a scheduled action, the recording target setting unit sets the observation data of the surrounding monitoring sensor with the forward and left / right detection directions and the environmental recognition results of the autonomous driving processing unit for the forward and left / right directions as the recording targets. According to this configuration, raw data for the forward and left / right directions, which may cause a collision with an oncoming vehicle or a pedestrian crossing the road, is recorded, while rearward observation data, which is less dangerous, is excluded from the recording targets. Therefore, it is possible to reduce the amount of data stored while maintaining the ability to perform post-mortem analysis.

[0163] [Technical philosophy (4)] The autonomous driving processing unit executes a process of notifying a driver in cooperation with the HMI system that the detection capability of the surroundings monitoring sensor is degraded when a signal indicating that the detection capability of the surroundings monitoring sensor is degraded is input from the surroundings monitoring sensor, The recording processing unit records whether or not the driver has, in response to the notification, performed an operation to switch the driving mode to an operation mode that can be used without using the sensor with the reduced detection capability within a predetermined time. With this configuration, it is possible to verify whether the driver has used the automatic driving device in a state in which the original functions are exerted.

[0164] [Technical philosophy (5)] The recording processing unit stores data indicating whether the software of the automatic driving processing unit is up to date or not in the data storage unit based on whether the driving power source is turned on or off. With this configuration, it is possible to verify whether the driver's seat occupant is using the automatic driving device in a state where the original functions are being performed.

[0165] [Technical philosophy (6)] The data storage unit includes an overwriteable area (M1) from which data can be deleted, and an overwrite-prohibited area (M2) from which data cannot be deleted, The recording processing unit includes: The data set as a record target by the record target setting unit and acquired by the information acquisition unit is stored in an overwritable area, When a predetermined recording event occurs, data acquired within a predetermined time from the time of the occurrence is stored in an overwrite-prohibited area. The recording events include exiting the ODD, executing the MRM, lowering the autonomous driving level, and override by the driver's seat occupant. This configuration reduces the risk of data indicating the situation at the time the event occurs being deleted.

[0166] [Technical philosophy (7)] The recording processing unit adds a hash value for preventing tampering to the data set as the recording target and stores the data. According to the above configuration, it is possible to reduce the risk of part of the stored data being tampered with. Alternatively, if the stored data has been tampered with illegally, it is possible to detect this fact.

[0167] [Technical philosophy (8)] The observation data of the perimeter monitoring sensor includes an image captured by a front camera (11a) that captures an image of the area in front of the vehicle. The data management unit (G5) manages the data stored in the data storage unit so that data related to the image captured by the front camera in the autonomous driving mode is stored for a predetermined period. The predetermined period can be set to six months or three months. This configuration makes it easier for data showing the forward situation during autonomous driving to be retained.

[0168] [Technical philosophy (9)] The data management unit stores the image data captured by the forward camera for the most recent month in a data format that maintains observation data or a specified resolution, while storing data captured earlier than that in a format with a lower resolution or frame rate. This configuration makes it possible to reduce the amount of data stored. In addition, since the most recent data that is highly useful is stored in a format with a large amount of information, it is possible to reduce the risk of compromising the analytic ability of the cause of the accident, etc.

[0169] [Technical philosophy (10)] The autonomous driving processing unit has at least one of an autonomous driving mode, a level 3 mode corresponding to an automation level 3, a level 4 mode corresponding to an automation level 4, and a level 5 mode corresponding to an automation level 5, and in the autonomous driving mode, evaluates the control margin of the autonomous driving function based on the output of an on-board sensor, The recording processing unit changes the density of information stored in the data storage unit according to the control margin. The control margin corresponds to the stability of the system. With this configuration, when the control margin is low, the amount of data stored increases, improving the ability to perform post-mortem analysis. On the other hand, when the control margin is high, the amount of data stored can be reduced.

[0170] [Technical philosophy (11)] In the vehicle recording device described in the above Technical Concept (10) section, when the control margin is equal to or greater than a predetermined threshold, data is acquired and stored at a predetermined first storage interval, whereas when the control margin is less than the predetermined threshold, data is acquired and stored at a second storage interval that is shorter than the first storage interval.

[0171] [Technical philosophy (12)] An automatic driving device that provides an automatic driving function for autonomously driving a vehicle, Equipped with multiple operating modes with different levels of automation of driving operations, a surrounding environment recognition unit (F1) that recognizes the surrounding environment including the positions of other moving objects based on observation data of a surrounding monitoring sensor; A mode control unit (F5) that switches an operation mode based on at least one of a recognition result of the surrounding environment recognition unit and an operation by an occupant in the driver's seat; an action planning unit (F6) that generates an action plan including a next action to be performed based on a recognition result of the surrounding environment recognition unit when the vehicle is in an autonomous driving mode that causes the vehicle to autonomously drive; a control execution unit (F7) that determines a control amount of the traveling actuator based on the action plan generated by the action planning unit and outputs a control signal to the traveling actuator; and an operating status output unit (F9) that outputs at least two of the observation data of the perimeter monitoring sensor, the recognition result of the surrounding environment recognition unit, the planned action indicating the next action to be performed planned by the action planning unit, and the control amount of the traveling actuator determined by the control execution unit to a recording device (30) as a data set for recording.

[0172] [Technical philosophy (13)] An automatic driving device according to the technical concept (12), When in autonomous driving mode, the operating status output unit outputs to a recording device observation data from the surrounding monitoring sensor in a direction related to the planned behavior and the recognition results by the environment recognition unit in the direction related to the planned behavior as a data set for recording.

[0173] [Technical philosophy (14)] An automatic driving device as described in the technical concept (13) column, When in autonomous driving mode, when a lane change to the right or left is planned as a scheduled action, the operation status output unit outputs to a recording device the observation data of the surrounding monitoring sensor whose detection range includes at least a portion of the diagonally forward, side, and diagonally rearward in the direction of the lane change, and the recognition results of the surrounding environment recognition unit for the above directions as a data set for recording.

[0174] [Technical philosophy (15)] A vehicle recording device as described in the technical concept (13) or (14), When in autonomous driving mode and deceleration is planned as a scheduled action, the operation status output unit outputs the observation data from the surrounding monitoring sensor with the rear as the detection direction and the recognition results of the surrounding environment recognition unit to the rear as a data set for recording to a recording device.

[0175] [Technical philosophy (16)] A vehicle recording device according to any one of technical concepts (13) to (15), When in autonomous driving mode, when a right or left turn at an intersection is planned as a planned action, the operation status output unit outputs to a recording device the observation data of the surrounding monitoring sensor, which has detection directions in the front and to the left and right, and the recognition results of the surrounding environment recognition unit in the front and to the left and right, as a data set for recording.

[0176] [Technical philosophy (17)] An automatic driving device that provides an automatic driving function for automatically driving a vehicle, As operation modes, the vehicle is provided with a level 3 mode corresponding to automation level 3, a level 4 mode corresponding to automation level 4, and an occupant involvement mode in which at least a portion of the driving operation is left to the driver's seat occupant. a surrounding environment recognition unit (F1) that recognizes the surrounding environment including the positions of other moving objects based on observation data of a surrounding monitoring sensor; A mode control unit (F5) that switches an operation mode based on at least one of a recognition result of the surrounding environment recognition unit and an operation by an occupant in the driver's seat; an occupant state recognition unit (F2) that acquires occupant state data indicating the state of an occupant in the driver's seat from an occupant state sensor (12a); A recording processing unit (G3) that records occupant status data when there is a possibility of transferring driving authority to an occupant in the driver's seat in the level 3 mode or the level 4 mode, The recording processing unit includes: In Level 3 mode, occupant status data is recorded, In level 4 mode, occupant status data is recorded when the remaining exit time, which is the time remaining until the vehicle leaves the operation design area set for the vehicle, is less than a predetermined threshold, but occupant status data is not recorded if the remaining exit time is equal to or greater than the threshold. [Explanation of symbols]

[0177] 1 Autonomous driving system, 11 Surrounding monitoring sensor, 11a Front camera, 12 Occupant status sensor, 13 Vehicle sensor, 14 Locator, 15 HMI system, 16 V2X onboard device, 17 Body ECU, 18 Travel actuator, 20 Autonomous driving ECU (Autonomous driving processing unit), 21 Processing unit, 30 Operation recording unit, 31 Processing unit, 51 Driving mode notification light, 52 External communication device, F1 Surrounding environment recognition unit, F2 Occupant status recognition unit, F3 Vehicle status acquisition unit, F4 Operation content acquisition unit, F5 Mode control unit, F51 ODD inside / outside determination unit, F6 Action planning unit, F7 Control execution unit, F8 Abnormality detection unit, F9 Operation status output unit, F30 Operation recording unit, G1 Information acquisition unit, G11 Mode acquisition unit, G2 Recording target setting unit, G3 Recording processing unit, G4 Abnormality detection unit, G5 Recording data management unit, S1 Mode acquisition step, S2 recording target setting step, S3 recording step, S5 recording step

Claims

1. A recording device for a vehicle that records the operating state of an autonomous driving system, the recording device including an autonomous driving processing unit (20) that is a computer that provides an autonomous driving function and has an occupant involvement mode corresponding to any one of autonomous driving levels 0 to 2 as an operation mode and an autonomous driving mode that causes a vehicle to drive autonomously, An information acquisition unit (G1) that acquires multiple types of information that can be recorded from sensors mounted on the vehicle; A communication unit (34) that acquires a signal indicating an operation mode of the automatic driving processing unit; a processing unit (31) that executes a process of changing a recording target in accordance with the operation mode and a process of storing information corresponding to the recording target in a data storage unit (Mm) that is realized using a predetermined storage medium; The processing unit further comprises: In the case of an operation mode in which the recognition result of the automated driving system is included in the recording target, setting information on the application operating in the automated driving system and version information of the software used as the recording target; and recording, in the data storage unit, information about the application that was running in the autonomous driving system at the time when a predetermined event occurred and information about the version of the software that was being used. The event includes at least one of a decrease in autonomous driving level based on the operation of a driver's seat occupant, a decrease in autonomous driving level due to leaving an operation design area, execution of a handover request process by the autonomous driving processing unit, and execution of an MRM (Minimum Risk Maneuver), a recording device for a vehicle.

2. The vehicle recording device according to claim 1, wherein, in an operating mode in which the recognition results of the autonomous driving system are included in the recording targets, the processing unit records data indicating whether the software used in the autonomous driving system is up to date.

3. The processing unit sets a mode notification state indicating a notification state to an occupant regarding the operation mode of the automatic driving processing unit as a record target, The vehicle recording device of claim 1 , wherein the data indicative of the mode notification state includes a screen shot of a display.

4. 4. The vehicle recording device according to claim 3, which is used in a vehicle configured to be remotely operable, The data indicating the mode notification state is data indicating a notification state of the operation mode to a remote operator.

5. 4. The vehicle recording device according to claim 1, wherein when the processing unit acquires multiple data elements to be recorded at the same time, the processing unit assigns the same time stamp to those data elements and groups and stores them.

6. The vehicle recording device according to claim 1 or 3, wherein the processing unit is configured to add a hash value to the data to be recorded to prevent tampering and then store the data.

7. The data storage unit includes an overwriteable area (M1) from which data can be deleted, and an overwrite-prohibited area (M2) from which data cannot be deleted, The processing unit includes: storing data on the elements set as the recording target in the overwritable area; 4. The recording device for a vehicle according to claim 1, wherein, when the event occurs, data acquired within a predetermined time period from the occurrence of the event is stored in the overwrite-prohibited area.

8. The processing unit further comprises: Manage the data stored in the data storage unit; 4. The recording device for a vehicle according to claim 1, further comprising: a data storage unit configured to delete older data on a priority basis when the storage capacity of the data storage unit becomes full.

9. The recording target setting storage unit (Md) stores recording target setting data, which is a data set indicating a recording target for each of the operation modes, 4. The vehicle recording device according to claim 1, wherein the contents of the recording target setting data stored in the recording target setting memory unit are changeable based on an instruction from a server, wireless communication with the server, or an operation by a driver in the driver's seat.

10. A computer used in an autonomous driving system having an occupant involvement mode corresponding to any one of autonomous driving levels 0 to 2 as an operation mode and an autonomous driving mode in which a vehicle is driven autonomously, acquiring information on a plurality of types of elements that may be recorded from sensors mounted on the vehicle; obtaining a signal indicative of the operational mode of the automated driving system; Changing a recording target according to the operation mode; storing information about the elements set as the recording target in a data storage unit (Mm) realized using a predetermined storage medium; and recording, in the data storage unit, information on the application that was running in the autonomous driving system at the time when a predetermined event occurred and information on the version of the software that was being used; The event includes at least one of a decrease in the autonomous driving level based on the operation of a driver's seat occupant, a decrease in the autonomous driving level due to leaving an operation design area, execution of a handover request process by the autonomous driving system, and execution of an MRM (Minimum Risk Maneuver), the operation status recording program.

Images