Driving control device, driving control method, and program

Abstract

To improve the continuity of vehicle driving control. [Solution] The driving control device of the embodiment includes a recognition unit that recognizes the surrounding conditions of the vehicle, a detection unit that detects at least one of the state of the vehicle's occupants and the instructions and operations made by the occupants, an action plan generation unit that generates an action plan for the vehicle based on the recognition results from the recognition unit and the detection results from the detection unit, and a driving control unit that controls at least one of the steering and speed of the vehicle based on the action plan to execute driving control of the vehicle, wherein the action plan generation unit includes a first action plan generation unit that generates a first action plan based on the recognition results from the recognition unit and a preset rule, and a second action plan generation unit that generates a second action plan with a logic different from that used by the first action plan generation unit to generate the first action plan based on the recognition results from the recognition unit, and the action plan generation unit regenerates the other action plan after either the first action plan or the second action plan has been selected.

Description

[[Technical Field]]

[0001] The present invention relates to a travel control device, a travel control method, and a program. [[Background Art]]

[0002] In recent years, efforts to provide access to sustainable transportation systems that also take into consideration people in vulnerable positions among traffic participants have become active. Toward achieving this, focus is placed on research and development that further improves traffic safety and convenience through research and development related to automated driving technology. In relation to this, there has been conventionally known an action planning device including: a scene generation unit that uses surrounding information of a moving body to generate scene information indicating a situation where the moving body is placed; a mode calculation unit that uses the scene information to calculate a plurality of modes that are candidates of actions that the moving body can take in parallel; and a mode selection unit that selects one mode from the modes calculated by the mode calculation unit and outputs the selected mode as an action of the moving body (see, for example, Patent Document 1). [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] International Publication No. 2023-144938 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] By the way, in automated driving technology, when progress becomes impossible while proceeding according to the selected action plan, there has been a possibility that control is canceled. Therefore, there has been a problem that an action plan is not generated, and appropriate travel control may not be executed for a vehicle.

[0005] To address the above-mentioned problems, this invention aims to provide a vehicle control device, a vehicle control method, and a program that can generate a more appropriate action plan according to the vehicle's condition, thereby improving the continuity of vehicle driving control. Ultimately, this contributes to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The travel control device, travel control method, and program according to this invention employ the following configuration. (1) A driving control device according to one aspect of the present invention comprises: a recognition unit that recognizes the surrounding conditions of the vehicle; a detection unit that detects at least one of the state of the vehicle's occupants and the instructions and operations made by the occupants; an action plan generation unit that generates an action plan for the vehicle based on the recognition results from the recognition unit and the detection results from the detection unit; and a driving control unit that controls at least one of the steering and speed of the vehicle to perform driving control of the vehicle based on the action plan generated by the action plan generation unit, wherein the action plan generation unit comprises: a first action plan generation unit that generates a first action plan based on the recognition results from the recognition unit and a set rule; and a second action plan generation unit that generates a second action plan based on the recognition results from the recognition unit using a logic different from the first action plan generation unit, wherein the action plan generation unit regenerates the action plan after either the first action plan or the second action plan has been selected.

[0007] (2): In the embodiment of (1) above, the action plan generation unit regenerates the other action plan after the driving control unit has started driving control according to the selected action plan.

[0008] (3) In the embodiment of (1) above, the action plan generation unit regenerates the other action plan based on the selected action plan.

[0009] (4): In the embodiment of (1) above, the action plan generation unit, when the driving control unit determines that driving control based on either the first action plan or the second action plan can be executed, and driving control based on the other action plan cannot be executed, assumes that driving control of the other action plan is possible based on the one of the action plans, and regenerates the other action plan.

[0010] (5) In the embodiment of (4) above, the action plan generation unit switches to driving control based on the other action plan if it determines that it is impossible to continue driving control while driving control is being performed based on either of the action plans.

[0011] (6) In the embodiment of (5) above, the driving control unit will stop the execution of the driving control if it determines that driving control based on the other action plan is also impossible.

[0012] (7) In the embodiment of (1) above, the one of the action plans is the second action plan, and the other action plan is the first action plan.

[0013] (8) In the embodiment of (1) above, the driving control includes lane change control of the vehicle, and the action plan generation unit, when the lane change control is executed, regenerates the other action plan after either the first action plan or the second action plan has been selected.

[0014] (9): A driving control method according to one aspect of the present invention is a driving control method in which a computer recognizes the surrounding conditions of the vehicle, detects at least one of the state of the occupants of the vehicle and the instructions and operations made by the occupants, generates an action plan for the vehicle based on the recognized results and the detected results, controls at least one of the steering and speed of the vehicle to perform driving control of the vehicle based on the generated action plan, generates a first action plan based on the recognized results and a pre-set rule, generates a second action plan based on the recognized results using a logic different from the generation of the first action plan, and after either the first action plan or the second action plan is selected, regenerates the other action plan.

[0015] (10): A program according to one aspect of the present invention causes a computer to recognize the surrounding conditions of its own vehicle, to detect at least one of the state of the vehicle's occupants and the instructions and operations made by the occupants, to generate an action plan for the vehicle based on the recognized results and the detected results, to control the vehicle's driving by controlling at least one of the steering and speed of the vehicle based on the generated action plan, to generate a first action plan based on the recognized results and a pre-set rule, to generate a second action plan based on the recognized results using a logic different from that used to generate the first action plan, and to regenerate the other action plan after either the first action plan or the second action plan has been selected. [Effects of the Invention]

[0016] According to the embodiments described in (1) to (10) above, a more appropriate action plan can be generated according to the status of the vehicle, thereby improving the continuity of the vehicle's driving control. [Brief explanation of the drawing]

[0017] [Figure 1] This is a diagram showing the configuration of a vehicle system 1 utilizing a driving control device according to the embodiment. [Figure 2] This is a functional configuration diagram of the first control unit 120 and the second control unit 160. [Figure 3] FIG. 1 is a diagram illustrating an example of a functional configuration of an action plan generation unit 140 according to the embodiment. [Figure 4] FIG. 2 is a diagram (Part 1) for explaining a flow until an action plan is regenerated. [Figure 5] FIG. 3 is a diagram (Part 2) for explaining a flow until an action plan is regenerated. [Figure 6] FIG. 4 is a diagram for explaining behavior of a host vehicle M when a second target trajectory K2 is no longer generated by a second action plan generation unit 143. [Figure 7] FIG. 5 is a flowchart illustrating an example of a flow of processing executed by an automatic driving control device 100 according to the embodiment. MODES FOR CARRYING OUT THE INVENTION

[0018] Hereinafter, embodiments of a travel control device, a travel control method, and a program according to the present invention will be described with reference to the drawings.

[0019] [Overall Configuration] FIG. 1 is a configuration diagram of a vehicle system 1 using a travel control device according to an embodiment. A vehicle on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and the drive source thereof is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or electric power discharged from a secondary battery or a fuel cell. Hereinafter, as an example, an embodiment in which the travel control device is applied to an autonomous driving vehicle will be described. Autonomous driving is, for example, automatically controlling at least one of steering and speed of a vehicle to execute travel control. The vehicle travel control described above includes various types of travel control such as, for example, ACC (Adaptive Cruise Control System), LKAS (Lane Keeping Assistance System), TJP (Traffic Jam Pilot), ALC (Auto Lane Changing), and CMBS (Collision Mitigation Brake System). Further, the travel control may include MRM (Minimum Risk Maneuver) control for moving the host vehicle M to a safe location (e.g., a road shoulder, etc.) and stopping the vehicle. In addition, the autonomous driving vehicle may have its driving controlled by manual operation of an occupant (driver).

[0020] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, a map information processing device 60, a driver monitor camera 70, a driver control device 80, an automatic driving control device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Note that the configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. HMI 30 is an example of an "output unit". Automatic driving control device 100 is an example of a "driving control device".

[0021] Camera 10 is a digital camera that utilizes a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle (hereinafter referred to as "vehicle M") on which the vehicle system 1 is installed. When imaging the area in front, camera 10 is mounted on the top of the front windshield, behind the rearview mirror, etc. Camera 10 periodically and repeatedly images the area around vehicle M. Camera 10 may also be a stereo camera.

[0022] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by objects (reflected waves) to determine at least the position (distance and bearing) of an object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0023] The LIDAR 14 irradiates light (or electromagnetic waves with a wavelength close to light) around the vehicle M and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 can be attached to any location on the vehicle M.

[0024] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the external environment detection units, namely the camera 10, radar device 12, and LIDAR 14, to recognize the position, type, speed, etc., of an object. The object recognition device 16 outputs the recognition results to the automatic driving control device 100. The object recognition device 16 may output the detection results from the external environment detection units directly to the automatic driving control device 100. The object recognition device 16 may be omitted from the vehicle system 1. Alternatively, the external environment detection unit may include the object recognition device 16.

[0025] The communication device 20 communicates with other vehicles in the vicinity of its own vehicle M, or with various server devices via a wireless base station, for example, by using a cellular network, Wi-Fi network, Bluetooth®, DSRC (Dedicated Short Range Communication), etc.

[0026] The HMI 30, under the control of the HMI control unit 170, presents various information to the occupants of the vehicle M and accepts various input operations from the occupants. The HMI 30 includes, for example, various display devices, speakers, microphones, buzzers, touch panels, switches, keys, etc. Switches include, for example, turn signal switches. The HMI 30 accepts, for example, instructions regarding the start and end of automated driving, instructions to select the type of driving mode, instructions to select one of several action plans described later, and instructions regarding the illumination of the turn signal lights using the turn signal switch, and outputs the received operation instructions to the automated driving control device 100.

[0027] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting angular velocity around the vertical axis, and an orientation sensor for detecting the orientation of the vehicle M.

[0028] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of vehicle sensors 40. The navigation HMI 52 includes a display device, speaker, touch panel, keys, etc. The navigation HMI 52 may be partially or completely shared with the HMI 30 described above. The route determination unit 53 determines, for example, a route (hereinafter referred to as a route on the map) from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52, by referring to the first map information 54. The first map information 54 is, for example, information in which the shape of a road is represented by links indicating roads and nodes connected by those links. The first map information 54 may also include information such as the curvature of the road and POI (Point of Interest) information. The route on the map is output to the map information processing device 60. The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be implemented, for example, by the functions of a terminal device such as a smartphone or tablet held by an occupant. The navigation device 50 may transmit the current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0029] The map information processing device 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the map route provided by the navigation device 50 into multiple blocks (for example, every 100m with respect to the vehicle's direction of travel) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 makes decisions such as which lane from the left the vehicle should travel in. For example, if there is a branching point on the map route, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel along a reasonable route to proceed to the branching point.

[0030] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may also include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. The second map information 62 may be updated as needed by the communication device 20 communicating with other devices. The first map information 54 and the second map information 62 may be provided together as map information. At least one of the first map information 54 and the second map information 62 may be stored in the automatic driving control device 100.

[0031] The driver monitor camera 70 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. The driver monitor camera 70 is mounted at any location in the vehicle M in a position and orientation that allows it to capture the head of the occupant (hereinafter referred to as the driver) seated in the driver's seat of the vehicle M from the front (in a direction that captures the face). For example, the driver monitor camera 70 is mounted on top of a display device located in the center of the instrument panel of the vehicle M.

[0032] The driver control elements 80 include, for example, the steering wheel 82, as well as the accelerator pedal, brake pedal, shift lever, and other controls. The driver control elements 80 are equipped with sensors that detect the amount of operation or whether or not an operation is being performed, and the detection results are output to the automatic driving control device 100, or to some or all of the driving force output device 200, brake device 210, and steering device 220. The steering wheel 82 is an example of a "control element that accepts steering operations by the driver." The control element does not necessarily have to be ring-shaped, and may take the form of an irregularly shaped steering wheel, joystick, button, etc. A steering grip sensor 84 is attached to the steering wheel 82. The steering grip sensor 84 is implemented by, for example, a capacitive sensor, and outputs a signal to the automatic driving control device 100 that can detect whether or not the driver is gripping the steering wheel 82 (meaning that it is in contact with the steering wheel in a state where force can be applied).

[0033] The automatic driving control device 100 includes, for example, a first control unit 120, a second control unit 160, an HMI control unit 170, and a storage unit 180. The first control unit 120, the second control unit 160, and the HMI control unit 170 are each implemented by a hardware processor, such as a CPU (Central Processing Unit), executing a program (software). Furthermore, some or all of these components may be implemented by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the automatic driving control device 100, or it may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 when the storage medium (non-transient storage medium) is mounted on a drive device. The second control unit 160 is an example of a "driving control unit". The driving control unit may also include some functions of the action plan generation unit 140. The HMI control unit 170 is an example of an "output control unit".

[0034] The memory unit 180 may be implemented using the various storage devices described above, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The memory unit 180 stores, for example, a learned model 182, information necessary to perform the driving control in this embodiment, and various other information and programs. The memory unit 180 may also store map information (for example, at least one of the first map information 54 and the second map information 62). Details of the learned model 182 will be described later.

[0035] Figure 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, a recognition unit 130, an action plan generation unit 140, and a mode determination unit 150. The first control unit 120 is realized by executing, for example, functions based on a pre-given model and functions based on AI (Artificial Intelligence) in parallel, or by prioritizing one over the other. For example, the function of "recognizing intersections" may be realized by executing recognition based on pre-given rules (such as pattern-matchable signals and road markings) and intersection recognition using machine learning (e.g., a neural network) in parallel, scoring both, and comprehensively evaluating them. This ensures the reliability of autonomous driving.

[0036] The recognition unit 130 recognizes the position and state, such as speed and acceleration, of objects around the vehicle M based on information input from the external detection unit (e.g., camera 10, radar device 12, and LIDAR 14) via the object recognition device 16. The position of an object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive axis) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or it may be represented by a region. The "state" of an object may include the object's acceleration, jerk, or "action state" (e.g., whether or not it is changing lanes).

[0037] Furthermore, the recognition unit 130 recognizes, for example, the lane in which the vehicle M is traveling (driving lane). For example, the recognition unit 130 recognizes the driving lane by comparing the pattern of road markings (for example, an arrangement of solid and dashed lines) obtained from the second map information 62 with the pattern of road markings around the vehicle M recognized from the image captured by the camera 10. Note that the recognition unit 130 may recognize the driving lane not only by road markings, but also by recognizing the road boundary (road boundary) including road markings, shoulders, curbs, median strips, guardrails, etc. In this recognition, the position of the vehicle M obtained from the navigation device 50 and the processing results by INS may also be taken into consideration. Furthermore, the recognition unit 130 recognizes stop lines, obstacles, red lights, toll booths, and other road events. Furthermore, the recognition unit 130 recognizes adjacent lanes adjacent to the driving lane. Adjacent lanes are, for example, lanes that can travel in the same direction as the driving lane.

[0038] When recognizing a driving lane, the recognition unit 130 recognizes the position and attitude of the vehicle M relative to the driving lane. For example, the recognition unit 130 may recognize the deviation of the vehicle M's reference point from the center of the lane, and the angle it makes with a line connecting the centers of the lanes in the direction of travel of the vehicle M, as the relative position and attitude of the vehicle M relative to the driving lane. Alternatively, the recognition unit 130 may recognize the position of the vehicle M's reference point relative to any side edge of the driving lane (road marking or road boundary), etc., as the relative position of the vehicle M relative to the driving lane. Here, the reference point of the vehicle M may be the center of the vehicle M, or its center of gravity. The reference point may also be an end of the vehicle M (front end, rear end), or the position of one of the multiple wheels of the vehicle M.

[0039] The action plan generation unit 140, based on, for example, the recognition results from the recognition unit 130 and the information obtained from the mode determination unit 150, will, in principle, drive in the recommended lane determined by the recommended lane determination unit 61, and further, automatically (without driver operation) generates a target trajectory for the vehicle M to travel in the future, in order to respond to the surrounding conditions of the vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is represented as a sequence of points (trajectory points) that the vehicle M should reach. The trajectory points are points that the vehicle M should reach at predetermined driving distances (e.g., a few meters), and separately, target speed and target acceleration at predetermined sampling times (e.g., a few tenths of a second) are generated as part of the target trajectory. Alternatively, the trajectory points may be the positions that the vehicle M should reach at the sampling time for each predetermined sampling time. In this case, the target speed and target acceleration information is represented by the intervals between trajectory points.

[0040] Furthermore, in this embodiment, the action plan generation unit 140 generates an action plan using pre-defined rule-based functions (first action plan) and an action plan using AI-based functions such as machine learning (second action plan), and selects one of the action plans according to the surrounding conditions of the vehicle M, the driver's condition, etc. Here, generating an action plan means, for example, generating a future target trajectory for the vehicle M. Details of the above will be described later.

[0041] The action plan generation unit 140 may also activate autonomous driving events (functions) when generating the target trajectory. Autonomous driving events include constant speed driving events, low-speed follow driving events, lane change events, branching events, merging events, degraded driving events, and takeover events. The action plan generation unit 140 generates a target trajectory according to the activated event.

[0042] The mode determination unit 150 determines the driving mode of the vehicle M according to the surrounding conditions of the vehicle M and the state of the occupants of the vehicle M, based on at least one of the following: for example, the recognition result from the recognition unit 130, the image captured by the driver monitor camera 70, the detection result from the steering grip sensor 84, the information received by the HMI 30, and the operation content of the driving control unit 80. For example, the mode determination unit 150 determines the driving mode to be performed by the vehicle M according to the conditions of the vehicle M and the state of the driver, to be one of several driving modes in which the tasks assigned to the driver differ (in other words, several modes in which the degree of automation differs).

[0043] Here, the driving modes in the embodiment include, for example, a first driving mode and a second driving mode in which the degree of driving assistance is lower than that of the first driving mode, or in which the driver of the vehicle M has a greater workload than that of the first driving mode. A lower degree of driving assistance means, for example, a low automation rate in driving control. A low automation rate means, for example, a low degree to which the automatic driving control device 100 controls the steering or speed of the vehicle M (a high degree to which the driver needs to intervene in steering or speed operation). A greater workload for the driver includes, for example, a large number of tasks assigned to the driver or tasks that are heavy. Tasks include, for example, the driver monitoring the surroundings of the vehicle M or the driver operating the driving controls 80. Operating the driving controls 80 includes, for example, the driver gripping the steering wheel (hereinafter referred to as the hands-on state). The driving modes may also include a third driving mode, etc., in which the degree of driving assistance is lower than that of the second driving mode, or in which the driver of the vehicle M has a greater workload than that of the second driving mode. Furthermore, the driving mode with the lowest level of driver assistance or the greatest task for the driver of the vehicle M may be the fully manual driving mode (a mode in which no driving control is performed).

[0044] For example, in the first driving mode, the driver has no (or minimal) tasks and driving control (e.g., ACC, LKAS, TJP, ALC, etc.) is permitted when the driver of the vehicle M is not holding the steering wheel (hereinafter referred to as the hands-off state). In the second driving mode, tasks assigned to the driver may include, for example, monitoring the surroundings of the vehicle M while also being in the hands-on state.

[0045] The mode determination unit 150 outputs the determined mode to the action plan generation unit 140, which generates a target trajectory corresponding to the mode. The mode determination unit 150 includes, for example, a detection unit 152 and a mode processing unit 154. The detection unit 152 includes, for example, a driver state detection unit 152A, an instruction content detection unit 152B, and a vehicle state detection unit 152C.

[0046] The driver state detection unit 152A detects whether the occupant (driver) is in a state suitable for driving (or, to put it another way, whether the occupant is in an abnormal state). For example, the driver state detection unit 152A monitors the driver's state for the above-mentioned mode change and detects whether the driver's state is appropriate for the task. For example, the driver state detection unit 152A performs known image analysis processing (e.g., feature extraction of edges, shape, size, and color, pattern matching, etc.) on the image captured by the driver monitor camera 70, performs driver posture estimation processing from the analysis results, and determines whether the driver is in a position suitable for manual driving (e.g., a position that allows for manual driving in response to a request from the system). The driver state detection unit 152A may also perform gaze estimation processing from the analysis results of the image captured by the driver monitor camera 70 and determine whether the driver is monitoring the area around the vehicle M (more specifically, in front). The driver state detection unit 152A may also determine whether the driver is gripping the steering wheel 82 based on the detection results of the steering grip sensor 84. Based on these various determination results, the driver state detection unit 152A detects that the driver is not in a state suitable for driving (the driver is in an abnormal state) if it determines that the driver is not in a state suitable for the task for a predetermined time or longer, and detects that the driver is in a state suitable for driving (the driver is not in an abnormal state) if it determines that the driver is in a state suitable for the task for a predetermined time or longer.

[0047] Furthermore, the driver state detection unit 152A may also detect, based on the analysis results of the images captured by the driver monitor camera 70, that the driver's facial expression is unsuitable for driving, such as appearing distressed or sleepy, and that this expression persists for a predetermined period of time or longer, indicating that the driver is not in a state suitable for driving (i.e., the driver is in an abnormal state).

[0048] The instruction content detection unit 152B detects the content of the driver's operation instructions received from the HMI 30 or the like. The content of the operation instructions may include, for example, instructions regarding the start or end of automated driving, instructions to select the type of driving control or driving mode, and instructions to select one of several action plans. The content of the operation instructions may also include instructions to execute a predetermined behavior control for the vehicle M. The predetermined behavior control for the vehicle M is, for example, ALC control (lane change control), but is not limited to this, and may be any driving control (specific function) such as LKAS, steering or speed. In the following explanation, as an example, the predetermined behavior control will be described as ALC control.

[0049] The vehicle status detection unit 152C detects the driving status of the vehicle M. The driving status of the vehicle M includes, for example, the current driving mode of the vehicle M, the position of the vehicle M on the road based on the recognition results of the recognition unit 130, and the speed of the vehicle M. The vehicle status detection unit 152C may also detect information regarding the road conditions while driving (for example, road shape, presence, number, position (relative position), and speed (relative speed) of other vehicles) based on the recognition results of the recognition unit 130, and may also detect whether or not lane markings that demarcate the lane in which the vehicle M is driving are recognized. Furthermore, the vehicle status detection unit 152C may also detect, based on the recognition results of the recognition unit 130, whether there is an adjacent lane adjacent to the driving lane.

[0050] The mode processing unit 154 determines the driving mode of its own vehicle M based on at least one of the determination results from the driver state detection unit 152A, the instruction content detection unit 152B, and the vehicle status detection unit 152C, and performs various processes for changing the driving mode if necessary.

[0051] For example, when the vehicle M is executing the second driving mode, the mode processing unit 154 detects an instruction to start the first driving mode by the instruction content detection unit 152B. Based on the determination result by the vehicle status detection unit 152C, the mode processing unit 154 determines whether the first driving mode can be executed. If it is determined that it can be started, it decides to switch to the first driving mode and outputs a decision to switch to the first driving mode to the action plan generation unit 140.

[0052] Furthermore, when the mode processing unit 154 detects an instruction to execute a specific driving control (e.g., LKAS or ALC) for the first driving mode by the instruction content detection unit 152B during the execution of the first driving mode, it determines whether the surrounding conditions of the vehicle M are suitable for executing the instructed driving control. If it determines that it is possible, it decides to switch to the instructed driving control. For example, if the instructed driving control is ALC, the mode processing unit 154 decides to execute ALC if the vehicle condition detection unit 152C detects the vehicle M's driving lane and the adjacent lane to which it will change lanes, and outputs an instruction to execute ALC to the action plan generation unit 140. Also, if the instructed driving control is LKAS for the first driving mode, the mode processing unit 154 decides to execute LKAS if the vehicle condition detection unit 152C detects the lane markings that demarcate the vehicle M's driving lane, and outputs an instruction to execute LKAS to the action plan generation unit 140.

[0053] Furthermore, the mode processing unit 154 determines, by the vehicle condition detection unit 152C, that the vehicle M is in a situation where it cannot execute (continue) the first driving mode while it is executing the first driving mode, and decides to execute the second driving mode (for example, manual driving). Situations where the first driving mode cannot be executed include, for example, when the vehicle M can no longer recognize the lane markings of its driving lane while ALC control or LKAS control is being executed, when the driving section is not one in which the first driving mode can be executed, when there is a malfunction in at least a part of the vehicle M's external environment detection unit (camera 10, radar device 12, LIDAR 14), or when the accuracy of the recognition unit 130 for recognizing the surroundings is reduced due to weather conditions such as heavy rain or snow.

[0054] When the mode processing unit 154 decides to switch from the first driving mode to the second driving mode, it causes the HMI control unit 170 to control the HMI 30 to prompt the driver to perform predetermined tasks for switching to the second driving mode. These predetermined tasks include, for example, the driver monitoring the surroundings of the vehicle M and gripping the steering wheel 82. For example, when the driver state detection unit 152A determines that the driver is in a state suitable for driving, it decides to switch to the second driving mode and outputs a command to the action plan generation unit 140 to switch to the second driving mode.

[0055] Furthermore, if a predetermined time has elapsed since the HMI 30 output information prompting the driver to perform a predetermined task to switch to the second driving mode, but the task has not been performed, or if the driver state detection unit 152A determines that the driver is not in a state suitable for driving, the mode processing unit 154 decides to activate (execute) degraded driving control, which involves moving the vehicle M to the shoulder of the road or the like and gradually stopping, without the driver's operation, and stopping (ending) the first driving mode. An instruction to generate an action plan corresponding to the degraded driving is output to the action plan generation unit 140. After stopping automatic driving, the vehicle M transitions to a driving mode with a lower degree of automation, and the vehicle M can be started by the driver's manual operation.

[0056] Furthermore, if the instruction content determined by the instruction content detection unit 152B during the execution of the first driving mode is a switch to the second driving mode, the mode processing unit 154 will determine whether the driver is in a state suitable for driving the second driving mode, as described above, and will perform processing according to the determination result. In addition, if the mode processing unit 154 determines during the execution of the first driving mode that the driver is not in a state suitable for driving (the driver is abnormal), it may decide to activate (execute) the degraded driving control described above and instruct the action plan generation unit 140 to do so. The mode processing unit 154 may also output the driver's state (for example, that the driver is not in a state suitable for driving the vehicle M) to the action plan generation unit 140.

[0057] The second control unit 160 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the vehicle M passes through the target trajectory (more specifically, the target trajectory selected from a plurality of action plans (target trajectories)) generated by the action plan generation unit 140 at the scheduled time.

[0058] The second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires information on the target trajectory (trajectory point) generated by the action plan generation unit 140 and stores it in memory (not shown). The speed control unit 164 controls the driving force output device 200 or the brake device 210 based on the speed elements associated with the target trajectory stored in memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in memory. The processing of the speed control unit 164 and the steering control unit 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 performs a combination of feedforward control according to the curvature of the road in front of the vehicle M and feedback control based on the deviation from the target trajectory.

[0059] The HMI control unit 170 notifies the occupants (including the driver) of the vehicle M via the HMI 30 of predetermined information. The predetermined information includes, for example, information related to the driving of the vehicle M, such as information about the status of the vehicle M and information about driving control. Information about the status of the vehicle M includes, for example, the speed of the vehicle M, engine speed, and shift position. Information about driving control may also include, for example, whether or not autonomous driving is being performed, information about changes in driving control and driving modes, the status of driving control (e.g., the content of events being performed), and the status of the driving mode (the driving mode being performed). The predetermined information may also include information unrelated to the driving control of the vehicle M, such as content stored on a storage medium such as a television program or DVD (e.g., a movie). The predetermined information may also include, for example, information about the current location and destination of the vehicle M, and the remaining fuel level.

[0060] For example, the HMI control unit 170 may generate an image containing the predetermined information described above and display the generated image on the display device of the HMI 30, or it may generate audio indicating the predetermined information and output the generated audio from the speaker of the HMI 30. In addition, the HMI control unit 170 may output the information received by the HMI 30 to the communication device 20, the navigation device 50, the first control unit 120, etc.

[0061] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the second control unit 160 or information input from the driver control unit 80.

[0062] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the second control unit 160 or from the driver control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may also include a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal included in the driver control unit 80 to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the second control unit 160 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0063] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the second control unit 160 or from the steering wheel 82 of the driver control unit 80.

[0064] [Action Plan Generation Department] The processing of the action plan generation unit 140 will be described in detail below. Figure 3 is a diagram showing an example of the functional configuration of the action plan generation unit 140 in this embodiment. The action plan generation unit 140 includes, for example, a preprocessing unit 141, a first action plan generation unit 142, a second action plan generation unit 143, and a selection control unit 144.

[0065] The pre-processing unit 141 determines a general policy for the future behavior of the vehicle M based on the recognition results of the recognition unit 130 and the information determined by the mode determination unit 150, and sets the semantics (target trajectory generation conditions) necessary to generate a target trajectory from that behavior. For example, when an ALC control execution instruction is received, target trajectory generation conditions (semantics) are set such that the vehicle changes lanes to the lane specified for the lane change, and when an LKAS control execution instruction is received, target trajectory generation conditions (semantics) are set such that the vehicle travels in the center of the current lane. The pre-processing unit 141 may also predict the future behavior of other vehicles and other targets in the vicinity of the vehicle M based on the recognition results. The pre-processing unit 141 outputs the above-mentioned setting results, prediction results, and recognition results from the recognition unit 130 to the first action plan generation unit 142 and the second action plan generation unit 143.

[0066] Furthermore, when the preprocessor 141 receives an instruction from the mode determination unit 150 to start (execute) degraded operation control, it may output an instruction to start degraded operation control to at least one of the first action plan generation unit 142 and the second action plan generation unit 143, or one predetermined one. Also, when the preprocessor 141 receives an instruction from the mode determination unit 150 to execute a second operation mode (for example, manual operation), it may output an instruction to the first action plan generation unit 142 and the second action plan generation unit 143 to terminate the generation of the action plan (target trajectory). However, even if manual operation is being performed, the process of generating an action plan by the first action plan generation unit 142 and the second action plan generation unit 143 may continue. In this case, the selection control unit 144 will not select any action plan.

[0067] Furthermore, the functions of the preprocessing unit 141 may be incorporated into the first action plan generation unit 142 or the second action plan generation unit 143, in which case the configuration of the preprocessing unit 141 is not required.

[0068] The first action plan generation unit 142 generates an action plan (first action plan) based on information obtained from the preprocessing unit 141 and pre-set rules (constraints, etc.). For example, when ALC, LKAS, etc., are instructed by the first driving mode, the first action plan generation unit 142 generates a future target trajectory (first target trajectory) of the vehicle M based on various constraints based on the rules, on an optimization problem basis, so that the instructed driving mode and driving control can be achieved.

[0069] For example, when ALC control is performed, the first action plan generation unit 142 sets a target position for the vehicle M on the lane to which the vehicle will change (adjacent lane), and generates an optimal first target trajectory that satisfies the constraints, with the constraint that the vehicle M must move to the target position within a predetermined time without contacting other vehicles around it. An optimal target trajectory is, for example, a target trajectory in which the amount of change in the behavior of the vehicle M is smallest, or a target trajectory in which the possibility of contact with obstacles such as other vehicles is smallest. Also, for example, when LKAS control is performed, the first action plan generation unit 142 generates an optimal first target trajectory (first vehicle trajectory) that satisfies the constraints, with the constraints that the reference point of the vehicle M (for example, the center of gravity or center of the vehicle M) passes over the center of the lane divided by the left and right lane lines, and that the vehicle travels at a speed that does not come into contact with the legal speed of the lane or other vehicles in front of or behind the vehicle. In this way, the first action plan generation unit 142 compares the information acquired by the mode determination unit 150 with all information such as external information (recognition results) and each rule (constraint condition) to generate a rule-based first target trajectory that satisfies each rule and is optimal.

[0070] The first target trajectory generated by the first action plan generation unit 142 is generated based on predetermined rules, making the vehicle behavior more stable and enabling smooth (sequential) behavior, especially during lane changes. However, in cases where the surrounding traffic environment of the vehicle M is complex, the first action plan generation unit 142 may take a long time to find a solution that satisfies the given constraints, or may not be able to find a solution at all, potentially preventing proper automated driving.

[0071] The second action plan generation unit 143 generates an action plan (second action plan) based on information processed by the preprocessing unit 141 and a model (trained model 182) that has been trained to generate action plans based on external information (recognition results), etc. For example, when ALC or LKAS is instructed by the first driving mode, the second action plan generation unit 143 generates a target trajectory (second target trajectory) that achieves driving control according to the instruction content, based on inference (AI-based) using the trained model 182. In other words, the second action plan generation unit 143 generates the second action plan using a different logic than the first action plan generation unit 142 generates the first action plan.

[0072] For example, the second action plan generation unit 143 uses machine learning (e.g., neural networks) or deep learning to pre-train a model that takes external information (e.g., recognition results) or semantic information as input and outputs a second target trajectory that is inferred to be optimal, using training data (ground truth data) or past performance data. Then, the second action plan generation unit 143 inputs the information processed by the pre-processing unit 141 into the trained model 182 (inference unit) to obtain the second target trajectory that is inferred to be optimal. The trained model 182 may be obtained from an external device via the communication device 20, or it may be trained by the second action plan generation unit 143. Furthermore, the trained model 182 may be updated each time it is used by performing feedback control based on ground truth data.

[0073] The second target trajectory generated by the second action plan generation unit 143 can infer a global (rough) understanding of the environment, even in complex traffic environments. Therefore, even in situations (surrounding conditions) that have never been driven before, the target trajectory will correspond to the determined instructions. Consequently, the vehicle M can continue autonomous driving without stopping. However, if the external information changes even slightly, the second target trajectory may easily be output as a completely different target trajectory, which could potentially lead to unstable behavior of the vehicle M.

[0074] The selection control unit 144 selects either a first target trajectory (an example of a first action plan) generated by the first action plan generation unit 142 or a second target trajectory (an example of a second action plan) generated by the second action plan generation unit 143. For example, the selection control unit 144 prioritizes using one of the two target trajectories (action plans) that has been set in advance, and uses the other target trajectory when the one currently in use reaches a predetermined condition such as a performance limit. For example, if the learned model 182 is not yet mature, such as immediately after the vehicle M is shipped, the selection control unit 144 may select the rule-based first target trajectory generated by the first action plan generation unit 142, and then select the second target trajectory generated by the second action plan generation unit 143 when the learned model 182 has matured (for example, when the vehicle M has traveled a predetermined distance or time and the learned model 182 has been updated according to the travel results).

[0075] Furthermore, the selection control unit 144 may select a target trajectory from either the first or second target trajectory, in accordance with the driver's instructions received by the HMI 30. This allows the driver to travel the vehicle M along the target trajectory desired by the driver.

[0076] The acquisition unit 162 of the second control unit 160 adjusts control quantities related to the driving control of the vehicle M, such as speed and steering, based on the target trajectory selected by the selection control unit 144. It outputs the control quantity for speed to the speed control unit 164 and the control quantity for steering to the steering control unit 166, thereby executing driving control according to the target trajectory. In this embodiment, since the action plan generation unit 140 generates two types of target trajectories (action plans), the vehicle M can be driven by selecting a more appropriate target trajectory to correspond to various driving scenes.

[0077] Here, the action plan generation unit 140 regenerates the other target trajectory after the selection control unit 144 has selected either the first or second target trajectory. For example, as shown in Figure 3, the selection control unit 144 outputs information about the selected content to the first action plan generation unit 142 and the second action plan generation unit 143. The first action plan generation unit 142 and the second action plan generation unit 143 receive information on whether the target trajectory they generated has been selected, and even if it has not been selected, they regenerate the action plan.

[0078] Figure 4 is a diagram (part 1) illustrating the flow until the action plan is regenerated. In the example in Figure 4, two lanes L1 and L2 are shown that can be traveled in the same direction (X-axis direction in the figure). Lane L1 is demarcated by lane markings LL and CL, and lane L2 is demarcated by lane markings CL and RL. Vehicle M is traveling in lane L1 at speed VM. In the example in Figure 4, other vehicles m1 to m3 are present around vehicle M. Other vehicle m1 is in front of vehicle M and is traveling in lane L1 at speed Vm1, other vehicle m2 is in front of vehicle M and is traveling in lane L2 at speed Vm2, and other vehicle m3 is to the right of vehicle M and is traveling in lane L2 at speed Vm3. In the following explanation, the positions of the vehicle M and other vehicles m1 to m3 at time T* will be represented as M(T*), m1(T*), m2(T*), and m3(T*), respectively, and their speeds will be represented as VM(T*), Vm1(T*), Vm2(T*), and Vm3(T*), respectively. Also, in the following explanation, T1 is assumed to be the earliest time, followed by T2 and then T3, in descending order. The recognition unit 130 recognizes the driving lane L1 and adjacent lane L2 of the vehicle M, as well as the positions (relative positions) and speeds (relative speeds) of the other vehicles m1 to m3.

[0079] The example in Figure 4 shows the situation at time T1. In the situation shown in Figure 4, for example, if the instruction content detection unit 152B receives an ALC instruction from the occupant (for example, an instruction to change lanes from lane L1 to lane L2 using the turn signal switch) while the first driving mode is being executed, the mode determination unit 150 outputs an instruction to the action plan generation unit 140 to execute ALC control in the first driving mode. Based on the surrounding conditions and rules (constraints), the first action plan generation unit 142 determines that other vehicles m1(T1) to m3(T1) are present around the vehicle M(T1), and that changing lanes (overtaking, etc.) is not possible due to their position and speed. As shown in Figure 4, it generates a first target trajectory K1 in which the vehicle proceeds straight without changing lanes and slows down to avoid contact with other vehicles m1.

[0080] Meanwhile, the second action plan generation unit 143 generates a second target trajectory K2 for changing lanes from lane L1 to lane L2, as shown in Figure 4, based on the trained model 182. The second target trajectory K2 shown in Figure 4 is, for example, a target trajectory that could not be generated by the rule-based system of the first action plan generation unit 142, and is a target trajectory in which the vehicle M's behavior is relatively large, allowing for a lane change at the last possible moment.

[0081] Here, the selection control unit 144 selects one of the two action plans (first target trajectory K1, second target trajectory K2) based on a preset priority, driver instructions, etc. In this case, the second target trajectory K2 is selected. As a result, the driving control unit (second control unit 160) starts lane change control of its own vehicle M based on the second target trajectory K2.

[0082] Here, the selection control unit 144 outputs the selected information to the first action plan generation unit 142 and the second action plan generation unit 143. The second action plan generation unit 143, being the selected unit, continues to generate the second target trajectory K2. Meanwhile, the first action plan generation unit 142, which was not selected, regenerates the first target trajectory K1.

[0083] Figure 5 is a diagram (part 2) illustrating the flow until the action plan is regenerated. In the example in Figure 5, the situation at time T2 is shown after a predetermined time has elapsed from time T1 shown in Figure 4. At time T2, the vehicle M(T2) has its right turn signal flashing. Also, the vehicle M(T2) has initiated a lane change based on the second target trajectory K2 by the driving control unit and is moving laterally towards lane L2 in order to change lanes from lane L1 to lane L2.

[0084] The first action plan generation unit 142 regenerates the first target trajectory K1 after driving control based on the selected action plan (second target trajectory K2) has started. As a result, the other target trajectory (first target trajectory K1) is regenerated (recalculated) based on the state after the behavior of the vehicle M has changed due to the currently executing target trajectory (second target trajectory K2). Therefore, even if it was previously not possible to generate a target trajectory corresponding to the instructed driving control (lane change), the possibility of generating the desired target trajectory in the current situation can be increased.

[0085] Furthermore, when the first action plan generation unit 142 regenerates the first target trajectory K1, it may regenerate the first target trajectory K1 based on one of the target trajectories (second target trajectory K2) selected by the selection control unit 144. In this case, the first action plan generation unit 142 obtains information on the second target trajectory K2 from the second action plan generation unit 143 or the selection control unit 144, and generates a first target trajectory K1a that follows (is closer to) the obtained second target trajectory K2 based on the rule base. This makes it easier to generate a first target trajectory K1a that performs a lane change based on the second target trajectory K2. With the above control, a smooth transition can be made when switching from the second target trajectory K2 to the first target trajectory K1a, and changes in the behavior of the vehicle M due to the switching of the target trajectory can be suppressed.

[0086] Furthermore, the first action plan generation unit 142 may regenerate the first target trajectory K1a described above, and, as in the case of time T1, also generate a first target trajectory K1b for traveling on lane L1 (center of the lane), as shown in Figure 5. By generating a first target trajectory K1b for maintaining travel in the current lane L1, even if the lane change control is canceled due to changes in surrounding conditions during the execution of a lane change, the vehicle can smoothly return to the center of lane L1 using the first target trajectory K1b.

[0087] Figure 6 is a diagram illustrating the behavior of the vehicle M when the second action plan generation unit 143 is unable to generate the second target trajectory K2. In the example in Figure 6, the situation at time T3 is shown, which is a predetermined time after the situation at time T2 shown in Figure 5. In the example in Figure 6, the positional relationship and relative speed of the vehicle M (T3) and other vehicles m1 (T3) to m3 (T3) have changed over time, so the surrounding conditions have changed, and it is assumed that the first action plan generation unit 142 can generate the first target trajectory K1a in a rule-based manner.

[0088] In this situation, for example, if the performance limits of the trained model 182 are reached and the second action plan generation unit 143 is unable to generate the second target trajectory K2 (if driving control using the second target trajectory K2 becomes impossible), the selection control unit 144 switches to the first target trajectory K1a generated by the first action plan generation unit 142 and continues lane change control. The situation in which the performance limits of the trained model 182 are reached is, for example, a situation in which there is insufficient information input to the trained model 182 or a situation in which the recognition accuracy of the input recognition results deteriorates, resulting in a situation in which a target trajectory cannot be generated (the inference of the learning machine becomes incorrect).

[0089] Thus, the selected target trajectory (action plan generation unit) switches between time T2 and time T3. However, since the first action plan generation unit 142 was searching for the first target trajectory K1a so that it could follow the second target trajectory K2 of the second action plan generation unit 143 even when the first target trajectory was not selected, the vehicle behavior can continue seamlessly even after the switch. Because seamless behavior can be maintained in this way, the continuity of the first driving mode (autonomous driving) can be improved.

[0090] [summary] Thus, in this embodiment, two target trajectories (action plans) with different logic, rule-based and AI-based (machine learning-based), are generated, and the target trajectory is flexibly switched according to the driving conditions of the vehicle M. For example, when the vehicle M is driving on a complex route on a public road, the AI-based target trajectory (second target trajectory) is selected and driving control is executed, and when degenerate driving (MRM) control is performed to move the vehicle M to a safe place (e.g., the shoulder of the road) and stop, the rule-based target trajectory (first target trajectory) is selected. In addition, in this embodiment, even if one target trajectory (action plan generation unit) is selected, the other action plan generation unit is kept running to regenerate (operate) the target trajectory. This ensures that during predetermined driving control by the first driving mode, the selected target trajectory is kept running. Even if a target track cannot be generated, the first driving mode can be continued using the other target track. Furthermore, according to this embodiment, when the other system regenerates a target track, it can be regenerated to approach the selected target track (for example, a target track for lane changes), making it easier to continue the current driving control even when switching target tracks. However, there are cases where the other target track cannot be regenerated along the selected target track, and a different target track (for example, a target track that returns to the center of the original lane instead of a target track for lane changes) is generated. In such a situation, if one of the target tracks becomes unavailable, driving control that returns to the center of the original lane is executed based on the other target track.

[0091] In this embodiment, if the driving control unit determines that driving control can be executed based on either the first action plan (first target trajectory) or the second action plan (second target trajectory), and that driving control cannot be executed based on the other action plan, the action plan generation unit 140 may regenerate the other action plan based on the other action plan, assuming that driving control of the other action plan is possible. As a result, even if the other action plan generation unit is unable to find a valid action plan (gives up), if it determines that driving control is possible based on the first action plan, it will regenerate the other action plan, assuming that driving control is possible. This leaves open the possibility of taking over to the other action plan even if driving control becomes difficult with the first action plan, thereby improving the continuity of driving control.

[0092] Furthermore, as described above, if the action plan generation unit 140 determines that it is impossible to continue driving control (for example, one action plan cannot be generated) while driving control is being performed using either of the action plans (for example, the second action plan), it switches to driving control based on the other action plan (for example, the first action plan). This prevents the operation from being terminated midway through, such as during a lane change, allowing for more appropriate driving control and improving the continuity of driving control.

[0093] Furthermore, when the action plan generation unit 140 (selection control unit 144) switches to driving control based on the other action plan, it may, for example, determine whether driving control based on the other action plan is possible, and if it determines that it is possible, it may switch to driving control based on the other action plan. In this case, the action plan generation unit 140 may, for example, determine that driving control based on the other action plan is possible if the other action plan has been regenerated in accordance with either of the selected action plans (for example, the second action plan), and determine that driving control based on the other action plan is impossible if the other action plan has not been regenerated in accordance with either of the selected action plans. In addition, the action plan generation unit 140 may determine that driving control based on the other action plan is possible if at least one of the other action plans (target trajectories) has been generated, and determine that driving control based on the other action plan is impossible if none have been generated.

[0094] Furthermore, if the action plan generation unit 140 determines that it is also impossible to perform driving control based on the other action plan (for example, it is not possible to generate the other action plan), it may stop executing the driving control. If the regenerated other action plan does not allow the driving control to continue in accordance with the first action plan, the started driving control can be canceled, allowing for the execution of more appropriate driving control without forcing it.

[0095] Furthermore, when lane change control is performed, the action plan generation unit 140 regenerates the other action plan after either the first action plan or the second action plan has been selected. In cases where, during lane change control, one action plan allows for a lane change, but the other action plan does not and the vehicle continues straight, the probability of completing the lane change can be improved by performing lane change control on one side while simultaneously regenerating an action plan for the other side that gave up on the first, allowing for a follow-up.

[0096] [Differentiation] In the embodiment described above, the action plan generation unit 140 and the mode determination unit 150 may be configured as a single unit. In this case, for example, the mode determination unit 150 may be included in the action plan generation unit 140, or the action plan generation unit 140 may be included in the mode determination unit 150. Also, in the embodiment, the first action plan (first target trajectory) was regenerated when the driving control according to the second action plan (second target trajectory) was executed, but the second action plan may be regenerated when the first action plan is executed, and the regenerated second action plan may be executed when the first action plan becomes impossible to execute.

[0097] Furthermore, in the embodiment, if, during driving control based on one action plan (e.g., a second action plan), it becomes impossible to generate an action plan (a second action plan), and after switching to the other action plan (a first action plan), if it becomes possible to generate an action plan (a second action plan) again, the system may switch to the action plan (a second action plan) that has become possible to generate again, or it may complete a predetermined driving control (e.g., lane change control) while remaining on the other action plan (a first action plan).

[0098] Furthermore, in this embodiment, the control by the first action plan generation unit 142 and the second action plan generation unit 143 described above may also be applied to driving control other than lane changes.

[0099] [Processing flow] Figure 7 is a flowchart showing an example of the processing flow performed by the automated driving control device 100 of the embodiment. In the example in Figure 7, the explanation will mainly focus on the driving control processing using multiple action plans among the processing performed by the automated driving control device 100. Furthermore, the processing shown in Figure 7 may be repeatedly executed at predetermined timings or predetermined cycles during the execution of automated driving or driving assistance.

[0100] In the example shown in Figure 7, the recognition unit 130 recognizes the surrounding conditions of the vehicle M (step S100). Next, the driver state detection unit 152A detects the state of the occupants of the vehicle M (step S110). Next, the first action plan generation unit 142 generates a first target trajectory (an example of a first action plan) based on the surrounding conditions of the vehicle M and the state of the occupants (step S120). Next, the second action plan generation unit 143 generates a second target trajectory (second action plan) based on the surrounding conditions of the vehicle M and the state of the occupants (step S130). Note that the processes in steps S120 and S130 may be executed in reverse order or in parallel.

[0101] Next, the selection control unit 144 selects either the first action plan or the second action plan based on predetermined conditions (step S140). The predetermined conditions may be, for example, a pre-set priority order or a user-specified item. Next, the driving control unit executes driving control based on the selected action plan (step S150). Next, the action plan generation unit 140 regenerates the other action plan that was not selected (step S160). Note that steps S150 and S160 may be executed in the reverse order.

[0102] Next, the action plan generation unit 140 determines whether or not it is impossible to perform driving control based on one of the selected target trajectories (step S170). If it is determined that driving control is impossible, the selection control unit 144 determines whether or not it is possible to perform driving control based on the other target trajectory (step S180). If it is determined that it is possible, the selection control unit 144 selects the other target trajectory (step S190). Next, the driving control unit executes driving control based on the selected target trajectory (step S200). Also, if it is determined in the process of step S180 that driving control based on the other target trajectory is not possible, the driving control unit terminates the current driving control (step S210). In this case, for example, control such as canceling a lane change or switching from the first driving mode to the second driving mode (manual driving) is executed. This completes the processing of this flowchart. Also, if it is determined in the process of step S170 that driving control based on one of the selected action plans is possible, the current driving control continues and the processing of this flowchart ends.

[0103] According to the embodiment described above, the automatic driving control device 100 (an example of a driving control device) includes: a recognition unit 130 that recognizes the surrounding conditions of the vehicle M; a detection unit 152 that detects at least one of the state of the occupants of the vehicle M and the instruction operations performed by the occupants; an action plan generation unit 140 that generates an action plan for the vehicle M based on the recognition results from the recognition unit 130 and the detection results from the detection unit 152; and a driving control unit (an example of a second control unit 160) that controls at least one of the steering and speed of the vehicle M to perform driving control of the vehicle M based on the action plan generated by the action plan generation unit 140. The motion plan generation unit 140 includes a first action plan generation unit 142 that generates a first action plan based on the recognition result from the recognition unit 130 and a pre-set rule, and a second action plan generation unit 143 that generates a second action plan based on the recognition result from the recognition unit 130 using a different logic than that used by the first action plan generation unit 142 to generate the first action plan. The motion plan generation unit 140 can improve the continuity of the vehicle's driving control by generating a more appropriate action plan according to the vehicle's situation by regenerating the other action plan after either the first or second action plan has been selected.

[0104] Specifically, in the embodiment described above, the two action plan generation units constantly monitor the content (output state) selected by the selection control unit 144, and even if their own action plan is not selected, they regenerate their own action plan so that they can realize a sequence in accordance with the other action plan. As a result, for example, even if the corresponding action plan of the other unit enters a state of functional degradation where it is unable to perform sequence processing, continuous driving control and driving assistance can be achieved without stopping the function by using the other action plan.

[0105] Furthermore, in this embodiment, among the rule-based and AI-based (machine learning-based) action plans, the one that was not selected (given up on) is used to generate a new action plan based on the assumption that it is possible to travel according to the selected action plan. This can be used as a backup when it becomes impossible to travel along the selected target trajectory (route) (given up on), thereby improving the continuity of travel control in the first driving mode, for example.

[0106] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Recognize the surrounding conditions of your vehicle, The system detects at least one of the following: the state of the occupants of the vehicle and the instructions and operations performed by the occupants. Based on the recognized and detected results, the vehicle's action plan is generated. Based on the generated action plan, the vehicle's steering and speed are controlled to perform driving control of the vehicle. Based on the results of the recognition and the pre-set rules, a first action plan is generated. Based on the results of the recognition, a second action plan is generated using a different logic than that used for generating the first action plan. After either the first action plan or the second action plan is selected, the other action plan is regenerated. Driving control device.

[0107] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0108] 1...Vehicle system, 10...Camera, 12...Radar device, 14...LIDAR, 16...Object recognition device, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 60...Map information processing device, 80...Driver's control unit, 100...Automatic driving control device, 120...First control unit, 130...Recognition unit, 140...Action plan generation unit, 141...Pre-processing unit, 142...First action plan generation unit, 143...Second action plan Generation unit, 144...Selection control unit, 150...Mode determination unit, 152...Detection unit, 152A...Driver state detection unit, 152B...Instruction content detection unit, 152C...Vehicle status detection unit, 154...Mode processing unit, 160...Second control unit, 162...Acquisition unit, 164...Speed ​​control unit, 166...Steering control unit, 170...HMI control unit, 180...Storage unit, 200...Driving force output device, 210...Brake device, 220...Steering device

Claims

1. A recognition unit that recognizes the surrounding conditions of the vehicle, A detection unit that detects at least one of the state of the occupants of the vehicle and the instructions and operations performed by the occupants, An action plan generation unit generates an action plan for the vehicle based on the recognition result from the recognition unit and the detection result from the detection unit, The vehicle includes a driving control unit that controls at least one of the steering and speed of the vehicle to perform driving control of the vehicle based on the action plan generated by the action plan generation unit, The aforementioned action plan generation unit, A first action plan generation unit generates a first action plan based on the recognition result from the recognition unit and pre-set rules, The system includes a second action plan generation unit that generates a second action plan using a logic different from that used to generate the first action plan by the first action plan generation unit, based on the recognition result by the recognition unit. The action plan generation unit regenerates the other action plan after either the first action plan or the second action plan has been selected. Driving control device.

2. The action plan generation unit regenerates the other action plan after the driving control unit has started driving control according to the selected action plan. The driving control device according to claim 1.

3. The action plan generation unit regenerates the other action plan based on the selected action plan. The driving control device according to claim 1.

4. If the driving control unit determines that driving control can be performed based on either the first or second action plan, and that driving control cannot be performed based on the other action plan, the action plan generation unit assumes that driving control of the other action plan is possible based on the first action plan, and regenerates the other action plan. The driving control device according to claim 1.

5. The action plan generation unit, if it determines that it is impossible to continue driving control during driving control based on either of the action plans, switches to driving control based on the other action plan. The driving control device according to claim 4.

6. The aforementioned driving control unit shall discontinue the execution of the driving control if it determines that driving control based on the other action plan is also impossible. The driving control device according to claim 5.

7. Either of the aforementioned action plans is the second action plan, The other action plan is the first action plan. The driving control device according to claim 5.

8. The aforementioned driving control includes lane change control of the vehicle, The action plan generation unit, when the lane change control is executed, regenerates the other action plan after either the first action plan or the second action plan has been selected. The driving control device according to claim 1.

9. Computers Recognize the surrounding conditions of your vehicle, The system detects at least one of the following: the state of the occupants of the vehicle and the instructions and operations performed by the occupants. Based on the recognized and detected results, the vehicle's action plan is generated. Based on the generated action plan, the vehicle's steering and speed are controlled to perform driving control of the vehicle. Based on the results of the recognition and the pre-set rules, a first action plan is generated. Based on the results of the recognition, a second action plan is generated using a different logic than that used for generating the first action plan. After either the first action plan or the second action plan is selected, the other action plan is regenerated. A method for controlling vehicle movement.

10. On the computer, Allow the vehicle to recognize its surroundings, The system detects at least one of the following: the state of the occupants of the vehicle and the instructions and operations performed by the occupants. Based on the recognized results and detected results, the system generates an action plan for the vehicle. Based on the generated action plan, the vehicle's steering and speed are controlled to perform driving control of the vehicle. Based on the recognized results and the pre-set rules, a first action plan is generated. Based on the recognized results, a second action plan is generated using a different logic than that used for generating the first action plan. After either the first action plan or the second action plan is selected, the other action plan is regenerated. program.

Images