Mobile device control device, mobile device control method, and program

Abstract

To perform more appropriate movement control based on the perception of the surrounding environment of the moving object. [Solution] The mobile body control device of the embodiment includes: a first recognition unit that recognizes a first demarcation line and a target that demarcates a travel path in the direction of travel of the mobile body using an image captured by an imaging unit; a second recognition unit that recognizes a second demarcation line that demarcates a travel path around the mobile body from map information based on the position information of the mobile body; a third recognition unit that recognizes a target in the direction of travel of the mobile body using a radar device; a determination unit that determines whether the first demarcation line and the second demarcation line coincide; and a movement control unit that controls the movement of the mobile body based on the determination result by the determination unit. The movement control unit controls the movement of the mobile body according to the travel path identified based on the target recognized by the third recognition unit when a target not recognized by the first recognition unit is recognized by the third recognition unit.

Description

Technical Field

[0001] The present invention relates to a movement control device, a movement control method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development related to autonomous driving technology has focused on research and development to further improve traffic safety and convenience. In this regard, conventionally, in the surrounding recognition of a moving object, when the road signs recognized from an image match the road signs stored in a storage unit, the reliability is increased, and when the reliability is equal to or higher than a predetermined value, the road signs stored in the storage unit are determined to be the road signs corresponding to the current position (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the conventional autonomous driving technology, due to the surrounding situation of the moving object or the like, the recognition accuracy of the surroundings using an image changes, so that the moving route cannot be accurately specified, and appropriate movement control may not be executed.

[0005] One of the objects of the present application is to provide a movement control device, a movement control method, and a program that can execute more appropriate movement control according to the recognition situation of the surroundings of a moving object in order to solve the above problems. And it contributes to the development of a sustainable transportation system.

Means for Solving the Problems

[0006] The mobile device control device, mobile device control method, and program according to this invention employ the following configuration. (1) A mobile body control device according to one aspect of the present invention comprises: a first recognition unit that recognizes a first demarcation line and a target that demarcates a travel path in the direction of travel of a mobile body using an image captured by an imaging unit; a second recognition unit that recognizes a second demarcation line that demarcates a travel path around the mobile body from map information based on the position information of the mobile body; a third recognition unit that recognizes a target in the direction of travel of the mobile body using a radar device; a determination unit that determines whether the first demarcation line and the second demarcation line coincide; and a movement control unit that controls the movement of the mobile body based on the determination result by the determination unit, wherein the movement control unit controls the movement of the mobile body according to a travel path identified based on the target recognized by the third recognition unit when a target not recognized by the first recognition unit is recognized by the third recognition unit.

[0007] (2) In the embodiment of (1) above, the movement control unit controls the movement of the moving body in accordance with the movement path identified based on the object recognized by the third recognition unit when the object recognized by the third recognition unit is recognized further inward in the width direction of the movement path from the perspective of the moving body.

[0008] (3) In the embodiment of (2) above, the target recognized by the third recognition unit is a target that is located further away from the moving body than the first demarcation line or target recognized by the first recognition unit.

[0009] (4) In the embodiment of (3) above, the target recognized by the third recognition unit includes a preceding moving body that moves in front of the moving body.

[0010] (5) In the embodiment of (3) above, the object recognized by the third recognition unit includes an object (nose object) that indicates a branch, merge, or end of a construction section of the travel path.

[0011] (6) In the embodiment of (3) above, the object recognized by the third recognition unit includes the tunnel side wall.

[0012] (7) In the embodiment of (1) above, the determination unit determines whether the recognition accuracy of the first recognition unit has decreased, and the movement control unit controls the movement of the moving body according to the movement path identified using the target recognized by the third recognition unit if it is determined that the recognition accuracy has decreased.

[0013] (8) In the embodiment of (1) above, the movement control unit controls the movement of the moving body based on a demarcation line that is within a predetermined distance from the target recognized by the third recognition unit.

[0014] (9) In the embodiment of (8) above, the movement control unit controls the movement of the moving body based on the boundary lines if the determination unit determines that the first boundary line and the second boundary line coincide, and if the boundary line exists within a predetermined distance from the target recognized by the third recognition unit.

[0015] (10): In the embodiment of (8) above, the movement control unit controls the movement of the moving body based on the boundary line, even if the positions of the first boundary line and the target recognized by the first recognition unit are within a predetermined distance, if the boundary line exists within a predetermined distance from the target recognized by the third recognition unit.

[0016] (11): In the embodiment of (1) above, the movement control unit adjusts the priority of the recognition result by the first recognition unit and the recognition result by the third recognition unit according to the distance between the moving body and a target located in the direction of travel of the moving body, and controls the movement of the moving body according to the movement path identified based on the recognition result with higher priority.

[0017] (12): A mobile body control method according to another aspect of the present invention is a mobile body control method in which a computer recognizes first demarcation lines and targets that define a travel path in the direction of travel of a mobile body using an image captured by an imaging unit, recognizes second demarcation lines that define a travel path around the mobile body from map information based on the position information of the mobile body, recognizes targets in the direction of travel of the mobile body using a radar device, determines whether the first demarcation lines and the second demarcation lines coincide, controls the movement of the mobile body based on the result of the determination, and controls the movement of the mobile body according to a travel path identified based on the targets recognized using the radar device if targets that were not recognized by the recognition process using the image are recognized by the recognition process using the radar device.

[0018] (13): A program according to another aspect of the present invention causes a computer to recognize first demarcation lines and targets that define a path in the direction of travel of a moving object using an image captured by an imaging unit, to recognize second demarcation lines that define a path around the moving object from map information based on the position information of the moving object, to recognize targets in the direction of travel of the moving object using a radar device, to determine whether the first demarcation lines and the second demarcation lines coincide, to control the movement of the moving object based on the determined result, and if a target that was not recognized by the recognition process using the image is recognized by the recognition process using the radar device, to control the movement of the moving object according to the path identified based on the target recognized using the radar device. [Effects of the Invention]

[0019] According to the embodiments described in (1) to (13) above, more appropriate movement control can be performed depending on the recognition status of the surroundings of the moving object. [Brief explanation of the drawing]

[0020] [Figure 1] This is a configuration diagram of a vehicle system 1 including a mobile control device according to an embodiment. [Figure 2]It is a functional block diagram of the first control unit 120 and the second control unit 160. [Figure 3] It is a diagram showing an example of a road (travel path) on which the host vehicle M travels. [Figure 4] It is a diagram for explaining specifying a travel lane based on recognition accuracy. [Figure 5] It is a flowchart showing an example of the flow of travel control processing in the embodiment.

Mode for Carrying Out the Invention

[0021] Hereinafter, embodiments of the movement control device, movement control method, and program of the present invention will be described with reference to the drawings. Hereinafter, a vehicle will be used as an example of a moving body, and an embodiment in which the movement control device is applied to an autonomous vehicle will be described. Autonomous driving is, for example, automatically controlling one or both of the steering and speed of a vehicle to execute driving control. The above-described driving control may include various driving controls such as, for example, LKAS (Lane Keeping Assistance System), ALC (Automated Lane Change), ACC (Adaptive Cruise Control System), TJP (Traffic Jam Pilot), and CMBS (Collision Mitigation Brake System). Further, in the autonomous vehicle, driving control (so-called manual driving) by manual operation of a user (for example, a passenger) of the vehicle may be executed. The moving body may include, in addition to a vehicle, for example, a ship that can move on the ground (road) like a hovercraft, an aircraft that can travel on a road, a standing vehicle having a power unit, and the like.

[0022] [Overall Configuration] Figure 1 is a diagram showing the configuration of a vehicle system 1 including a mobile control device according to an embodiment. The vehicle on which the vehicle system 1 is mounted (hereinafter referred to as "vehicle M") is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle or a micromobility, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a battery (storage battery) such as a secondary battery or fuel cell.

[0023] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driver control device 80, an automatic driving control device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Note that the configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The combination of camera 10, radar device 12, and LiDAR 14 is an example of a "detection device DD". HMI 30 is an example of an "output device". Automatic driving control device 100 is an example of a "mobile vehicle control device".

[0024] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle M on which the vehicle system 1 is installed. When imaging the front, camera 10 is mounted on the top of the front windshield, behind the rearview mirror, or on the front of the vehicle body. When imaging the rear, camera 10 is mounted on the top of the rear windshield or on the tailgate. When imaging the side, camera 10 is mounted on the door mirror or the like. Camera 10 periodically and repeatedly images the area around the vehicle M. Camera 10 may also be a stereo camera.

[0025] The radar device 12 emits radio waves (radar) such as millimeter waves around the vehicle M and detects radio waves (reflected waves) reflected by surrounding objects to detect at least the position (distance and bearing) of an object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0026] The LIDAR 14 illuminates the area around the vehicle M with light and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time from emission to reception. The emitted light is, for example, pulsed laser light. The LIDAR 14 can be attached to any location on the vehicle M.

[0027] The communication device 20 communicates with other vehicles in the vicinity of its own vehicle M, terminal devices of users using its own vehicle M, or various server devices, for example, by utilizing networks such as cellular networks, Wi-Fi networks, Bluetooth®, DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet.

[0028] The HMI30 outputs various information to the occupants of the vehicle M (including the driver) and accepts input operations from the occupants. The HMI30 includes, for example, various display devices, speakers, buzzers, touch panels, switches, keys, microphones, etc.

[0029] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting yaw rate (for example, the angular velocity of rotation around the vertical axis passing through the center of gravity of the vehicle M), and an orientation sensor for detecting the orientation of the vehicle M. The vehicle sensor 40 may also be provided with a position sensor for detecting the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50. The vehicle sensor 40 may derive the speed of the vehicle M from the difference (i.e., distance) of position information at a predetermined time in the position sensor. The vehicle sensor 40 may also be provided with sensors for weather acquisition (for example, a humidity sensor, a rain sensor). The results detected by the vehicle sensor 40 are output to the automatic driving control device 100.

[0030] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, speaker, touch panel, keys, etc. The GNSS receiver 51 may be provided on the vehicle sensor 40. The navigation HMI 52 may be partially or completely shared with the HMI 30 described above. The route determination unit 53 determines, for example, a route (hereinafter referred to as a route on the map) from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52, by referring to the first map information 54. The first map information 54 is information in which the shape of a road is represented by links indicating roads (an example of a travel route) and nodes connected by those links. The first map information 54 may also include POI (Point of Interest) information, etc. The route on the map is output to the MPU 60. The navigation device 50 may perform route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may transmit its current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server. The navigation device 50 outputs the determined route on the map to the MPU 60.

[0031] The MPU 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the map route provided by the navigation device 50 into multiple blocks (for example, every 100m with respect to the vehicle's direction of travel) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 makes decisions such as which lane from the left the vehicle should travel in. If there is a branching point on the map route, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel along a reasonable route to proceed to the branching point.

[0032] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, the number of lanes (number of travel routes), the type and shape of road markings (hereinafter referred to as markings), information on the center of the lanes, or information on road boundaries. The second map information 62 may also include information on whether the road boundary is a boundary (physical boundary) that includes a structure that vehicles cannot pass through (including crossing and contact). A physical boundary is, for example, a guardrail, curb, median strip, fence, tunnel side wall, nose marker (soft nose or hard nose), etc. "Not passable" may also include the existence of a low step that can be passed through if the vehicle is willing to tolerate vibrations that would not normally occur. The second map information 62 may also include road shape information, traffic regulation information, address information (address and postal code), facility information, parking information, telephone number information, etc. Road shape information includes, for example, the curvature of the road (which may be rephrased as radius of curvature; the same applies hereafter), width, road surface gradient, branching points, and merging points. The second map information 62 may be updated as needed by the communication device 20 communicating with an external device. The first map information 54 and the second map information 62 may be provided together as map information. The map information may also be stored in the storage unit 190.

[0033] The driver control unit 80 includes, for example, a steering wheel, an accelerator pedal, and a brake pedal. The driver control unit 80 may also include a shift lever, a modified steering wheel, a joystick, or other controls. Each control of the driver control unit 80 is equipped with an operation detection unit that detects, for example, the amount of operation performed by the driver on the control unit or whether or not an operation has been performed. The operation detection unit detects, for example, the steering angle and steering torque of the steering wheel, the amount of depression of the accelerator pedal and brake pedal, etc. The operation detection unit then outputs the detection result to the automatic driving control device 100, or to one or both of the driving force output device 200, the brake device 210, and the steering device 220.

[0034] The automatic driving control device 100 performs various driving controls belonging to automatic driving on its own vehicle M. The automatic driving control device 100 includes, for example, a first control unit 120, a second control unit 160, an HMI control unit 180, and a storage unit 190. The first control unit 120, the second control unit 160, and the HMI control unit 180 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The above-mentioned program may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the automatic driving control device 100, or it may be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the automatic driving control device 100 when the storage medium (non-transient storage medium) is inserted into a drive device or card slot.

[0035] The storage unit 190 may be implemented using the various storage devices described above, or an EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 190 may store, for example, various information and programs as described in the embodiment. The storage unit 190 may also store map information (for example, first map information 54 and second map information 62).

[0036] Figure 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, a recognition unit 130 and an action plan generation unit 140. The first control unit 120 implements, for example, functions using AI (Artificial Intelligence) and functions using a pre-defined model in parallel. For example, the function of "recognizing intersections" may be implemented by simultaneously performing intersection recognition using deep learning and recognition based on pre-defined conditions (such as pattern-matchable signals and road markings), scoring both, and comprehensively evaluating them. This ensures the reliability of autonomous driving. The first control unit 120 also performs control related to the autonomous driving of its own vehicle M based on instructions from, for example, the MPU 60 and the HMI control unit 180.

[0037] The recognition unit 130 recognizes the surrounding conditions of the vehicle M based on the detection results of the detection device DD (information input from the camera 10, radar device 12, and LIDAR 14). For example, the recognition unit 130 performs sensor fusion processing on some or all of the detection results from the camera 10, radar device 12, and LIDAR 14 to recognize the position (relative position), size, speed (relative speed), acceleration, and other conditions of targets (objects) present around the vehicle M (within a predetermined distance). Targets recognized by the recognition unit 130 may include, for example, physical boundaries that demarcate roads (travel paths) (for example, physical boundaries included in map information), obstacles such as signs temporarily placed on the road, other vehicles, pedestrians, cyclists, and other traffic participants. The position of a target is recognized as a position on an absolute coordinate system with the vehicle M's representative point (such as the center of gravity or drive axis center) as the origin, and is used for control. The position of a target may be represented by its center of gravity, a representative point such as a corner, or by the represented region. The "state" of the target may include, for example, the acceleration or jerk of a moving object such as another vehicle, or its "action state" (for example, whether the other vehicle is changing lanes or is about to change lanes).

[0038] Furthermore, the recognition unit 130 recognizes, for example, stop lines, red lights, toll booths, other road events, road signs, and markings drawn on the road (e.g., speed limits). The recognition unit 130 also includes, for example, a first recognition unit 132, a second recognition unit 134, and a third recognition unit 136. Details of these functions will be described later.

[0039] The action plan generation unit 140 generates an action plan for the vehicle M to drive (move) by automatic driving based on the recognition results of the recognition unit 130, etc. For example, the action plan generation unit 140 will, in principle, drive in the recommended lane determined by the recommended lane determination unit 61, and further generates a target trajectory (target driving route) that the vehicle M will automatically (without driver operation) travel in the future, based on the recognition results of the recognition unit 130 and the surrounding road shape based on the current position of the vehicle M obtained from map information, so as to correspond to the surrounding conditions of the vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a sequence of points (trajectory points) that the vehicle M should reach. The trajectory points are points that the vehicle M should reach at predetermined driving distances (e.g., a few [m]) along the road, and separately, target speed and target acceleration at predetermined sampling times (e.g., a few tenths [sec]) are generated as part of the target trajectory. Alternatively, the trajectory points may be the positions that the vehicle M should reach at the sampling time for each predetermined sampling time. In this case, information about the target velocity and target acceleration is represented by the interval between trajectory points.

[0040] The action plan generation unit 140 may set automated driving events when generating a target trajectory. These events include, for example, a lane departure prevention event that drives the vehicle M without deviating from its lane, a constant speed driving event that drives the vehicle M in the same lane at a constant speed, a follow driving event that makes the vehicle M follow the nearest vehicle that is within a predetermined distance (for example, within 100 [m]) in front of the vehicle M, a lane change event that makes the vehicle M change lanes from its own lane to an adjacent lane, a branching event that makes the vehicle M branch off to the destination lane at a road branching point, a merging event that makes the vehicle M merge onto the main road at a merging point, and a takeover event that ends automated driving and switches to manual driving. Furthermore, events may include, for example, an overtaking event in which the vehicle M changes lanes to an adjacent lane, overtakes a preceding vehicle in the adjacent lane, and then changes lanes back to the original lane, and an avoidance event in which the vehicle M brakes and / or steers to avoid an obstacle in front of the vehicle M.

[0041] Furthermore, the action plan generation unit 140 may, for example, change an event already determined for the current section to another event, or set a new event for the current section, depending on the surrounding conditions of the vehicle M recognized while the vehicle M is in motion. Also, the action plan generation unit 140 may change an event already set for the current section to another event, or set a new event for the current section, depending on the occupant's operation to the HMI 30. The action plan generation unit 140 generates a target trajectory according to the set event.

[0042] Furthermore, the action plan generation unit 140 includes, for example, a determination unit 142, a specification unit 144, and a driving control unit 146. The driving control unit 146 is an example of a "movement control unit." Details of these functions will be described later.

[0043] The second control unit 160 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the vehicle M passes through the target trajectory generated by the action plan generation unit 140 at the scheduled time.

[0044] The second control unit 160 includes, for example, a target trajectory acquisition unit 162, a speed control unit 164, and a steering control unit 166. The target trajectory acquisition unit 162 acquires information on the target trajectory (trajectory points) generated by the action plan generation unit 140 and stores it in memory (not shown). The speed control unit 164 controls the driving force output device 200 or the brake device 210 based on the speed elements associated with the target trajectory stored in memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in memory. The processing of the speed control unit 164 and the steering control unit 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 performs a combination of feedforward control according to the curvature of the road in front of the vehicle M and feedback control based on the deviation from the target trajectory.

[0045] Returning to Figure 1, the HMI control unit 180 notifies the occupant of predetermined information via the HMI 30 and receives information input via the HMI 30. The predetermined information includes, for example, information related to the driving of the vehicle M, such as information regarding the status of the vehicle M and information regarding driving control. Information regarding the status of the vehicle M includes, for example, the speed of the vehicle M, engine speed, and shift position. Information regarding driving control includes, for example, information to inquire whether or not driving control by automated driving is being performed, whether or not to start automated driving, information regarding the status of driving control by automated driving, information regarding the level of automation, and information prompting the driver to take action when switching from automated driving to manual driving. The predetermined information may also include information regarding the surrounding conditions recognized by the detection device DD. The predetermined information may also include information unrelated to the driving of the vehicle M, such as content stored on a storage medium such as a television program or DVD (e.g., a movie). The predetermined information may also include, for example, the current position and destination in automated driving, and information regarding the remaining fuel level of the vehicle M. The HMI control unit 180 may output the information received by the HMI 30 to the communication device 20, the navigation device 50, the first control unit 120, etc.

[0046] Furthermore, the HMI control unit 180 may output to the HMI 30 information such as inquiry information for the occupants and processing results from the first control unit 120 and the second control unit 160. In addition, the HMI control unit 180 may transmit various information to be output to the HMI 30 to terminal devices used by the occupants of the vehicle M via the communication device 20.

[0047] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the second control unit 160 or information input from the accelerator pedal of the driver control unit 80.

[0048] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the second control unit 160 or from the brake pedal of the driver control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may be equipped with a backup mechanism that transmits the hydraulic pressure generated by the operation of the brake pedal to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the second control unit 160 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0049] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the second control unit 160 or information input from the steering wheel of the driver control unit 80.

[0050] [Recognition unit and action plan generation unit] Next, we will explain in detail the functions of the recognition unit 130 (mainly the first recognition unit 132, the second recognition unit 134, and the third recognition unit 136) and the action plan generation unit 140 (mainly the determination unit 142, the identification unit 144, and the driving control unit 146). The following explanation will also include driving control for the vehicle M itself.

[0051] Figure 3 shows an example of a road (travel route) on which the vehicle M travels. In the example in Figure 3, lane markings CL1 to CL3 recognized by the camera 10 and lane markings ML1 to ML3 obtained from map information (for example, second map information 62) based on the position information of the vehicle M are shown. The road RD1 shown in Figure 3 has two lanes L1 and L2 that can be traveled in the same direction. In the map information, lane L1 is demarcated by lane markings ML1 and ML2, and lane L2 is demarcated by lane markings ML2 and ML3. In the example in Figure 3, lane markings CL1 to CL3 are an example of "first lane markings," and lane markings ML1 to ML3 are an example of "second lane markings." Furthermore, below, lane markings CL1 to CL3 may be referred to as "camera lane markings CL1 to CL3," and lane markings ML1 to ML3 may be referred to as "map lane markings ML1 to ML3." Furthermore, if camera section lines CL1 to CL3 are not distinguished, they may simply be referred to as "camera section line CL," and if map section lines ML1 to ML3 are not distinguished, they may simply be referred to as "map section line ML."

[0052] Furthermore, in the example in Figure 3, when viewed from lane L1 in the direction of travel (X-axis direction in the figure), a landmark (e.g., a physical boundary such as a guardrail) OB1 is located to the left (far out) of road RD1, along the extension direction of lane L2. In addition, in the example in Figure 3, in section S1 from point P1 to P2 on road RD1, since road construction is underway on lane L2 (or just before the construction area), the actual lane lines separating lane L1 and lane L2 are not drawn, and a landmark (e.g., a sign) OB2 is placed on lane L2 to indicate that construction is underway or to encourage drivers of surrounding vehicles to use lane L1. Furthermore, in the example in Figure 3, a landmark (e.g., a soft nose) OB3 is placed to indicate the end of a junction (or the end of a construction section). As viewed from the position of the vehicle M shown in Figure 3, landmarks OB2 and OB3 are located inside the width direction (lateral direction) of road RD1 compared to landmark OB1. Furthermore, targets OB2 and OB3 are located further away than target OB1 from the position of the vehicle M shown in Figure 3, and target OB3 is located even further away than target OB2.

[0053] In the example in Figure 3, the vehicle M is traveling in lane L1 at speed VM, and another vehicle m1 is traveling in lane L1 ahead of vehicle M at speed Vm1. Figure 3 shows the travel trajectory (movement trajectory) K1 of the other vehicle m1. The other vehicle m1 is an example of a "preceding moving object". In the example in Figure 3, LKAS control is performed on vehicle M, and automatic driving is performed so that vehicle M is kept within the driving lane (in other words, so that vehicle M does not deviate from the driving lane). In this case, the automatic driving control device 100 identifies the driving lane of vehicle M based on recognized lane markings, generates a target trajectory so that vehicle M travels in the center of the identified driving lane, and performs driving control (movement control) including steering of vehicle M so that vehicle M travels along the generated target trajectory. During driving control, feedforward control and feedback control are performed as needed based on the target trajectory and the position of vehicle M to adjust the steering angle, speed, etc. of vehicle M.

[0054] [First recognition unit 132] The first recognition unit 132 uses the image captured by the camera 10 to recognize camera lane markings (first lane markings) CL and objects (camera objects) that demarcate the lanes (travel paths) in the vicinity of the vehicle M, including the direction of travel. For example, the first recognition unit 132 performs known analysis processing (e.g., edge extraction, feature extraction such as color, shape, and size, pattern matching processing, character recognition processing, etc.) on the image captured by the camera 10 (hereinafter referred to as the camera image), and recognizes the camera lane markings CL and camera objects near the road (within a predetermined distance from the road and including on the road RD1) on which the vehicle M is traveling, based on the image analysis results. When recognizing camera lane markings CL, the first recognition unit 132 extracts edge points with a large difference in brightness from adjacent pixels in the camera image and connects the edge points to recognize the camera lane markings CL in the image plane. The first recognition unit 132 also converts the position of the camera lane markings CL to the vehicle coordinate system (e.g., XY plane coordinates in Figure 3) with respect to the position of a representative point of the vehicle M.

[0055] In the example shown in Figure 3, the first recognition unit 132 can recognize camera lane markings CL1 to CL3 and object targets OB1 to OB3 based on the camera image. The first recognition unit 132 also recognizes the position and speed Vm1 of another vehicle m1 that is preceding the vehicle M. Another vehicle m1 is also an example of an "object." The first recognition unit 132 may also recognize the curvature of lanes L1 and L2 and the road surface gradient based on the camera image, and may recognize the amount of change in curvature of camera lane markings CL1 to CL3. The amount of change in curvature is, for example, the rate of change over time of the curvature of camera lane markings CL1 to CL3 recognized by the camera 10 at a distance x [m] in front of the vehicle M. The first recognition unit 132 may also recognize the area in which the vehicle M can travel (move) (or an area without obstacles such as physical boundaries) as a free space (hereinafter referred to as camera free space) based on the recognized camera lane markings CL and camera object targets.

[0056] Furthermore, the recognition accuracy of the first recognition unit 132 may decrease if the target is located at a long distance (more than a predetermined distance from the vehicle M), or due to surrounding conditions such as weather (e.g., bad weather like thunderstorms, direct sunlight, etc.), time of day (nighttime, etc.), or the occurrence of blind spots due to traffic congestion, etc.

[0057] [Second recognition unit 134] The second recognition unit 134, based on the position information of its own vehicle M acquired by the vehicle sensor 40 or the GNSS receiver 51, refers to map information (first map information 54, second map information 62) and recognizes from the map information map lane markings (second lane markings) that demarcate lanes (travel paths) in the vicinity including the direction of travel of its own vehicle M.

[0058] In the example shown in Figure 3, the second recognition unit 134 can recognize map lane markings ML1 to ML3 and landmark OB1 based on map information. The second recognition unit 134 may also recognize the curvature of lanes L1 and L2 and the road surface gradient from the map information, and may recognize the amount of change in curvature of map lane markings ML1 to ML3. However, landmarks installed for road construction, such as landmark OB2, are not reflected in the map information and therefore cannot be recognized by the second recognition unit 134. Furthermore, the second recognition unit 134 cannot recognize other vehicles m1. The second recognition unit 134 can communicate with an external device via the communication device 20 and obtain information from the external device indicating that construction is underway near section S1 (within a predetermined distance from section S1, and also within section S1).

[0059] [Third recognition unit 136] The third recognition unit 136 recognizes objects (radar targets) in the vicinity, including the direction of travel of the vehicle M, based on the detection results from a detection device DD other than the camera 10 (for example, the radar device 12). For example, the third recognition unit 136 recognizes the position (distance and direction) and speed of radar targets in the direction of travel of the vehicle M from the detection results from the radar device 12. Alternatively, the third recognition unit 136 may recognize surrounding radar targets, including the direction of travel of the vehicle M, based on the detection results from the LIDAR 14 instead of (or in addition to) the radar device 12.

[0060] In the example shown in Figure 3, the third recognition unit 136 recognizes targets OB1 to OB3 and other vehicles m1 that are located around the vehicle M. The third recognition unit 136 may also recognize an area where the vehicle M can travel (or an area without obstacles such as physical boundaries) as a free space (hereinafter referred to as radar free space) based on the recognized radar targets.

[0061] Furthermore, if there are reflective objects (reflective areas) that reflect radio waves or light in the vicinity of the section of travel of the vehicle M, such as tunnels, underpasses, or under bridges, the third recognition unit 136 will experience a decrease in the recognition accuracy of targets OB1 to OB3 because the radio waves and light are scattered by the reflective objects (such as tunnel walls), resulting in too much reflected light. Thus, the recognition accuracy of the third recognition unit 136 may decrease depending on the surrounding conditions, similar to the first recognition unit 132. However, the third recognition unit 136 can recognize targets near the tunnel entrance before entering the tunnel without being significantly affected by scattered reflections. Therefore, the third recognition unit 136 may, for example, recognize the corner (end) of the tunnel entrance as a substitute for a nose target.

[0062] [Determination unit 142] The determination unit 142 determines whether the camera lane markings CL1 to CL3 recognized by the first recognition unit 132 match the map lane markings ML1 to ML3 recognized by the second recognition unit 134. For example, the determination unit 142 derives the degree of match between lane markings CL1 and ML1 located closest to the right side of the vehicle M, the degree of match between lane markings CL2 and ML2 located closest to the left side of the vehicle M, and the degree of match between lane markings CL3 and ML3 on the adjacent lane side. The determination unit 142 then determines that the camera lane markings CL and map lane markings ML match if the derived degree of match is greater than or equal to a threshold, and determines that they do not match if the degree of match is less than the threshold. The determination of whether or not they match may be performed repeatedly at predetermined timings or periods.

[0063] For example, the determination unit 142 superimposes camera lane lines CL1, CL2, and CL3 on the vehicle coordinate system plane (XY plane) based on the position of the representative point of the vehicle M, and also superimposes map lane lines ML1, ML2, and ML3. When determining the lane lines to be compared (lane lines CL1 and ML1, lane lines CL2 and ML2, lane lines CL3 and ML3), the determination unit 142 determines that the lane lines match if the degree of matching of all lane lines is above a threshold, and determines that they do not match if at least one lane line is below the threshold.

[0064] Here, the degree of agreement in the agreement determination described above refers to, for example, the degree of deviation in the road width direction (moving road width direction, lateral direction, Y-axis direction in the figure) (matching distance, moving road width direction deviation). In the example of Figure 3, agreement determination may be made using the lateral positional deviation D1 of the lane markings CL1 and ML1, the lateral positional deviation D2 of the lane markings CL2 and ML2, and the lateral positional deviation D3 of the lane markings CL3 and ML3, respectively. Alternatively, agreement determination may be made using the average value, maximum value, or minimum value of the deviation amounts D1, D2, and D3.

[0065] Furthermore, the degree of agreement may, for example, be the degree of the angle formed by the two dividing lines being compared (the degree of deviation corresponding to the deviation angle), instead of (or in addition to) the amount of horizontal displacement described above. For example, the larger the degree of deviation, the smaller the degree of agreement. In the example in Figure 3, only the angle θ formed by dividing lines CL2 and ML2 is shown, but agreement may also be determined using the angle formed by dividing lines CL1 and ML1 or the angle formed by dividing lines CL3 and ML3, or the average value, maximum value, or minimum value of each angle θ may be used to determine agreement.

[0066] Furthermore, the degree of agreement may be determined by the degree of difference (magnitude) in the amount of curvature change of the lane markings, instead of (or in addition to) the degree of deviation corresponding to the degree of lateral positional displacement or the deviation angle formed by the lane markings mentioned above. The amount of curvature change is mainly used when the lane is a curved road as shown in Figure 3. For example, the determination unit 142 may determine agreement using the difference in the amount of curvature change between lane markings CL1 and ML1, the difference in the amount of curvature change between lane markings CL2 and ML2, and the difference in the amount of curvature change between lane markings CL3 and ML3, respectively, or it may determine agreement using the average value of the differences, or the maximum or minimum value of the differences.

[0067] Furthermore, the system determines whether the recognition accuracy of the first recognition unit 132 and the third recognition unit 136 has decreased. For example, the system determines whether the driving scene (driving conditions) of the vehicle M is a difficult scene for both the first recognition unit 132 and the third recognition unit 136, and if it determines that the vehicle is driving in a difficult scene, it determines that the recognition accuracy has decreased. For example, driving scenes near locations where there are nose-shaped landmarks such as tunnel entrances, junction ends, and median strip ends in the direction of travel, as well as road gradients and bad weather, are likely to affect the camera image and are therefore difficult scenes for the first recognition unit 132 to recognize (camera difficult scenes). Also, driving scenes in locations where there are reflective objects such as inside tunnels, under bridges, and under elevated structures are likely to affect the detection results using the radar device 12 and are therefore difficult scenes for the third recognition unit 136 to recognize (radar difficult scenes). Therefore, the determination unit 142 determines that the recognition accuracy of the target recognition unit has decreased when the driving scene of its own vehicle M is a difficult scene. The determination unit 142 may also determine whether or not the vehicle is driving in a difficult scene (whether or not the current driving scene is a difficult scene).

[0068] [Specific section 144] The identification unit 144 identifies the driving lane of the vehicle M based on the recognition results from the first recognition units 132 to the third recognition units 136 and the determination results from the determination unit 142. For example, if the determination unit 142 determines that the camera lane markings CL and the map lane markings ML match, the identification unit 144 identifies the lane demarcated by camera lane markings CL1 and CL2, or the lane demarcated by map lane markings ML1 and ML2, as the driving lane of the vehicle M.

[0069] Furthermore, the identification unit 144 may estimate a physical boundary line based on the camera target recognized by the first recognition unit 132 and the radar target recognized by the third recognition unit 136, and identify the driving lane of the vehicle M based on the estimated physical boundary line, the camera lane markings, and the map lane markings. The specific processing details of the determination unit 142 and the identification unit 144 will be described later.

[0070] [Driving control unit 146] The driving control unit 146 determines driving control for its own vehicle M based on the recognition results of the first recognition unit 132 to the third recognition unit 136 and the driving lane identified by the identification unit 144, and generates a target trajectory based on the determined driving control. "Determining driving control" may include, for example, determining the content (type) of the driving control or deciding whether or not to execute (suppress) the driving control. Furthermore, "executing driving control" may include, for example, switching and executing the content of the driving control, as well as continuing the driving control that is already being executed. Suppressing driving control may include not only not executing driving control, but also lowering the level of automation of the driving control.

[0071] For example, when the driving control unit 146 performs LKAS control as driving control (autonomous driving), it controls the steering of the vehicle M to drive the vehicle M, at least, so that the representative point of the vehicle M passes through the center of the driving lane, based on the driving lane identified by the identification unit 144. In addition, if the determination unit 142 determines that the camera lane markings CL and the map lane markings ML do not match, and that the recognition accuracy of the first recognition unit 132 and the third recognition unit 136 has decreased, the driving control unit 146 terminates the driving control for the vehicle M and switches to manual driving by the driver, or performs control to reduce the automation level of the autonomous driving. The automation level includes, for example, a first level, a second level with a lower degree of automation of driving control than the first level, and a third level with a lower degree of automation of driving control than the second level. The automation level may also include a fourth level with a lower degree of automation of driving control than the third level. The automation level may be a level defined by standardized information or regulations, or it may be an index value set independently of such standards. Therefore, the types, content, and number of automation levels are not limited to the following examples. A low degree of automation in driving control means, for example, that the automation rate in driving control is small and the tasks imposed on the driver are large (severe). Also, a low degree of automation in driving control means that the degree to which the automatic driving control device 100 controls the steering or speed of the vehicle M is low (the degree to which the driver needs to intervene in steering or speed operation is high). Tasks imposed on the driver include, for example, monitoring the surroundings of the vehicle M and operating the driver controls. Operating the driver controls includes, for example, the state in which the driver is holding the steering wheel (hereinafter referred to as the hands-on state). Tasks imposed on the driver are, for example, tasks (driver tasks) necessary for the driver to maintain the automatic driving of the vehicle M. Therefore, if the driver is unable to perform the assigned tasks, the automation level will decrease.

[0072] Furthermore, at Level 1, there are no tasks assigned to the driver (or the tasks assigned to the driver are the least demanding), so driving control is permitted even when the driver of the vehicle M is not holding the steering wheel (hereinafter referred to as the hands-off state). At Level 2, the tasks assigned to the driver include, for example, monitoring the surroundings of the vehicle M (especially the area in front). At Level 3, the tasks assigned to the driver include, for example, monitoring the surroundings of the vehicle M in addition to being in a hands-on state. At Level 4, the tasks assigned to the driver include, for example, monitoring the surroundings of the vehicle M and being in a hands-on state, in addition to controlling the steering and speed of the vehicle M using the driving control device 80. In other words, at Level 4, the driver can take over immediately, and the tasks assigned to the driver are the most demanding. The content of the driving control and the tasks assigned to the driver at each automation level are not limited to the examples described above. The automated driving control device 100 performs driving control at one of the Levels 1 to 4 based on the surrounding conditions of the vehicle M and the tasks the driver is currently performing. The details of the driving control operations are output from the HMI 30 by the HMI control unit 180 and notified to the driver.

[0073] [Specific processing details by the determination unit 142 and the identification unit 144] Next, the specific processing details by the determination unit 142 and the identification unit 144 will be explained. Figure 4 is a diagram illustrating the identification of a driving lane based on recognition accuracy. As shown in the example in Figure 4, the identification unit 144 identifies the driving lane of the vehicle M based on the determination result by the determination unit 142, the information of the camera lane line CL and map lane line ML, and the estimation result of the physical boundary line. Here, the determination unit 142, based on the various input information, performs not only a determination of the match between the camera lane line CL and the map lane line ML, but also a determination of difficult scenes based on the various information, and a determination of whether or not a predetermined target (for example, a nose target) has been recognized as a recognition result. Difficult scenes include camera difficult scenes in which the recognition accuracy using the camera image by the first recognition unit 132 decreases, and radar difficult scenes in which the recognition accuracy using the detection result of the radar device 12 by the third recognition unit 136 decreases.

[0074] [Camera-challenged scene assessment] When determining a scene that is difficult for the camera to capture, the determination unit 142 acquires information about the surrounding conditions (driving scene) of the vehicle M, and determines that the surrounding conditions are a scene that is difficult for the camera to capture if the acquired surrounding conditions meet predetermined conditions. The predetermined conditions include the presence of nose landmarks such as tunnel entrances or branch ends (or merging ends) within a predetermined distance in the direction of travel of the vehicle M, the presence of a road surface gradient of a predetermined value or higher, and the surrounding weather conditions being a specific type of weather (e.g., thunderstorm, heavy rain, typhoon, snow). Information on the presence or absence of tunnels, branches, and merging points, as well as road shape such as road surface gradient, may be acquired from map information. In addition, weather conditions may be acquired from a weather acquisition sensor included in the vehicle sensor 40, or the weather conditions may be acquired from an external device connected to the vehicle M via the communication device 20 based on location information. Furthermore, in determining a scene that is difficult for the camera to capture, information on camera-free space based on camera images captured by the camera 10 may be used in addition to (or instead of) the surrounding conditions information.

[0075] [Nose Judgment] Furthermore, the determination unit 142 determines the presence or absence of a nose target based on the radar-free space and / or surrounding conditions acquired from an external device via the communication device 20, based on the detection results of the radar device 12 and / or map information such as junctions and construction sections. For example, since the hard nose at the end of a junction is a road structure (having a three-dimensional shape), it is easier to determine its presence or absence from the detection results of the radar device 12 than from recognition from camera images. Therefore, the determination unit 142 performs the above-mentioned nose determination using the recognition results recognized from the detection results of the radar device 12 by the third recognition unit 136, rather than the recognition results based on camera images by the first recognition unit 132. In addition to determining the presence or absence of a nose target, the determination may also include whether or not a physical boundary (or boundary line) is recognized near the junction.

[0076] [Radar-related scene assessment] When determining a radar-incompatible scene, the determination unit 142, for example, refers to map information based on the position information of the vehicle M and obtains the road shape corresponding to the position of the vehicle M obtained from the map information. Then, if the road shape around the vehicle obtained is a predetermined shape, it is determined that it is a radar-incompatible scene. The predetermined shape is a tunnel, under a bridge, or under an overpass. For example, in tunnels and under overpasses, there is too much reflection of radio waves (radar) due to reflective objects such as ceilings and side walls, so it may not be possible to correctly recognize the position of radar targets. Therefore, if the vehicle M is traveling on such a road shape, it is determined that the current driving scene of the vehicle M is a radar-incompatible scene. Alternatively, the determination unit 142 may determine that it is not a radar-incompatible scene.

[0077] The identification unit 144 estimates the physical boundary line around the vehicle M (a lane line corresponding to the extension direction of the physical boundary) based on, for example, the information on the camera-free space and radar-free space described above, the determination result of camera-difficult scenes, the determination result of radar-difficult scenes, and the determination result of nose detection. For example, the identification unit 144 estimates the physical boundary line from a matching area by combining the camera-free space and the radar-free space. Furthermore, if the surrounding conditions (driving scene) of the vehicle M are radar-difficult scenes but not camera-difficult scenes, the identification unit 144 may estimate the physical boundary line based on the position of a camera target recognized using the camera image (for example, target OB1 shown in Figure 3). Also, if the surrounding conditions of the vehicle M are camera-difficult scenes but not radar-difficult scenes, the identification unit 144 may estimate the physical boundary line based on radar targets (for example, targets OB1 to OB3) acquired by the radar device 12. Furthermore, if the surrounding conditions are neither a camera-dependent scene nor a radar-dependent scene, the specific unit 144 may prioritize the recognition results from the camera image to estimate the physical boundary line. This allows the recognition results from the camera image to be prioritized under normal circumstances, while in situations where the recognition accuracy from the camera image decreases, the recognition of the driving lane can be supplemented by the detection results from the radar device 12.

[0078] Here, conditions for estimating the physical boundary using the detection results of the radar device 12 may include the fact that it is a scene that is difficult for the camera to capture, or the results of nose detection. However, if the map information has not been updated (for example, if it is old information from more than a predetermined period ago), the recognition accuracy of the second recognition unit 134 (for example, the accuracy of information such as branching points) may not be good. Therefore, in this embodiment, recognition is always performed by the third recognition unit 136, and if the position of the radar target or the position of the preceding vehicle (driving trajectory, etc.) recognized by the third recognition unit 136 deviates by a predetermined value or more from the result of the agreement judgment between the camera lane line CL and the map lane line ML, or the camera target recognized based on the camera lane line CL or the camera image, the physical boundary may be estimated using the detection results of the radar device 12.

[0079] [Processing by the specific unit 144 and the travel control unit 146] The identification unit 144 identifies the driving lane of the vehicle M based on the information obtained from the estimation of the physical boundary line and the information obtained from the camera lane markings CL and map lane markings ML (including the matching judgment result). The driving control unit 146 generates a target trajectory for the vehicle M based on the driving lane identified by the identification unit 144. At least some of the functions of the identification unit 144 may be included in the driving control unit 146.

[0080] For example, in section S1 shown in Figure 3, some camera lane markings are not recognized because the lane markings separating lanes L1 and L2 are not drawn due to the influence of construction zones, etc., and the matching judgment determines that the camera lane marking CL and the map lane marking ML do not match. In addition, in conditions such as bad weather, objects OB2 and OB3 and other vehicles m1 located far from the vehicle M cannot be recognized from the camera image. In such circumstances, if the recognition result by the third recognition unit 136 is not used, the lane demarcated by camera lane markings CL1 and CL3 is judged to be the driving lane of the vehicle M, and if the vehicle M is under LKAS control, a driving control is executed that temporarily steers the vehicle towards the center of road RD1, which consists of lanes L1 and L2. Furthermore, when the vehicle M approaches objects OB2 and OB3 and they become recognizable from the camera image, a driving control is executed that steers the vehicle towards the center of lane L1 based on the physical boundary line estimated by objects OB2 and OB3. As a result, the behavior of the vehicle M in section S1 becomes unstable.

[0081] Therefore, in the embodiment, if the determination unit 142 determines that the lane markings do not match, the identification unit 144 identifies the lane of the vehicle M based on the radar targets OB1 to OB3 (or the physical boundary line estimated based on the targets) recognized by the third recognition unit 136. For example, in the case of the road shape shown in Figure 3, the third recognition unit 136 recognizes targets OB1 to OB3 and other vehicle m1, so the identification unit 144 estimates a physical boundary line based on the extension direction of target OB1, or estimates a physical boundary line extended from the position of target OB3 in a predetermined direction (for example, the extension direction of target OB1, or the extension direction of the travel trajectory K1 of other vehicle m1). The physical boundary line may be estimated in the front-rear direction from the position of target OB3 (in front of and behind target OB3 as seen from the vehicle M). Also, since target OB2 is not a nose target, it is not necessary to estimate a physical boundary line based on target OB2. The identification unit 144 then identifies the driving lane in section S1 using the estimated physical boundary line (for example, a physical boundary line based on target OB3 that is not included in the camera target). In this case, information from camera lane markings CL and map lane markings ML may be used in addition to the physical boundary line, and the estimated physical boundary line information may be used to correct (or complement) the camera lane markings CL and map lane markings ML.

[0082] As a result, even in sections where the camera lane markings CL and the map lane markings ML do not coincide, the lane L1 can be identified more accurately, and even when LKAS control is performed to drive the vehicle M along a target trajectory that travels in the center of the lane, the vehicle M's swaying in section S1 can be suppressed, and stable driving control can be maintained. In addition, the identification unit 144 may, regardless of the result of the matching determination by the judgment unit 142, identify the driving lane based on the radar target if a target not recognized by the first recognition unit 132 is recognized by the third recognition unit 136 (if it exists as a radar target), and execute driving control of the vehicle M according to the identified driving lane.

[0083] For example, in one embodiment, the driving control unit 146 may control the driving of the vehicle M in accordance with the driving lane (path of travel) identified based on the radar target recognized by the third recognition unit 136, if the radar target recognized by the third recognition unit 136 (for example, target OB3 not recognized by the first recognition unit 132) is recognized further inside the width direction of the road RD1 from the perspective of the vehicle M. As a result, when the radar target is closer, the radar target is more accurate in identifying the driving lane (path of travel), and by using this information to identify the driving lane, the vehicle M can be made to sway and stable driving control can be maintained.

[0084] Furthermore, in the embodiment, if the radar target recognized by the third recognition unit 136 is recognized inside the width direction of the road RD1 from the perspective of the vehicle M (i.e., within the road RD1) relative to the camera lane marking CL or camera target, and is located at a distance (a certain distance or greater) relative to the vehicle M, the driving control unit 146 may control the driving of the vehicle M according to the driving lane identified based on the radar target recognized by the third recognition unit 136. This makes it possible to more appropriately identify the driving lane using radar targets located at a distance, where the recognition accuracy of camera images decreases at close range.

[0085] In another embodiment, the identification unit 144 may identify the driving lane based on the position of the other vehicle m1 and the driving trajectory K1 of the other vehicle m1 when the radar target recognized by the third recognition unit 136 is another vehicle m1 traveling in front of the vehicle M. In this case, as described above, it is estimated that the other vehicle m1 is traveling in the center of the lane, and the physical boundary line is estimated at a predetermined distance to the left and right (lateral direction) from the driving trajectory K1, assuming that the driving trajectory K1 of the other vehicle m1 is in the center of the lane. The identification unit 144 then identifies the lane demarcated by the estimated left and right physical boundary lines as the driving lane. Even if the other vehicle m1 is not recognized from the camera image, the driving lane (road) can be accurately identified by the preceding vehicle detected by the radar device 12.

[0086] Furthermore, in this embodiment, the identification unit 144 may, when the target recognized by the third recognition unit 136 is a nose target (soft nose or hard nose), identify the lane the vehicle M is traveling in based on the position of the nose target, etc. In this case, as described above, since the target OB2 shown in Figure 3 is not a nose target, a physical boundary line based on the target OB2 is not estimated. As a result, even if the nose cannot be recognized from the camera image due to surrounding conditions such as bad weather, the lane can be identified more accurately according to changes in road structure such as junctions and construction sections by using the nose target included in the radar target.

[0087] Furthermore, in this embodiment, if the radar target recognized by the third recognition unit 136 is a tunnel side wall, the identification unit 144 may use the physical boundary line estimated by the left and right side walls inside the tunnel to identify the lane in which the vehicle M is traveling. This allows the vehicle M to more accurately identify its lane based on the positional information of the object (tunnel side wall) obtained from the detection results of the radar device 12, even if the tunnel side wall cannot be recognized from the camera image due to surrounding conditions, etc. Note that the recognition accuracy of the first recognition unit 132, which performs recognition using camera images, decreases near the tunnel entrance due to the influence of shadows and other factors caused by the tunnel, and the recognition accuracy of the third recognition unit 136, which performs recognition using the radar device 12, decreases inside the tunnel due to the influence of diffuse reflection and other factors caused by the walls. Therefore, for example, when the vehicle M is traveling near (including inside) a tunnel, the identification unit 144 may use appropriate recognition results according to these difficult scenes to identify the lane in which the vehicle M is traveling.

[0088] In addition, in this embodiment, the determination unit 142 may determine whether the recognition accuracy of the first recognition unit 132 has decreased, and if the determination unit 146 determines that the recognition accuracy of the first recognition unit 132 has decreased, it may control the driving of its own vehicle M according to the driving lane identified using the radar target recognized by the third recognition unit 136. In this case, for example, even if the determination unit 142 determines that the camera lane markings CL and the map lane markings ML match, if it determines that the recognition accuracy of the first recognition unit 132 has decreased, the driving lane will be identified based on the radar target. Furthermore, in this case, even if the radar target does not include any targets that have not been recognized by the first recognition unit 132, the driving lane may be identified based on the radar target.

[0089] In addition, in the embodiment, the identification unit 144 may normally prioritize the recognition result of the first recognition unit 132 to identify the driving lane, and if it is determined that the recognition accuracy of the first recognition unit 132 has decreased, it may prioritize the recognition result of the third recognition unit 136 to identify the driving lane. This allows for correction or supplementation of the recognition of the driving lane using radar targets when the recognition accuracy using camera images has decreased. Alternatively, if it is determined that the recognition accuracy of the third recognition unit 136 has decreased, the identification unit 144 may prioritize the recognition result of the first recognition unit 132 to identify the driving lane.

[0090] In another embodiment, the identification unit 144 may adjust the priority between the object recognition result by the first recognition unit 132 and the object recognition result by the third recognition unit 136 according to the distance between the object and the object in the direction of travel of the vehicle M, and identify the driving lane based on the recognition result with higher priority. This allows for appropriate switching of the priority of the recognition results, enabling more accurate identification of the driving lane.

[0091] In addition, in the embodiment, the driving control unit 146 may perform driving control based on lane markings if they exist within a predetermined distance from the radar target. For example, when identifying a driving lane using a radar target, the system generates a target trajectory so that the vehicle M travels along the direction of extension of the lane markings near the radar target (in the example in Figure 3, camera lane marking CL2 or map lane marking ML2). This allows for more accurate identification of lane markings based on the radar target.

[0092] Furthermore, in this embodiment, even if the determination unit 142 determines that the camera lane marking CL and the map lane marking ML coincide, the driving control unit 146 may perform driving control of its own vehicle M based on the lane markings that exist within a predetermined distance from the radar target. Even if the camera lane marking CL and the map lane marking ML coincide, by adopting the lane markings corresponding to the radar target, the driving lane can be accurately identified even if, for example, the recognition accuracy of the first recognition unit 132 is reduced due to the effects of bad weather, and a misdetermined coincidence occurs.

[0093] Furthermore, even if the camera lane markings CL recognized by the first recognition unit 132 and the camera target recognized by the first recognition unit 132 are within a predetermined distance of each other, the driving control unit 146 may perform driving control of its own vehicle M based on the lane markings if the lane markings exist within a predetermined distance from the radar target. For example, even if camera lane markings CL are recognized within a predetermined distance from the camera target recognized by the first recognition unit 132, by adopting the lane markings corresponding to the radar target, the camera target and camera lane markings CL will not be adopted when the recognition accuracy of the first recognition unit 132 is reduced, thereby enabling more accurate identification of the driving lane.

[0094] Furthermore, in the embodiment, if the determination unit 142 determines that it is a camera-unsuitable scene and also determines that it is a radar-unsuitable scene, the driving control unit 146 may terminate driving control such as LKAS control and switch to manual driving (or reduce the automation level) without specifying the driving lane. Furthermore, in the embodiment, if the determination unit 142 determines that the camera lane lines CL and the map lane lines ML do not match, and there are no radar targets that have not been recognized by the first recognition unit 132, the driving control unit 146 may terminate driving control such as LKAS control and switch to manual driving (or reduce the automation level).

[0095] [Processing flow] The following describes the processes performed by the automated driving control device 100 of this embodiment. The following description will focus primarily on the driving control processes based on the surrounding conditions of the vehicle M, among the processes performed by the automated driving control device 100. At the start of the flow, it is assumed that the vehicle M is undergoing a predetermined driving control (e.g., LKAS control). The processes described below may be repeatedly executed at predetermined timings or intervals (e.g., while the driving control by the automated driving control device 100 is being performed).

[0096] Figure 5 is a flowchart showing an example of the flow of the driving control process in the embodiment. In the example in Figure 5, the first recognition unit 132 recognizes the surrounding situation, including lane markings (camera lane markings CL) and objects, that exist in the vicinity of the vehicle M, including the direction of travel, based on the camera image (step S100).

[0097] Next, the second recognition unit 134 uses the position information of the vehicle M to refer to map information and recognizes the lane markings (map lane markings ML) present around the vehicle M based on the map information (step S110). In addition to the map lane markings, information on targets included in the map information may also be recognized in the process of step S110. Next, the third recognition unit 136 recognizes targets present around the vehicle M, including in the direction of travel, based on the detection results of the radar device 12 (and / or LIDAR 14) (step S120).

[0098] Next, the determination unit 142 determines whether the camera lane markings CL and the map lane markings ML match (step S130). If it determines that they match, the identification unit 144 identifies the driving lane based on at least one of the lane markings, the camera lane markings CL and the map lane markings ML (step S140). If it determines in step S130 that they do not match, the determination unit 142 determines whether there are any targets within the radar target that have not been recognized by the camera 10 (step S150). In step S140, for example, the camera target recognized by the first recognition unit 132 is compared with the radar target recognized by the second recognition unit 134 to determine whether there are any targets within the radar target that have not been recognized by the first recognition unit 132.

[0099] If the camera 10 determines that a target not recognized by the camera is present on the radar, the identification unit 144 identifies the vehicle M's lane based on the radar (step S160). After processing in step S140 or S160, the driving control unit 146 generates a target trajectory for the vehicle M so that it travels in the center of the identified lane, and drives the vehicle M along the generated target trajectory (step S170). Furthermore, if the camera 10 determines in step S150 that a target not recognized by the camera is present on the radar, the driving control unit 146 terminates the ongoing driving control and executes a control to switch to manual driving (step S180). This completes the processing of this flowchart.

[0100] It should be noted that the driving control processing of this embodiment is not limited to the processing shown in Figure 5. For example, if the identification unit 144 determines that there is a radar target that is not recognized by the camera 10, regardless of whether the camera lane line CL and the map lane line ML match, the identification unit 144 may identify the driving lane of the vehicle M based on the radar target. Also, in the processing of step S180, instead of simply switching to manual driving, control may be performed to lower the automation level. Furthermore, the determination unit 142 may determine camera-poor scenes and radar-poor scenes, and based on the determination result, the driving lane of the vehicle M may be identified using the target with the higher priority among the camera target and radar target.

[0101] [Differentiation] In the above-described embodiment, instead of determining whether the camera lane line CL and the map lane line ML coincide, it may be determined whether the camera lane line CL and the map lane line ML diverge. In addition to the above-described driving control, at least one of the steering and speed of the vehicle M may be controlled to avoid contact with an object recognized by the recognition unit 130. Furthermore, although the above-described embodiment mainly described driving control during the execution of LKAS control, it can also be applied to other driving controls such as ALC control.

[0102] According to the above-described embodiment, the automatic driving control device (an example of a mobile device control device) 100 includes: a first recognition unit 132 that recognizes first lane markings (camera lane markings) and targets (camera targets) that demarcate the driving lanes (travel paths) in the direction of travel of the vehicle (an example of a mobile device) M using images captured by a camera (an example of an imaging unit) 10; a second recognition unit 134 that recognizes map lane markings (second lane markings) that demarcate the driving lanes around the vehicle M from map information based on the position information of the vehicle M; and a third recognition unit 134 that recognizes targets (radar targets) in the direction of travel of the vehicle M using a radar device 12. The system comprises a recognition unit 136, a determination unit 142 that determines whether the camera lane markings and the map lane markings match, and a driving control unit (an example of a movement control unit) 146 that controls the movement of the vehicle M based on the determination result by the determination unit 142. The driving control unit 146 controls the movement of the vehicle M according to the driving lane (movement path) identified based on the target recognized by the third recognition unit 136 when a target not recognized by the first recognition unit 132 is recognized by the third recognition unit 136. This allows for more appropriate movement control according to the recognition conditions around the vehicle M, and ultimately contributes to the development of a sustainable transportation system.

[0103] Specifically, according to the embodiment, the route is selected based on targets (physical boundaries, preceding vehicles, etc.) recognized by the radar. By utilizing boundary lines detected by radar, the accuracy of route selection can be further improved. For example, since the radar device 12 can detect targets regardless of junctions, bad weather, gradients, etc., it can detect targets more accurately and stably than camera images, even at the beginning of a physical boundary (junction hard nose) or when preceding vehicles are far away.

[0104] Furthermore, according to the embodiment, even if the road cannot be accurately identified using only camera and map information, the road can be identified using radar targets that are not recognized by the camera, thereby suppressing the vehicle M's sway and maintaining stable driving control. In addition, according to the embodiment, if there is a radar target closer by, the radar target may be more accurate in identifying the road. Therefore, by using this information to identify the road, the vehicle M's sway in driving control such as LKAS control can be suppressed and more stable driving control can be maintained.

[0105] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Using the image captured by the imaging unit, the system recognizes the first demarcation line and targets that define the path of movement in the direction of the moving object's movement. Based on the location information of the moving object, a second demarcation line is recognized from the map information that demarcates the movement path around the moving object. Using a radar device, the moving object recognizes a target in the direction of travel of the moving object. Determine whether the first boundary line and the second boundary line coincide. Based on the determined result, the movement of the moving body is controlled. If a target that was not recognized by the recognition process using the aforementioned image is recognized by the recognition process using the radar device, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. Mobile device control system.

[0106] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0107] 1...Vehicle system, 10...Camera, 12...Radar device, 14...LIDAR, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 60...MPU, 80...Driver control unit, 100...Automatic driving control device, 120...First control unit, 130...Recognition unit, 132...First recognition unit, 134...Second recognition unit, 136...Third recognition unit, 140...Action plan generation unit, 142...Determination unit, 144...Specification unit, 146...Driving control unit, 160...Second control unit, 162...Target trajectory acquisition unit, 164...Speed ​​control unit, 166...Steering control unit, 180...HMI control unit, 190...Memory unit, 200...Driving force output device, 210...Brake device, 220...Steering device, M...Own vehicle, m1...Other vehicle

Claims

1. A first recognition unit recognizes first demarcation lines and targets that define the path of movement in the direction of travel of a moving object using images captured by the imaging unit, A second recognition unit recognizes a second demarcation line that demarcates the movement path around the moving object from map information based on the location information of the moving object, A third recognition unit that recognizes an object in the direction of travel of the moving object using a radar device, A determination unit that determines whether the first dividing line and the second dividing line coincide, The system includes a movement control unit that controls the movement of the moving body based on the determination result from the determination unit, The movement control unit controls the movement of the moving body according to the movement path specified based on the target recognized by the third recognition unit when a target not recognized by the first recognition unit is recognized by the third recognition unit. The movement control unit controls the movement of the moving body according to the movement path identified based on the target recognized by the third recognition unit, when the target recognized by the third recognition unit is recognized further inward in the width direction of the movement path from the perspective of the moving body. Mobile device control system.

2. The target recognized by the third recognition unit is a target that is located further away from the moving object than the first demarcation line or target recognized by the first recognition unit. The mobile device control device according to claim 1.

3. The target recognized by the third recognition unit includes a preceding moving body moving in front of the moving body. The mobile device control device according to claim 2.

4. The object recognized by the third recognition unit includes objects indicating the branching, merging, or end of a construction section of the travel path. The mobile device control device according to claim 2.

5. The object recognized by the third recognition unit includes the tunnel side wall, The mobile device control device according to claim 2.

6. The determination unit determines whether or not the recognition accuracy of the first recognition unit has decreased. If the movement control unit determines that the recognition accuracy has decreased, it controls the movement of the moving body according to the movement path identified using the target recognized by the third recognition unit. The mobile device control device according to claim 1.

7. A first recognition unit recognizes first demarcation lines and targets that define the path of movement in the direction of travel of a moving object using images captured by the imaging unit, A second recognition unit recognizes a second demarcation line that demarcates the movement path around the moving object from map information based on the location information of the moving object, A third recognition unit that recognizes an object in the direction of travel of the moving object using a radar device, A determination unit that determines whether the first dividing line and the second dividing line coincide, The system includes a movement control unit that controls the movement of the moving body based on the determination result from the determination unit, The movement control unit controls the movement of the moving body according to the movement path specified based on the target recognized by the third recognition unit when a target not recognized by the first recognition unit is recognized by the third recognition unit. The movement control unit controls the movement of the moving body based on a demarcation line that is within a predetermined distance from the target recognized by the third recognition unit. Mobile device control system.

8. Even if the determination unit determines that the first demarcation line and the second demarcation line coincide, the movement control unit controls the movement of the moving body based on the demarcation line if the demarcation line exists within a predetermined distance from the target recognized by the third recognition unit. The mobile device control device according to claim 7.

9. Even if the positions of the first demarcation line and the target recognized by the first recognition unit are within a predetermined distance, the movement control unit controls the movement of the moving body based on the demarcation line if the demarcation line exists within a predetermined distance from the target recognized by the third recognition unit. The mobile device control device according to claim 7.

10. A first recognition unit recognizes first demarcation lines and targets that define the path of movement in the direction of travel of a moving object using images captured by the imaging unit, A second recognition unit recognizes a second demarcation line that demarcates the movement path around the moving object from map information based on the location information of the moving object, A third recognition unit that recognizes an object in the direction of travel of the moving object using a radar device, A determination unit that determines whether the first dividing line and the second dividing line coincide, The system includes a movement control unit that controls the movement of the moving body based on the determination result from the determination unit, The movement control unit controls the movement of the moving body according to the movement path specified based on the target recognized by the third recognition unit when a target not recognized by the first recognition unit is recognized by the third recognition unit. The movement control unit adjusts the priority of the recognition result from the first recognition unit and the recognition result from the third recognition unit according to the distance between the moving body and a target located in the direction of the moving body's movement, and controls the movement of the moving body according to the movement path identified based on the recognition result with the higher priority. Mobile device control system.

11. Computers Using the image captured by the imaging unit, the system recognizes the first demarcation line and targets that define the path of movement in the direction of the moving object's movement. Based on the location information of the moving object, a second demarcation line is recognized from the map information that demarcates the movement path around the moving object. Using a radar device, the moving object recognizes a target in the direction of travel of the moving object. Determine whether the first dividing line and the second dividing line coincide. Based on the determined result, the movement of the moving body is controlled. If a target that was not recognized by the recognition process using the aforementioned image is recognized by the recognition process using the radar device, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. If the target recognized by the recognition process using the radar device is located further inside the width of the path from the perspective of the moving body than the target recognized by the recognition process using the image, the movement of the moving body is controlled according to the path identified based on the target recognized by the radar device. A method for controlling a mobile object.

12. Computers Using the image captured by the imaging unit, the system recognizes the first demarcation line and targets that define the path of movement in the direction of the moving object's movement. Based on the location information of the moving object, a second demarcation line is recognized from the map information that demarcates the movement path around the moving object. Using a radar device, the moving object recognizes a target in the direction of travel of the moving object. Determine whether the first dividing line and the second dividing line coincide. Based on the determined result, the movement of the moving body is controlled. If a target that was not recognized by the recognition process using the aforementioned image is recognized by the recognition process using the radar device, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. The movement of the moving object is controlled based on lane markings that exist within a predetermined distance from a target recognized using the radar device. A method for controlling a mobile object.

13. Computers Using the image captured by the imaging unit, the system recognizes the first demarcation line and targets that define the path of movement in the direction of the moving object's movement. Based on the location information of the moving object, a second demarcation line is recognized from the map information that demarcates the movement path around the moving object. Using a radar device, the moving object recognizes a target in the direction of travel of the moving object. Determine whether the first dividing line and the second dividing line coincide. Based on the determined result, the movement of the moving body is controlled. If a target that was not recognized by the recognition process using the aforementioned image is recognized by the recognition process using the radar device, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. The system adjusts the priority between the recognition results obtained using the image and the recognition results obtained using the radar device, depending on the distance between the moving body and a target in the direction of the moving body's movement, and controls the movement of the moving body according to the movement path identified based on the recognition result with the higher priority. A method for controlling a mobile object.

14. On the computer, Using the image captured by the imaging unit, the system recognizes the first demarcation line and target that define the path of the moving object in the direction of its movement. Based on the location information of the moving object, a second demarcation line is recognized from the map information to demarcate the movement path around the moving object. Using a radar device, the moving object is made to recognize a target in the direction of travel. Determine whether the first lane line and the second lane line coincide. Based on the determined result, the movement of the moving body is controlled. If a target that was not recognized by the recognition process using the aforementioned image is recognized by the recognition process using the radar device, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. If the target recognized by the recognition process using the radar device is located further inward in the width direction of the movement path from the perspective of the moving body than the target recognized by the recognition process using the image, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. program.

15. On the computer, Using the image captured by the imaging unit, the system recognizes the first demarcation line and target that define the path of the moving object in the direction of its movement. Based on the location information of the moving object, a second demarcation line is recognized from the map information to demarcate the movement path around the moving object. Using a radar device, the moving object is made to recognize a target in the direction of travel. Determine whether the first lane line and the second lane line coincide. Based on the determined result, the movement of the moving body is controlled. If a target that was not recognized by the recognition process using the aforementioned image is recognized by the recognition process using the radar device, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. The movement of the moving object is controlled based on lane markings that exist within a predetermined distance from a target recognized using the radar device. program.

16. On the computer, Using the image captured by the imaging unit, the system recognizes the first demarcation line and target that define the path of the moving object in the direction of its movement. Based on the location information of the moving object, a second demarcation line is recognized from the map information to demarcate the movement path around the moving object. Using a radar device, the moving object is made to recognize a target in the direction of travel. Determine whether the first lane line and the second lane line coincide. Based on the determined result, the movement of the moving body is controlled. If a target that was not recognized by the recognition process using the aforementioned image is recognized by the recognition process using the radar device, the movement of the moving body is controlled according to the movement path identified based on the target recognized by the radar device. Depending on the distance between the moving body and a target in the direction of the moving body's movement, the priority of the recognition results obtained by the image recognition process and the recognition results obtained by the radar device is adjusted, and the movement of the moving body is controlled according to the movement path identified based on the recognition result with the higher priority. program.

Images