Mobile body control device, mobile body control method, and storage medium

Abstract

Provided are a moving body control device, a moving body control method, and a storage medium capable of performing more appropriate movement control in accordance with the recognition status of the surroundings of a moving body. The moving body control device includes: a first recognition unit that recognizes, using an image captured by an imaging unit, a first division line that divides a moving path present in a traveling direction of a moving body, and an object target; a second recognition unit that recognizes, from the map information, a second division line that divides a movement path around the moving body on the basis of the position information of the moving body; a third recognition unit that recognizes an object target in the traveling direction of the moving body using a radar device; a determination unit that determines whether or not the first division line and the second division line coincide; and a movement control unit for controlling the movement of the moving body on the basis of the determination result by the determination unit, the movement control unit controlling the movement of the moving body on the basis of a movement path determined on the basis of the object target recognized by the third recognition unit when the object target not recognized by the first recognition unit is recognized by the third recognition unit.

Description

Technical Field

[0001] This invention relates to a mobile body control device, a mobile body control method, and a storage medium. Background Technology

[0002] In recent years, efforts to promote the use of sustainable transportation systems, which also take into account vulnerable groups among traffic participants, have become active. To achieve this, research and development efforts are focused on further improving the safety and convenience of transportation through research and development related to autonomous driving technology. In this context, techniques previously known for improving the reliability of road signs in the recognition of the surroundings of moving objects, where the road sign recognized from the image matches the road sign stored in the storage unit, and where the reliability is above a predetermined value, the road sign stored in the storage unit is determined to be the road sign corresponding to the current location (e.g., Japanese Patent Application Publication No. 2019-212188). Summary of the Invention

[0003] However, in previous autonomous driving technologies, the accuracy of image-based perimeter recognition varied depending on the surrounding conditions of the moving object, which sometimes made it impossible to accurately determine the path and execute appropriate motion control.

[0004] To address the aforementioned issues, one objective of this application is to provide a mobile body control device, mobile body control method, and storage medium capable of performing more appropriate motion control based on the identification of the mobile body's surroundings. Furthermore, this will contribute to the development of sustainable transport systems.

[0005] The mobile body control device, mobile body control method, and storage medium of the present invention adopt the following structure.

[0006] (1): A mobile body control device according to one aspect of the present invention includes: a first identification unit that uses an image captured by an imaging unit to identify a first dividing line and an object target that divides a movement path existing in the direction of travel of the mobile body; a second identification unit that identifies a second dividing line that divides a movement path around the mobile body from map information based on the position information of the mobile body; a third identification unit that uses a radar device to identify an object target in the direction of travel of the mobile body; a determination unit that determines whether the first dividing line and the second dividing line are consistent; and a movement control unit that controls the movement of the mobile body based on the determination result made by the determination unit, wherein if the movement control unit identifies an object target that was not identified by the first identification unit through the third identification unit, the movement control unit controls the movement of the mobile body according to a movement path determined based on the object target identified by the third identification unit.

[0007] (2): Based on the above (1) scheme, when the object target identified by the third identification unit is located inside the width direction of the movement path when viewed from the moving body, the movement control unit controls the movement of the moving body according to the movement path determined based on the object target identified by the third identification unit.

[0008] (3): Based on the above (2) scheme, the object target identified by the third identification unit is an object target that exists at a distance from the moving body when viewed from the first identification unit, compared with the first dividing line or object target identified by the first identification unit.

[0009] (4): Based on the above (3) scheme, the object target identified by the third identification unit includes a moving body moving in front of the moving body.

[0010] (5): Based on the above (3) scheme, the object targets identified by the third identification unit include object targets representing the ends of branches, merging or construction sections of the moving road (isolated end object targets).

[0011] (6): Based on the above (3) scheme, the object target identified by the third identification unit includes the tunnel sidewall.

[0012] (7): Based on the above (1) scheme, the determination unit determines whether the recognition accuracy of the first recognition unit has decreased. If the recognition accuracy is determined to be decreased, the movement control unit controls the movement of the moving body according to the movement path determined by the object target recognized by the third recognition unit.

[0013] (8): Based on the above (1) scheme, the movement control unit controls the movement of the moving body based on the dividing line that exists within a specified distance from the object target identified by the third identification unit.

[0014] (9): Based on the above (8) scheme, even if the determination unit determines that the first dividing line and the second dividing line are consistent, when there is a dividing line within a specified distance from the object target identified by the third identification unit, the movement control unit controls the movement of the moving body based on the dividing line.

[0015] (10): Based on the above (8) scheme, even when the first dividing line is within a predetermined distance from the position of the object target identified by the first identification unit, when there is a dividing line within a predetermined distance from the object target identified by the third identification unit, the movement control unit also controls the movement of the moving body based on the dividing line.

[0016] (11): Based on the above (1) scheme, the movement control unit adjusts the priority of the recognition results based on the first recognition unit and the recognition results based on the third recognition unit according to the distance between the object target existing in the direction of travel of the movement body and the movement body, and controls the movement of the movement body according to the movement path determined based on the recognition result with higher priority.

[0017] (12): Another aspect of the present invention provides a mobile body control method in which a computer performs the following processing: using an image captured by a camera to identify a first dividing line and an object target that divides a movement path existing in the direction of travel of the mobile body; based on the position information of the mobile body, identifying a second dividing line that divides a movement path around the mobile body from map information; using a radar device to identify an object target in the direction of travel of the mobile body; determining whether the first dividing line and the second dividing line are consistent; and controlling the movement of the mobile body based on the determination result, wherein if an object target that was not identified by the image recognition processing is identified by the radar device, the movement of the mobile body is controlled according to the movement path determined based on the object target identified by the radar device.

[0018] (13): In another aspect of the present invention, the storage medium stores a program that causes a computer to perform the following processing: using an image captured by a camera to identify a first dividing line and an object target that divides a movement path existing in the direction of travel of the moving body; based on the position information of the moving body, identifying a second dividing line that divides a movement path around the moving body from map information; using a radar device to identify an object target in the direction of travel of the moving body; determining whether the first dividing line and the second dividing line are consistent; and controlling the movement of the moving body based on the determination result, wherein if an object target that was not identified by the image recognition processing is identified by the radar device, the movement of the moving body is controlled according to the movement path determined based on the object target identified by the radar device.

[0019] According to the schemes (1) to (13) above, more appropriate motion control can be performed based on the identification status of the surroundings of the moving body. Attached Figure Description

[0020] Figure 1 This is a structural diagram of a vehicle system including a mobile body control device according to an embodiment.

[0021] Figure 2 This is a functional structure diagram of the first control unit and the second control unit.

[0022] Figure 3 This is a diagram showing an example of a road (moving road) on which the vehicle M travels.

[0023] Figure 4 This diagram is used to illustrate how driving lanes are determined based on recognition accuracy.

[0024] Figure 5 This is a flowchart illustrating an example of the driving control processing flow in the implementation method. Detailed Implementation

[0025] Hereinafter, embodiments of the mobile body control device, mobile body control method, and storage medium of the present invention will be described with reference to the accompanying drawings. Hereinafter, an embodiment of the mobile body control device applied to an autonomous vehicle will be described, using a vehicle as an example of a mobile body. Autonomous driving refers to automatically controlling one or both of the vehicle's steering and speed to perform driving control. The aforementioned driving control may include various driving controls such as LKAS (Lane Keeping Assistance System), ALC (Automated Lane Change), ACC (Adaptive Cruise Control System), TJP (Traffic Jam Pilot), and CMBS (Collision Mitigation Brake System). Autonomous vehicles can also be driven by manual operation (so-called manual driving) by the user of the vehicle (e.g., a passenger). In addition to vehicles, mobile bodies may also include, for example, ships capable of moving on land (roads) such as hovercraft, flying vehicles capable of traveling on roads, and standing vehicles with power units.

[0026] [Overall Structure]

[0027] Figure 1 This is a structural diagram of a vehicle system 1 including the mobile body control device of the embodiment. The vehicle equipped with vehicle system 1 (hereinafter referred to as the vehicle M) is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, a micro-mobile body, etc., and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or electricity discharged from a battery such as a secondary battery or a fuel cell.

[0028] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) system 14, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU (Map Positioning Unit) 60, driving controls 80, an automatic driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected through multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. Figure 1 The structure shown is merely an example; some parts of the structure may be omitted, and other structures may be added. Combining camera 10, radar device 12, and LIDAR 14 is an example of a "detection device DD". HMI 30 is an example of an "output device". Automatic driving control device 100 is an example of a "movement control device".

[0029] Camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). Camera 10 is mounted anywhere on the vehicle M equipped with vehicle system 1. When shooting forward, camera 10 is mounted on the upper part of the windshield, behind the interior rearview mirror, or at the front of the vehicle body. When shooting backward, camera 10 is mounted on the rear windshield, tailgate, etc. When shooting to the side, camera 10 is mounted on the side mirror on the door, etc. Camera 10 periodically and repeatedly photographs the perimeter of the vehicle M. Camera 10 can also be a stereo camera.

[0030] Radar device 12 radiates millimeter-wave or other radio waves (radar) to the periphery of the vehicle M, and detects radio waves (reflected waves) reflected from surrounding objects to at least detect the position (distance and orientation) of the objects. Radar device 12 can be installed at any location on the vehicle M. Radar device 12 can also detect the position and speed of objects using FM-CW (Frequency Modulated Continuous Wave) mode.

[0031] The LIDAR 14 illuminates the periphery of the vehicle M and measures the scattered light. The LIDAR 14 detects the distance to an object based on the time from the emission of light to the reception of light. The illuminated light is, for example, a pulsed laser. The LIDAR 14 is mounted at any location on the vehicle M.

[0032] The communication device 20 uses networks such as cellular networks, Wi-Fi networks, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet to communicate with other vehicles in the vicinity of the vehicle M, terminal devices of users of the vehicle M, or various server devices.

[0033] The HMI30 outputs various information to the occupants (including the driver) of the vehicle M and accepts input operations performed by the occupants. The HMI30 includes, for example, various display devices, speakers, buzzers, touch panels, switches, buttons, microphones, etc.

[0034] Vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting yaw rate (e.g., the rotational angular rate about a vertical axis passing through the center of gravity of the vehicle M), and an orientation sensor for detecting the orientation of the vehicle M. Vehicle sensor 40 may also include a position sensor for detecting the position of the vehicle M. The position sensor may be a sensor that obtains position information (longitude / latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that obtains position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50. Vehicle sensor 40 can also derive the speed of the vehicle M from the difference (i.e., distance) in position information over a specified time period from the position sensor. Weather sensors (e.g., humidity sensors, rain sensors) may also be included in vehicle sensor 40. The results detected by vehicle sensor 40 are output to the automatic driving control device 100.

[0035] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M can also be determined or supplemented using the INS (Inertial Navigation System) output from the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, buttons, etc. The GNSS receiver 51 may also be installed in the vehicle sensor 40. The navigation HMI 52 may also be partially or entirely shared with the aforementioned HMI 30. The route determination unit 53, for example, refers to the first map information 54 to determine the path (hereinafter referred to as the map path) from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52. The first map information 54, for example, represents information about the shape of a road by displaying road segments (an example of a moving road) and nodes connecting the road segments. The first map information 54 may also include POI (Point of Interest) information, etc. The path on the map is output to the MPU 60. The navigation device 50 may also provide route guidance using the navigation HMI 52 based on the path on the map. The navigation device 50 may also send its current location and destination to the navigation server via the communication device 20, and obtain a path equivalent to the path on the map from the navigation server. The navigation device 50 outputs the determined path on the map to the MPU 60.

[0036] MPU 60 includes, for example, a lane recommendation unit 61 that stores the second map information 62 in a storage device such as an HDD or flash memory. The lane recommendation unit 61 divides the path on the map provided by the navigation device 50 into multiple blocks (e.g., every 100 [m] in the vehicle's direction of travel) and determines a recommended lane for each block with reference to the second map information 62. The lane recommendation unit 61 determines which lane to drive in from the left. When the path on the map has branching points, the lane recommendation unit 61 determines the recommended lane in a manner that allows the vehicle M to travel on a reasonable path to the branch destination.

[0037] The second map information 62 is map information with higher precision compared to the first map information 54. The second map information 62 may include, for example, the number of lanes (moving lanes), the type and shape of road markings (hereinafter referred to as markings), information about the center of lanes, or information about road boundaries. The second map information 62 may also include information about whether the road boundary includes structures that vehicles cannot pass through (including those that cross or come into contact with). Physical boundaries refer to, for example, guardrails, curbs, median strips, fences, tunnel sidewalls, and nose objects (soft nose, hard nose). "Cannot pass through" may also include situations where there are low steps that allow passage only to the extent that normally non-existent vehicle vibrations. The second map information 62 may include road shape information, traffic restriction information, address information (address / postal code), facility information, parking information, telephone number information, etc. Road shape information refers to, for example, the curvature (or radius of curvature, the same below), width, road surface slope, branching points, merging points, etc. The second map information 62 can be updated at any time by communicating with an external device through the communication device 20. The first map information 54 and the second map information 62 can also be set as map information as a whole. The map information can also be stored in the storage unit 190.

[0038] The driving control unit 80 includes, for example, a steering wheel, an accelerator pedal, and a brake pedal. The driving control unit 80 may also include a gear shift lever, a custom-shaped steering wheel, a joystick, and other operating components. Each operating component of the driving control unit 80 is equipped with an operation detection unit that detects the amount of operation performed by the driver or whether operation has occurred. The operation detection unit detects, for example, the steering angle of the steering wheel, steering torque, and the amount of pressure applied to the accelerator and brake pedals. Furthermore, the operation detection unit outputs the detection results to one or both of the following: the automatic driving control unit 100, the driving force output device 200, the braking device 210, and the steering device 220.

[0039] The automatic driving control device 100 performs various driving controls belonging to automatic driving on the vehicle M. The automatic driving control device 100 includes, for example, a first control unit 120, a second control unit 160, an HMI control unit 180, and a storage unit 190. The first control unit 120, the second control unit 160, and the HMI control unit 180 are implemented by executing programs (software) through hardware processors such as CPUs (Central Processing Units). Some or all of these components can be implemented using hardware (including circuitry) such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), GPUs (Graphics Processing Units), and SOCs (System-on-Chip), or through the coordinated use of software and hardware. The aforementioned program can be pre-stored in the HDD, flash memory or other storage device (a storage device with a non-temporary storage medium) of the automatic driving control device 100, or it can be stored in a removable storage medium such as a DVD, CD-ROM or memory card, and installed in the storage device of the automatic driving control device 100 by assembling the storage medium (non-temporary storage medium) into the drive unit, card slot or the like.

[0040] The storage unit 190 can also be implemented using the various storage devices described above, or EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 190 may store, for example, various information and programs as described in the embodiments. Map information (e.g., first map information 54 and second map information 62) may also be stored in the storage unit 190.

[0041] Figure 2This is a functional structure diagram of the first control unit 120 and the second control unit 160. The first control unit 120, for example, includes a recognition unit 130 and an action plan generation unit 140. The first control unit 120, for example, implements AI (Artificial Intelligence) based functions and functions based on pre-given models in parallel. For example, the function of "recognizing intersections" can be achieved by simultaneously executing intersection recognition based on deep learning and other methods, and recognition based on pre-given conditions (the existence of signals capable of pattern matching, road signs, etc.), and then comprehensively evaluating both. This ensures the reliability of autonomous driving. The first control unit 120, for example, executes controls related to the autonomous driving of the vehicle M based on instructions from the MPU 60, HMI control unit 180, etc.

[0042] The identification unit 130 identifies the surrounding conditions of the vehicle M based on the detection results of the detection device DD (information input from the camera 10, radar device 12, and LIDAR 14). For example, the identification unit 130 performs sensor fusion processing on some or all of the detection results from the camera 10, radar device 12, and LIDAR 14 to identify the position (relative position), size, speed (relative speed), acceleration, and other states of objects (objects) existing in the vicinity (within a specified distance) of the vehicle M. Among the objects identified by the identification unit 130, in addition to the physical boundaries that divide the road (moving road) (e.g., the physical boundaries contained in map information), it may also include obstacles such as billboards temporarily placed on the road, other vehicles, pedestrians, bicycles, and other traffic participants. The position of the object is identified, for example, as the position on the absolute coordinates with the representative point of the vehicle M (center of gravity, drive shaft center, etc.) as the origin, for control purposes. The position of the object can be represented by the representative point such as the center of gravity or corner of the object, or by the area it represents. For example, when the target object is a moving body such as another vehicle, the so-called "state" of the target object can also include the acceleration, jerk, or "action state" of the moving body (e.g., whether other vehicles are changing lanes or about to change lanes).

[0043] The identification unit 130 can identify, for example, temporary stop lines, red lights, toll booths, other road phenomena, road signs, and markings depicted on the road (e.g., speed limits). The identification unit 130 may include, for example, a first identification unit 132, a second identification unit 134, and a third identification unit 136. Details regarding these functions will be described later.

[0044] The action plan generation unit 140 generates an action plan for the vehicle M to drive (move) via autonomous driving based on the recognition results of the recognition unit 130. For example, the action plan generation unit 140 generates a target trajectory (target travel path) for the vehicle M to drive automatically (without driver intervention) in the future, based on the recognition results identified by the recognition unit 130 or the surrounding road shape and other factors based on the current position of the vehicle M obtained from map information, in a manner that can cope with the surrounding conditions of the vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is represented by a track that arranges the locations (track points) that the vehicle M should reach in sequence. Track points are the locations that the vehicle M should reach at predetermined travel distances (e.g., a few meters) along the route. In contrast, target speeds and target accelerations are generated as part of the target trajectory at predetermined sampling times (e.g., a few tenths of a second). Track points can also be the locations that the vehicle M should reach at the sampling time at predetermined sampling times. In this case, the target velocity and target acceleration information are represented by the intervals of the orbital points.

[0045] The action plan generation unit 140 can set events for automatic driving when generating the target track. These events include, for example, a lane departure suppression event that causes the vehicle M to travel without leaving its lane; a constant speed driving event that causes the vehicle M to travel at a constant speed in the same lane; a following driving event that causes the vehicle M to follow another vehicle that is closest to it within a specified distance (e.g., within 100 m) in front of it; a lane change event that causes the vehicle M to change lanes from its current lane to an adjacent lane; a branching event that causes the vehicle M to branch off at a road junction to a lane on the destination side; a merging event that causes the vehicle M to merge onto the main road at a merging point; and a takeover event that terminates automatic driving and switches to manual driving. Events may also include, for example, an overtaking event that causes the vehicle M to temporarily change lanes to an adjacent lane, overtake a vehicle in the adjacent lane, and then change lanes back to its original lane; and an avoidance event that causes the vehicle M to brake and steer at least one of the actions to avoid an obstacle in front of it.

[0046] The action plan generation unit 140 can, for example, change a predetermined event to another event or set a new event for the current segment based on the surrounding conditions of the vehicle M as it travels. The action plan generation unit 140 can also change a predetermined event to another event or set a new event for the current segment based on the occupant's operation of the HMI 30. The action plan generation unit 140 generates a target track corresponding to the set events.

[0047] The action plan generation unit 140 includes, for example, a decision unit 142, a determination unit 144, and a driving control unit 146. The driving control unit 146 is an example of a "movement control unit". Details of these functions will be described later.

[0048] The second control unit 160 controls the driving force output device 200, the braking device 210 and the steering device 220 so that the vehicle M passes through the target track generated by the action plan generation unit 140 at a predetermined time.

[0049] The second control unit 160 includes, for example, a target track acquisition unit 162, a speed control unit 164, and a steering control unit 166. The target track acquisition unit 162 acquires information about the target track (track point) generated by the action plan generation unit 140 and stores it in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the braking device 210 based on the speed elements associated with the target track stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target track stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is achieved, for example, through a combination of feedforward control and feedback control. As an example, the steering control unit 166 combines feedforward control corresponding to the curvature of the road ahead of the vehicle M with feedback control based on deviation from the target track.

[0050] return Figure 1 The HMI control unit 180 notifies the occupants of prescribed information or receives information input from the HMI 30 via the HMI 30. Prescribed information includes, for example, information related to the state of the vehicle M, information related to driving control, and other information relevant to the movement of the vehicle M. Information related to the state of the vehicle M includes, for example, the vehicle M's speed, engine speed, and gear position. Information related to driving control includes, for example, whether driving control based on autonomous driving is being executed, information asking whether to start autonomous driving, information related to the status of driving control based on autonomous driving, information related to the level of automation, and information urging the driver to drive when switching from autonomous driving to manual driving. Prescribed information may include information related to the surrounding conditions identified by the detection device DD. Prescribed information may include content stored on storage media such as television programs or DVDs (e.g., movies) that is not relevant to the movement of the vehicle M. Prescribed information may include, for example, information related to the current location, destination, and remaining fuel level of the vehicle M during autonomous driving. The HMI control unit 180 can also output information received by the HMI 30 to the communication device 20, navigation device 50, first control unit 120, etc.

[0051] HMI control unit 180 can also output occupant inquiry information and processing results from first control unit 120 and second control unit 160 to HMI 30. HMI control unit 180 can also transmit various information output to HMI 30 to a terminal device used by the occupants of vehicle M via communication device 20.

[0052] The driving force output device 200 outputs driving force (torque) for vehicle movement to the drive wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above structure based on information input from the second control unit 160 or information input from the accelerator pedal of the driving operation unit 80.

[0053] The braking device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a brake ECU. The brake ECU controls the electric motor based on information input from the second control unit 160 or from the brake pedal of the driving control unit 80, and outputs braking torque corresponding to the braking operation to each wheel. The braking device 210 may have a backup mechanism for transmitting the hydraulic pressure generated by the operation of the brake pedal to the hydraulic cylinder via the master hydraulic cylinder. The braking device 210 is not limited to the structure described above; it may also be an electronically controlled hydraulic braking device that controls the actuator based on information input from the second control unit 160 and transmits hydraulic pressure from the master hydraulic cylinder to the hydraulic cylinder.

[0054] The steering system 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels based on information input from the second control unit 160 or from the steering wheel of the driving control unit 80.

[0055] [Identification Department and Action Plan Generation Department]

[0056] Next, the functions of the identification unit 130 (mainly the first identification unit 132, the second identification unit 134, and the third identification unit 136) and the action plan generation unit 140 (mainly the determination unit 142, the determination unit 144, and the driving control unit 146) will be explained in detail. The driving control of this vehicle M will also be explained below.

[0057] Figure 3 This diagram illustrates an example of a road (moving road) on which vehicle M travels. Figure 3In the example shown, dividing lines CL1 to CL3 identified by camera 10 and dividing lines ML1 to ML3 obtained from map information (e.g., second map information 62) based on the location information of the vehicle M are shown. Figure 3 The road RD1 shown has two lanes, L1 and L2, that allow travel in the same direction. In the map information, lane L1 is divided by dividing lines ML1 and ML2, and lane L2 is divided by dividing lines ML2 and ML3. Figure 3 In the example, dividing lines CL1 to CL3 are examples of "first dividing lines," and dividing lines ML1 to ML3 are examples of "second dividing lines." Hereinafter, dividing lines CL1 to CL3 will sometimes be referred to as "camera dividing lines CL1 to CL3," and dividing lines ML1 to ML3 will sometimes be referred to as "map dividing lines ML1 to ML3." When not distinguishing between camera dividing lines CL1 to CL3, they are sometimes simply referred to as "camera dividing lines CL," and when not distinguishing between map dividing lines ML1 to ML3, they are sometimes simply referred to as "map dividing lines ML."

[0058] exist Figure 3 In the example, considering the traversable direction (X-axis direction in the diagram) viewed from lane L1, an object target (e.g., a physical boundary such as a guardrail) OB1 exists to the left (far to the outside) of road RD1, along the extension direction of lane L2. Furthermore, in Figure 3 In the example, within the interval S1 between locations P1 and P2 on road RD1, lane L2 is under road construction (or is near the construction area). Therefore, the actual dividing line between lane L1 and lane L2 is not drawn. An object (e.g., a billboard) OB2 indicating construction or to alert drivers of surrounding vehicles to driving in lane L1 is placed on lane L2. Furthermore, in Figure 3 In the example, an object target (e.g., a soft isolation end) OB3 is set to represent the end of a branch (or the end of a construction section). Object targets OB2 and OB3 are located at... Figure 3 When observing the position of vehicle M, it is located inside the width (lateral) of road RD1, which is greater than object OB1. Furthermore, objects OB2 and OB3 are located from... Figure 3 The position of the vehicle M shown is located further away from the object OB1, and the object OB3 is located further away from the object OB2.

[0059] exist Figure 3 In the example, vehicle M is traveling in lane L1 at speed VM, and another vehicle m1 is traveling in lane L1 ahead of vehicle M at speed Vm1. Figure 3 The diagram shows the trajectory (movement trajectory) K1 of another vehicle m1. The other vehicle m1 is an example of a "moving body moving forward." Figure 3 In this example, vehicle M performs LKAS control to maintain vehicle M within its driving lane (in other words, to prevent vehicle M from veering off the driving lane) during automatic driving. In this case, the automatic driving control device 100 determines the driving lane of vehicle M based on identified lane markings, generates a target track to keep vehicle M centered within the determined driving lane, and performs driving control (motion control) including at least steering of vehicle M to keep vehicle M traveling along the generated target track. During driving control, feedforward control and feedback control are performed continuously based on the target track and the position of vehicle M to adjust the steering angle, speed, etc., of vehicle M.

[0060] [First Identification Unit 132]

[0061] The first recognition unit 132 uses images captured by the camera 10 to identify camera dividing lines (first dividing lines) CL that divide the surrounding lanes (moving roads), including the direction of travel of the vehicle M, and object targets (camera object targets). For example, the first recognition unit 132 performs known analytical processing on the images captured by the camera 10 (hereinafter, camera images) (e.g., edge extraction, feature extraction such as color, shape, and size, pattern matching processing, character recognition processing, etc.), and identifies the camera dividing lines CL that the vehicle M is traveling on and camera object targets near the road (within a predetermined distance from the road, including on road RD1) based on the image analysis results. In the case of identifying the camera dividing lines CL, the first recognition unit 132 extracts edge points with large brightness differences from adjacent pixels in the camera image, connects the edge points to identify the camera dividing lines CL in the image plane. The first recognition unit 132 converts the position of the camera dividing lines CL, based on the position of a representative point of the vehicle M, into a vehicle coordinate system (e.g., Figure 3 (XY plane coordinates).

[0062] exist Figure 3In this example, the first recognition unit 132 can identify camera dividing lines CL1 to CL3 and object targets OB1 to OB3 based on camera images. The first recognition unit 132 identifies the position and speed Vm1 of other vehicles m1, which are the vehicles ahead of the vehicle M. Other vehicles m1 are also an example of "object targets". The first recognition unit 132 can identify the curvature of lanes L1 and L2 and the road surface slope based on camera images, and can also identify the curvature change of camera dividing lines CL1 to CL3. The curvature change is, for example, the rate of change of curvature of camera dividing lines CL1 to CL3 at a point x [m] ahead when viewed from the vehicle M, as identified by the camera 10. The first recognition unit 132 can also identify the area where the vehicle M can drive (move) (or the area without physical boundaries or other obstacles) as free space (hereinafter referred to as camera free space) based on the identified camera dividing lines CL1 and camera object targets.

[0063] The first identification unit 132 considers factors such as the object being located at a distance (a distance or more from the vehicle M), weather conditions (e.g., severe weather such as thunderstorms or direct sunlight), date and time (nighttime, etc.), and the surrounding conditions of blind spots caused by congestion. Sometimes, the camera's dividing line CL and the camera's object recognition accuracy decrease.

[0064] [Second Identification Unit 134]

[0065] The second identification unit 134, based on the location information of the vehicle M obtained by the vehicle sensor 40 or the GNSS receiver 51, and referring to the map information (first map information 54, second map information 62), identifies the map dividing line (second dividing line) that divides the lanes (moving roads) that exist around the direction of travel of the vehicle M.

[0066] exist Figure 3 In this example, the second identification unit 134 can identify map dividing lines ML1-ML3 and object target OB1 based on map information. The second identification unit 134 can identify the curvature of lanes L1 and L2, the road surface slope, and the curvature change of map dividing lines ML1-ML3 from the map information. Object targets such as object target OB2, set up for road construction, are not reflected in the map information and therefore cannot be identified by the second identification unit 134. The second identification unit 134 cannot identify other vehicles m1. The second identification unit 134 can communicate with an external device via the communication device 20 to obtain information indicating that construction is underway near section S1 (within a predetermined distance from section S1, and also within section S1).

[0067] [Third Identification Department 136]

[0068] The third identification unit 136 identifies objects (radar targets) existing in the vicinity of the vehicle M's direction of travel based on detection results from a device other than the camera 10 in the detection device DD (e.g., radar device 12). For example, the third identification unit 136 identifies the position (range and azimuth) and speed of radar targets existing in the vehicle M's direction of travel based on the detection results from the radar device 12. The third identification unit 136 may also identify radar targets in the vicinity of the vehicle M's direction of travel based on the detection results from the LIDAR 14, instead of the radar device 12 (or based thereon).

[0069] exist Figure 3 In the example, the third identification unit 136 identifies objects OB1 to OB3 and other vehicles m1 existing around the vehicle M. The third identification unit 136 may also identify the area where the vehicle M can drive (or the area without physical boundaries or other obstacles) as free space (hereinafter referred to as radar free space) based on the identified radar objects.

[0070] When the third identification unit 136 is located in a region surrounding the vehicle M's travel area, such as in tunnels, under overpasses, or under bridges, and there are reflective objects (reflective areas) that reflect electromagnetic waves or light, the electromagnetic waves or light are diffusely reflected by the reflective objects (such as tunnel walls), resulting in excessive reflected light and thus reducing the identification accuracy of object targets OB1 to OB3. Therefore, the third identification unit 136, like the first identification unit 132, also experiences a decrease in identification accuracy due to surrounding conditions. However, the third identification unit 136 can perform identification less affected by diffuse reflection, as long as it is located near the sidewall of the tunnel entrance before entering the tunnel. Therefore, the third identification unit 136 can, for example, identify the corner (end) of the tunnel entrance instead of isolating end targets.

[0071] [Judgment Section 142]

[0072] The determination unit 142 determines whether the camera dividing lines CL1 to CL3 identified by the first recognition unit 132 are consistent with the map dividing lines ML1 to ML3 identified by the second recognition unit 134. For example, the determination unit 142 calculates the consistency between the dividing lines CL1 and ML1, which are the closest dividing lines on the right side when viewed from the vehicle M, the consistency between the dividing lines CL2 and ML2, which are the closest dividing lines on the left side when viewed from the vehicle M, and the consistency between the dividing lines CL3 and ML3 on the adjacent lane side. Then, if the calculated consistency is above a threshold, the determination unit 142 determines that the camera dividing lines CL1 and ML3 are consistent; if the consistency is below the threshold, it determines that they are inconsistent. The consistency determination can be performed repeatedly at predetermined times or periods.

[0073] For example, in the vehicle coordinate system plane (XY plane), the determination unit 142 uses the position of the representative point of the vehicle M as a reference to make the camera dividing lines CL1, CL2, and CL3 overlap, and also makes the map dividing lines ML1, ML2, and ML3 overlap. Furthermore, when determining the dividing lines of the comparison objects (dividing lines CL1 and ML1, dividing lines CL2 and ML2, dividing lines CL3 and ML3), the determination unit 142 determines that the dividing lines are consistent if the consistency of all dividing lines is above a threshold, and determines that they are inconsistent if at least one dividing line is below the threshold.

[0074] Here, the degree of consistency in the aforementioned consistency determination refers, for example, to the degree of offset in the road width direction (moving road width direction, lateral direction, Y-axis direction in the diagram) (consistency distance, deviation in the moving road width direction). It should be noted that, in Figure 3 In the example, the horizontal offsets D1 between dividing lines CL1 and ML1, D2 between dividing lines CL2 and ML2, and D3 between dividing lines CL3 and ML3 can be used to determine consistency. Alternatively, the average, maximum, or minimum values ​​of the offsets D1, D2, and D3 can be used to determine consistency.

[0075] The degree of consistency, for example, can be used to replace the lateral position offset mentioned above (or, based on this), and is determined by the magnitude of the angle between the two dividing lines of the compared objects (the degree of deviation corresponding to the angle of deviation). For example, the greater the degree of deviation, the smaller the degree of consistency. Figure 3 In the example, only the angle θ formed by dividing lines CL2 and ML2 is shown, but the angle formed by dividing lines CL1 and ML1, the angle formed by dividing lines CL3 and ML3 can also be used for consistency determination, and the average, maximum or minimum value of each angle θ can also be used for consistency determination.

[0076] Consistency can also be used to replace the degree of lateral positional offset mentioned above, the degree of deviation corresponding to the deviation angle formed by the dividing lines (or, based on this), and the degree (magnitude) of the difference in the curvature change of the dividing lines. The curvature change mainly occurs in lanes where... Figure 3 This is used in the case of a curved road as shown. For example, the determination unit 142 can use the difference in curvature change between dividing lines CL1 and ML1, the difference in curvature change between dividing lines CL2 and ML2, and the difference in curvature change between dividing lines CL3 and ML3 to perform consistency determination separately, or it can use the average value, maximum value, or minimum value of the respective differences to perform consistency determination.

[0077] The determination unit 142 determines whether the recognition accuracy based on the first recognition unit 132 and the recognition accuracy based on the third recognition unit 136 has decreased. For example, the determination unit 142 pre-determines whether the driving scenario (driving condition) of the vehicle M is a scenario where the recognition accuracy is easily reduced by either the first recognition unit 132 or the third recognition unit 136. If it is determined that the vehicle is driving in a scenario where it is not well-suited, the determination is that the recognition accuracy has decreased. For example, driving scenarios such as near tunnel entrances, branch ends, or ends of central medians where there are objects at the ends of barriers, road slopes, and bad weather are likely to affect camera images, and therefore become scenarios where the recognition is not well-suited based on the first recognition unit 132 (camera not well-suited scenarios). Driving scenarios such as inside tunnels, under bridges, or under overpasses where there are reflective objects affect the detection results using the radar device 12, and therefore become scenarios where the recognition is not well-suited based on the third recognition unit 136 (radar not well-suited scenarios). Therefore, when the driving scenario of the vehicle M is a scenario where it is not well-suited, the determination unit 142 determines that the recognition accuracy of the object recognition unit has decreased. The determination unit 142 can also determine whether the vehicle is driving in an unfamiliar scenario (whether the current driving scenario is an unfamiliar scenario).

[0078] [Determined Section 144]

[0079] The determining unit 144 determines the driving lane of the vehicle M based on the recognition results performed by the first recognition unit 132 to the third recognition unit 136 and the determination result performed by the determination unit 142. For example, if the determination unit 142 determines that the camera dividing line CL and the map dividing line ML are consistent, the determining unit 144 determines the lane divided by the camera dividing lines CL1 and CL2 or the lane divided by the map dividing lines ML1 and ML2 as the driving lane of the vehicle M.

[0080] The determination unit 144 can also estimate the physical boundary line based on the camera object target identified by the first identification unit 132 and the radar object target identified by the third identification unit 136, and determine the driving lane of the vehicle M based on the estimated physical boundary line, camera dividing line and map dividing line. The specific processing of the determination unit 142 and the determination unit 144 will be described later.

[0081] [Traffic Control Unit 146]

[0082] Based on the recognition results from the first recognition units 132 to the third recognition units 136 and the driving lane determined by the determination unit 144, the driving control unit 146 determines the driving control for the vehicle M and generates a target track based on the determined driving control. "Determining driving control" may include, for example, determining the content (type) of driving control and deciding whether to execute (suppress) driving control. "Executing driving control" may include, for example, switching and executing the content of driving control, or continuing driving control that is already being executed. Suppressing driving control may include not executing driving control, or reducing the automation level of driving control.

[0083] For example, when the driving control unit 146 performs LKAS control as driving control (automatic driving), based on the driving lane determined by the determination unit 144, it controls at least the steering of the vehicle M to drive the vehicle M so that the representative point of the vehicle M passes through the center of the driving lane. If the determination unit 142 determines that the camera dividing line CL and the map dividing line ML are inconsistent and the recognition accuracy of the first recognition unit 132 and the third recognition unit 136 is reduced, the driving control unit 146 terminates driving control of the vehicle M and switches to manual driving by the driver, or performs control that lowers the automation level of the automatic driving system. Automation levels may include, for example, a first level, a second level where the degree of automation of driving control is lower than the first level, and a third level where the degree of automation of driving control is lower than the second level. Automation levels may also include a fourth level where the degree of automation of driving control is lower than the third level. Automation levels can be levels determined by standardized information, regulations, etc., or they can be index values ​​set independently of these. Therefore, the types, contents, and number of automation levels are not limited to the examples below. Low automation in driving control refers to a low rate of automation and a heavy workload (high level of driver intervention) required from the driver. Specifically, it means the automatic driving control device 100 controls the steering or speed of the vehicle M to a low degree (high need for driver intervention in steering or speed control). Tasks assigned to the driver include, for example, monitoring the surroundings of the vehicle M and operating the driving controls. Operating the driving controls includes, for example, the driver holding the steering wheel (hereinafter, the hand-held position). Tasks assigned to the driver include those necessary for maintaining the automatic driving of the vehicle M (driver tasks). Therefore, when the driver is unable to perform the assigned tasks, the level of automation decreases.

[0084] At the first level, there are no tasks assigned to the driver (the tasks assigned to the driver are the lightest), so driving control is allowed, for example, in a state where the driver of vehicle M is not holding the steering wheel (hereinafter, non-hands-on state). At the second level, the tasks assigned to the driver are, for example, monitoring the surroundings (especially the front) of vehicle M. At the third level, the tasks assigned to the driver are, for example, monitoring the surroundings of vehicle M and being in a hands-on state. At the fourth level, the tasks assigned to the driver are, for example, monitoring the surroundings of vehicle M and being in a hands-on state, as well as controlling the steering and speed of vehicle M by the driving operation unit 80. That is, at the fourth level, the driver can be immediately switched driving positions, and the tasks assigned to the driver are the most demanding. The content of driving control and the tasks assigned to the driver at each level of automation are not limited to the examples above. The automatic driving control device 100 performs driving control at any of the first to fourth levels based on the surrounding conditions of vehicle M and the tasks being performed by the driver. The content of the driving control is output from HMI 30 by HMI control unit 180 and notified to the driver.

[0085] [Specific processing procedures performed by the decision-making unit 142 and the determination unit 144]

[0086] Next, the specific processing performed by the determination unit 142 and the determination unit 144 will be explained. Figure 4 This diagram illustrates how driving lanes are determined based on recognition accuracy. For example... Figure 4 As shown in the example, the determination unit 144 determines the driving lane of the vehicle M based on the determination result performed by the determination unit 142, the information of the camera dividing line CL and the map dividing line ML, and the estimation result of the physical boundary line. Here, the determination unit 142, based on various input information, not only performs consistency determination of the camera dividing line CL and the map dividing line ML, but also performs determination of unfavorable scenarios based on various information, and determines whether a specified object target (e.g., an isolated end object target) is identified as a recognition result. Unfavorable scenarios include camera unfavorable scenarios where the recognition accuracy of the first recognition unit 132 is reduced by using camera images, and radar unfavorable scenarios where the recognition accuracy of the third recognition unit 136 is reduced by using the detection result of the radar device 12.

[0087] [The camera is not good at scene recognition]

[0088] In cases where the camera is deemed unsuitable for a particular scene, the determination unit 142 obtains information about the surrounding conditions (driving scene) of the vehicle M, for example. If the obtained surrounding conditions meet predetermined conditions, the surrounding conditions are determined to be a scene unsuitable for the camera. These predetermined conditions include the presence of isolated end objects such as tunnel entrances, branch ends (or merging ends) in the direction of travel of the vehicle M (within a predetermined distance), the presence of road surface slopes exceeding a predetermined value, and specific weather conditions (e.g., thunderstorms, heavy rain, typhoons, snow). The presence or absence of tunnels, branches, merging ends, and road surface slopes can also be obtained from map information. Weather conditions can be obtained from weather sensors included in the vehicle sensor 40, or from an external device connected via the communication device 20 based on the location information of the vehicle M. In determining a scene unsuitable for the camera, in addition to (or instead of) information about the surrounding conditions, information about the camera's free space based on camera images captured by the camera 10 can also be used.

[0089] [Isolation End Determination]

[0090] The determination unit 142 determines whether there is an isolated end object target based on the radar free space and / or map information such as branches and construction areas obtained from external devices via the communication device 20, based on the detection results of the radar device 12. For example, since the hard isolation end of a branch is a road structure (with a three-dimensional shape), its presence is easier to determine than recognition based on camera images. Therefore, the determination unit 142 does not use the recognition results based on camera images from the first recognition unit 132, but uses the recognition results identified by the third recognition unit 136 based on the detection results of the radar device 12 to perform the above-mentioned isolation end determination. In the isolation end determination, instead of determining the presence or absence of an isolated end object target (or based on this), it may also determine whether a physical boundary (or boundary line) is identified near the branch, etc.

[0091] [Radar is not good at scene determination]

[0092] In cases where the radar is deemed ineffective, the determination unit 142, for example, obtains the road shape corresponding to the location of the vehicle M obtained from the map information based on the vehicle M's location information and referring to map information. Then, if the obtained road shape around the vehicle is a predetermined shape, it is determined to be a radar-ineffective scenario. The predetermined shape includes tunnels, under bridges, or under overpasses, etc. For example, under tunnels or overpasses, radar wave (radar) reflections are excessive due to reflective objects such as tops and side walls, making it possible to accurately identify the location of radar targets. Therefore, when driving on such a road shape, the current driving scenario of the vehicle M is determined to be a radar-ineffective scenario. The determination unit 142 can also determine that it is not a radar-ineffective scenario.

[0093] The determination unit 144, for example, estimates the physical boundary line (a dividing line corresponding to the extending direction of the physical boundary) of the vehicle M based on information from the camera free space and radar free space, the determination results of scenes where the camera is not proficient, the determination results of scenes where the radar is not proficient, and the determination results of the isolation end determination, etc. For example, the determination unit 144 combines the camera free space and radar free space and estimates the physical boundary line based on a consistent area. If the surrounding conditions (driving scene) of the vehicle M are not scenes where the camera is not proficient but scenes where the radar is not proficient, the determination unit 144 may also estimate the physical boundary line based on camera object targets (e.g., identified using camera images) from camera images. Figure 3 The physical boundary line is estimated based on the position of the object target OB1 shown. When the surrounding conditions of the vehicle M are a scenario where the camera is not well-suited for observation and the radar is not well-suited for observation, the determination unit 144 can also estimate the physical boundary line based on the radar object target (e.g., object targets OB1 to OB3) acquired by the radar device 12. When the surrounding conditions are neither a scenario where the camera is well-suited for observation nor a scenario where the radar is not well-suited for observation, the determination unit 144 can also prioritize the recognition results from the camera image to estimate the physical boundary line. Therefore, the recognition results from the camera image are generally prioritized, and when the recognition accuracy based on the camera image is reduced, the recognition of the driving lane can be supplemented based on the detection results of the radar device 12.

[0094] Here, as conditions for estimating the physical boundary line using the detection results of radar device 12, scenarios where the camera is not well-suited for certain scenes or results from isolated end determinations can be used. However, if the map information has not been updated (e.g., if it is old information older than a specified period), the recognition accuracy (e.g., the accuracy of branch information) based on the second recognition unit 134 may be poor. Therefore, in this embodiment, recognition based on the third recognition unit 136 can always be performed. If the position of the radar object target identified by the third recognition unit 136, the position of the forward vehicle (trajectory, etc.), deviates from the consistency determination result based on the camera dividing line CL and the map dividing line ML, the camera dividing line CL, or the camera object target identified by the camera image by a specified value, the detection results of radar device 12 can be used to estimate the physical boundary line.

[0095] [Processing of the determination unit 144 and the driving control unit 146]

[0096] The determination unit 144 determines the driving lane of the vehicle M based on information from the estimation results of the physical boundary lines, the camera dividing lines CL, and the map dividing lines ML (including consistency determination results). The driving control unit 146 generates a target track for the vehicle M based on the driving lane determined by the determination unit 144. At least some of the functions of the determination unit 144 may also be included in the driving control unit 146.

[0097] For example, in Figure 3 In the section S1 shown, due to the influence of construction areas, the dividing lines for lanes L1 and L2 are not drawn, so a portion of the camera dividing lines are not identified. Through consistency determination, it is determined that the camera dividing line CL is inconsistent with the map dividing line ML. In adverse weather conditions, it is impossible to identify distant objects OB2, OB3, and other vehicles m1 based on camera images. In such situations, without using the recognition result based on the third recognition unit 136, the lane divided by camera dividing lines CL1 and CL3 is determined as the driving lane of vehicle M. While vehicle M is under LKAS control, driving control is temporarily implemented to turn towards the center of the road RD1 formed by lanes L1 and L2. Furthermore, if vehicle M can be identified from the camera image by approaching objects OB2 and OB3, driving control is implemented to turn towards the center of lane L1 by using the physical boundary lines estimated by objects OB2 and OB3. Therefore, the behavior of vehicle M in section S1 will sway.

[0098] Therefore, in the embodiment, when the determination unit 144 determines that the dividing lines are inconsistent based on the consistency determination of the determination unit 142, it determines the driving lane of the vehicle M based on the radar object targets OB1 to OB3 identified by the third identification unit 136 (or the physical boundary line estimated based on the object targets). For example, in Figure 3 In the case of the road shape shown, the third identification unit 136 identifies object targets OB1 to OB3 and other vehicles m1. Therefore, the determination unit 144 estimates the physical boundary line based on the extension direction of object target OB1, or estimates the physical boundary line extending from the position of object target OB3 in a predetermined direction (e.g., the extension direction of object target OB1 or the extension direction of the travel trajectory K1 of other vehicles m1). The physical boundary line can also be estimated in the front-rear direction (closer to the front and depth side of object target OB3 when viewed from the vehicle M) based on the position of object target OB3. Object target OB2 is not an isolated end object target, so the physical boundary line based on object target OB2 may not be estimated. Then, the determination unit 144 uses the estimated physical boundary line (e.g., the physical boundary line based on object target OB3 which is not included in the camera object target) to determine the driving lane in section S1. In this case, in addition to the physical boundary line, information from the camera dividing line CL and the map dividing line ML can also be used, and the information from the estimation result of the physical boundary line can also be used for the correction (or supplementation) of the camera dividing line CL and the map dividing line ML.

[0099] Therefore, even in sections where the camera dividing line CL and the map dividing line ML are inconsistent, lane L1 can be determined more accurately. Even when LKAS control is executed to make the vehicle M travel along the target trajectory in the center of the lane, the swaying of the vehicle M in section S1 can be suppressed, and stable driving control can continue. The determination unit 144 can also determine the driving lane based on the radar object target when the third identification unit 136 identifies an object target that was not identified by the first identification unit 132 (in the case of a radar object target), regardless of the result of the consistency determination based on the determination unit 142, and perform driving control of the vehicle M based on the determined driving lane.

[0100] For example, in one implementation, if a radar object target identified by the third identification unit 136 (e.g., object target OB3 not identified by the first identification unit 132) is located closer to the inside of the road RD1 in the width direction when viewed from the vehicle M than a camera object target identified by the first identification unit 132 (e.g., object target OB1), the driving control unit 146 controls the driving of the vehicle M based on the driving lane (moving path) determined by the radar object target identified by the third identification unit 136. Thus, when a radar object target is present in a closer location, the accuracy in determining the driving lane (driving path) is good, and by using this information to determine the driving lane, stable driving control can be maintained while suppressing the swaying of the vehicle M.

[0101] Furthermore, in one embodiment, if a radar target identified by the third identification unit 136 is located inside the width direction of the road RD1 as observed from the vehicle M (i.e., within the road RD1) and at a distance (above a predetermined distance) as observed from the vehicle M, and is identified as being closer than the camera dividing line CL or the camera target, the driving control unit 146 controls the driving of the vehicle M based on the driving lane determined by the radar target identified by the third identification unit 136. Thus, at close range, a driving lane can be more appropriately determined using radar targets located at a distance where the recognition accuracy based on camera images is reduced.

[0102] In this embodiment, the determination unit 144 can also determine the driving lane based on the position of the other vehicle m1 and its driving trajectory K1 when the radar object target identified by the third identification unit 136 is another vehicle m1 traveling in front of the vehicle M. In this case, as described above, it is assumed that the other vehicle m1 is traveling in the center of the lane, and as the driving trajectory K1 of the other vehicle m1 is in the center of the lane, a physical boundary line is estimated at a position separating the vehicle m1 to the left and right (laterally). Then, the determination unit 144 determines the lane divided by the estimated left and right physical boundary lines as the driving lane. Even if the other vehicle m1 is not identified from the camera image, the driving lane (driving path) can be accurately determined by the vehicle in front detected by the radar device 12.

[0103] In this embodiment, the determining unit 144 may also determine the driving lane of the vehicle M based on the position of the isolated end object target (soft isolation end, hard isolation end) when the object target identified by the third identification unit 136 is an isolated end object target. In this case, as described above, Figure 3The object OB2 shown is not an isolated end object, therefore the physical boundary line based on object OB2 is not presumed. Thus, even if the isolated end cannot be identified from the camera image due to surrounding conditions such as bad weather, the driving lane corresponding to changes in road structure such as branches and construction sections can be more accurately determined by the isolated end object included in the radar object target.

[0104] In this embodiment, the determination unit 144 can also determine the driving lane of the vehicle M using physical boundary lines estimated from the left and right sidewalls inside the tunnel when the radar object target identified by the third identification unit 136 is a tunnel sidewall. Therefore, even when the tunnel sidewall cannot be identified from camera images due to surrounding conditions, the driving lane of the vehicle M can be determined more accurately based on the position information of the object (tunnel sidewall) obtained from the detection results of the radar device 12. In the first identification unit 132, which uses camera images for identification, the identification accuracy decreases near the tunnel entrance due to the influence of shadows and other factors caused by the tunnel. Similarly, in the third identification unit 136, which uses the radar device 12 for identification, the identification accuracy decreases inside the tunnel due to diffuse reflection caused by the walls. Therefore, the determination unit 144 can determine the driving lane of the vehicle M using appropriate identification results, for example, when the vehicle M is driving near (or inside) a tunnel, even in these less favorable scenarios.

[0105] In one implementation, the determination unit 142 may determine whether the recognition accuracy of the first recognition unit 132 has decreased, and if the driving control unit 146 determines that the recognition accuracy of the first recognition unit 132 has decreased, it may control the driving of the vehicle M based on the driving lane determined using the radar object target identified by the third recognition unit 136. In this case, for example, even if the determination unit 142 determines that the camera dividing line CL and the map dividing line ML are consistent, the driving lane may still be determined based on the radar object target when the recognition accuracy of the first recognition unit 132 is determined. Furthermore, in this case, the driving lane may still be determined based on the radar object target even if the radar object target does not include any object targets not identified by the first recognition unit 132.

[0106] In this implementation, the determining unit 144 can typically prioritize the identification result of the first identification unit 132 to determine the driving lane, and if it determines that the identification accuracy based on the first identification unit 132 has decreased, it can prioritize the identification result of the third identification unit 136 to determine the driving lane. Therefore, when the identification accuracy using camera images decreases, radar object targets can be used to correct or supplement the identification of the driving lane. The determining unit 144 can also prioritize the identification result of the first identification unit 132 to determine the driving lane if it determines that the identification accuracy based on the third identification unit 136 has decreased.

[0107] In this embodiment, the determining unit 144 can also adjust the priority of the recognition results of the object target based on the first recognition unit 132 and the object target based on the third recognition unit 136 based on the distance between the object target existing in the direction of travel of the vehicle M and the vehicle M, and determine the driving lane based on the recognition result with higher priority. Therefore, the priority of the recognition results can be switched appropriately, and the driving lane can be determined more accurately.

[0108] In one implementation, the driving control unit 146 can also perform driving control based on a dividing line when a dividing line exists within a predetermined distance from the radar target. For example, when using a radar target to determine the driving lane, the vehicle M can be directed to drive along a dividing line near the radar target (in... Figure 3 In the example, the target trajectory is generated by traveling in the direction of the extension of the camera dividing line CL2 or the map dividing line ML2. This allows for more precise determination of the dividing lines based on radar targets.

[0109] In this implementation, even if the determination unit 142 determines that the camera dividing line CL and the map dividing line ML are consistent, the driving control unit 146 can still perform driving control of the vehicle M based on the dividing line when the dividing line exists within a predetermined distance from the radar target. Even if the camera dividing line CL and the map dividing line ML are determined to be consistent, by using a dividing line corresponding to the radar target, for example, even if the first identification unit 132 mistakenly determines that they are consistent due to reduced identification accuracy caused by adverse weather conditions, the driving lane can be determined with high accuracy.

[0110] Even when the camera dividing line CL identified by the first identification unit 132 is within a predetermined distance from the camera target identified by the first identification unit 132, the driving control unit 146 can still perform driving control of the vehicle M based on the dividing line when the dividing line exists within a predetermined distance from the radar target. For example, even when the camera dividing line CL is identified within a predetermined distance from the camera target identified by the first identification unit 132, by using a dividing line corresponding to the radar target, the camera target and the camera dividing line CL will no longer be used when the identification accuracy of the first identification unit 132 is reduced, thus enabling high-precision determination of the driving lane.

[0111] In one embodiment, the driving control unit 146 may also terminate driving controls such as LKAS control and perform control to switch to manual driving (or reduce the level of automation) if the determination unit 142 determines that the scene is not well-suited for both the camera and the radar. In another embodiment, the driving control unit 146 may also terminate driving controls such as LKAS control and perform control to switch to manual driving (or reduce the level of automation) if the determination unit 142 determines that the camera dividing line CL and the map dividing line ML are inconsistent and that objects not identified by the first recognition unit 132 do not exist in the radar target area.

[0112] [Processing Flow]

[0113] The following describes the processes performed by the automated driving control device 100 according to the embodiment. The description will focus primarily on the driving control processes performed by the automated driving control device 100, which are based on the surrounding conditions of the vehicle M. It should be noted that at the start of the process, the vehicle M is performing a predetermined driving control (e.g., LKAS control). The processes described below can be repeatedly performed at predetermined times or periods (e.g., during the execution of driving control by the automated driving control device 100).

[0114] Figure 5 This is a flowchart illustrating an example of the driving control processing flow in an embodiment. Figure 5 In the example, the first recognition unit 132 identifies the surrounding conditions of the object target, including the dividing line (camera dividing line CL) existing in the direction of travel including the vehicle M, based on the camera image (step S100).

[0115] Next, the second identification unit 134 uses the location information of the vehicle M and refers to the map information to identify the dividing lines (map dividing lines ML) existing around the vehicle M based on the map information (step S110). In the processing of step S110, in addition to the map dividing lines, information such as object targets contained in the map information can also be identified. Next, the third identification unit 136 identifies object targets existing around the vehicle M, including the direction of travel, based on the detection results of the radar device 12 (and / or LIDAR 14) (step S120).

[0116] Next, the determination unit 142 determines whether the camera dividing line CL and the map dividing line ML are consistent (step S130). If they are consistent, the determination unit 144 determines the driving lane based on at least one of the dividing lines of the camera dividing line CL and the map dividing line ML (step S140). In the process of step S130, if they are inconsistent, the determination unit 142 determines whether there is an object target not detected by the camera 10 in the radar object target (step S150). In the process of step S140, for example, the camera object target detected by the first identification unit 132 is compared with the radar object target detected by the second identification unit 134 to determine whether there is an object target not detected by the first identification unit 132 within the radar object target.

[0117] If it is determined that an object not detected by camera 10 exists within the radar target area, the determination unit 144 determines the driving lane of the vehicle M based on the radar target area (step S160). After the processing of step S140 or S160, the driving control unit 146 generates a target track for the vehicle M in a manner that causes the vehicle M to drive in the center of the determined driving lane, and causes the vehicle M to drive along the generated target track (step S170). In the processing of step S150, if it is determined that an object not detected by camera 10 exists within the radar target area, the driving control unit 146 terminates the driving control in progress and executes the control to switch to manual driving (step S180). Thus, the processing of this flowchart ends.

[0118] The driving control processing of the implementation method is not limited to Figure 5The processing is as shown. For example, regardless of whether the camera dividing line CL and the map dividing line ML are consistent, if it is determined that an object not detected by the camera 10 exists in the radar object target, the determination unit 144 determines the driving lane of the vehicle M based on the radar object target. In the processing of step S180, the control of reducing the level of automation can be performed simply instead of switching to manual driving. Moreover, the determination unit 142 can determine the scene that the camera is not good at and the scene that the radar is not good at, and based on the determination result, use the object target with the higher priority between the camera object target and the radar object target to determine the driving lane of the vehicle M.

[0119] [Variation Example]

[0120] In the above embodiments, instead of determining whether the camera dividing line CL and the map dividing line ML are consistent, it is also possible to determine whether the camera dividing line CL and the map dividing line ML deviate. In addition to the driving control described above, to avoid contact with objects detected by the recognition unit 130, control of at least one of the vehicle M's steering and speed can also be performed. In the above embodiments, driving control during the execution of LKAS control has been mainly described, but it can also be applied to other driving controls such as ALC control.

[0121] According to the above-described embodiments, the automatic driving control device (an example of a mobile vehicle control device) 100 includes: a first identification unit 132, which uses images captured by a camera (an example of an imaging unit) 10 to identify a first dividing line (camera dividing line) dividing a driving lane (moving road) existing in the direction of travel of the vehicle (an example of a mobile vehicle) M and an object target (camera object target); a second identification unit 134, which identifies a map dividing line (second dividing line) dividing the driving lanes around the vehicle M from map information based on the location information of the vehicle M; and a third identification unit 136, which uses a radar device 12 to identify... The vehicle M is guided by an object target (radar object target) in its direction of travel; a determination unit 142 determines whether the camera dividing line matches the map dividing line; and a driving control unit (an example of a movement control unit) 146 controls the movement of the vehicle M based on the determination result made by the determination unit 142. When the third identification unit 136 identifies an object target not identified by the first identification unit 132, the driving control unit 146 controls the movement of the vehicle M based on the driving lane (moving path) determined by the object target identified by the third identification unit 136. This allows for more appropriate movement control based on the identification status of the vehicle M's surroundings. Furthermore, this contributes to the development of a sustainable transportation system.

[0122] Specifically, according to the implementation method, the driving path is selected based on the object target (physical boundary, preceding vehicle, etc.) identified by radar. This allows for effective utilization of the boundary line detected by radar, further improving the accuracy of driving path selection. For example, the radar device 12 can detect object targets unaffected by branches, inclement weather, slope, etc., thus enabling more accurate and stable detection of object targets than camera images, even at the beginning of a physical boundary (the hard-isolated end of a branch) or when the preceding vehicle is far away.

[0123] According to the implementation method, even when the driving route cannot be accurately determined solely by camera and map information, determining the driving route using radar targets that the camera cannot identify can suppress the shaking of the vehicle M and continue stable driving control. According to the implementation method, when radar targets are present nearby, the accuracy of the radar targets in determining the driving route is sometimes good; therefore, by using this information to determine the driving route, shaking of the vehicle M during driving controls such as LKAS control can be suppressed, and more stable driving control can continue.

[0124] The implementation methods described above can be performed as follows.

[0125] A mobile body control device comprising:

[0126] Storage medium, which stores computer-readable instructions; and

[0127] The processor, which is connected to the storage medium,

[0128] The processor performs the following processing by executing computer-readable instructions:

[0129] The images captured by the camera unit are used to identify the first dividing line and the object target that divides the path of movement in the direction of travel of the moving body;

[0130] Based on the location information of the moving body, a second dividing line is identified from the map information to divide the movement path around the moving body;

[0131] Using radar devices to identify objects in the direction of travel of the moving body;

[0132] Determine whether the first dividing line and the second dividing line are consistent; and

[0133] The movement of the moving body is controlled based on the determined result.

[0134] If an object target that was not identified by the image recognition process is identified using the radar device's recognition processing, the movement of the moving body is controlled according to a movement path determined based on the object target identified using the radar device.

[0135] The above describes specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A mobile body control device, wherein, The moving body control device includes: The first identification unit uses images captured by the imaging unit to identify a first dividing line and an object target that divides the movement path existing in the direction of travel of the moving body. The second identification unit identifies a second dividing line from map information to divide the movement path around the moving body based on the location information of the moving body. The third identification unit uses a radar device to identify the target object in the direction of travel of the moving body; The determination unit determines whether the first dividing line and the second dividing line are consistent; as well as The movement control unit controls the movement of the moving body based on the determination result made by the determination unit. When the movement control unit detects an object target that was not detected by the first identification unit through the third identification unit, it controls the movement of the moving body based on a movement path determined by the object target detected by the third identification unit.

2. The moving body control device according to claim 1, wherein, When an object target identified by the third identification unit is located inside the width direction of the movement path when viewed from the moving body, compared to an object target identified by the first identification unit, the movement control unit controls the movement of the moving body based on the movement path determined by the object target identified by the third identification unit.

3. The moving body control device according to claim 2, wherein, The object target identified by the third identification unit is an object target that exists at a distance from the moving body when viewed from the first identification unit, compared to the first dividing line or object target identified by the first identification unit.

4. The moving body control device according to claim 3, wherein, The object target identified by the third identification unit includes a moving body moving in front of the moving body.

5. The moving body control device according to claim 3, wherein, The object targets identified by the third identification unit include object targets representing the ends of branches, merging points, or construction sections of the moving road.

6. The moving body control device according to claim 3, wherein, The objects identified by the third identification unit include the tunnel sidewalls.

7. The moving body control device according to claim 1, wherein, The determination unit determines whether the recognition accuracy of the first recognition unit has decreased. If the recognition accuracy is determined to be reduced, the movement control unit controls the movement of the moving body based on the movement path determined by the object target recognized by the third recognition unit.

8. The moving body control device according to claim 1, wherein, The movement control unit controls the movement of the moving body based on a dividing line that exists within a predetermined distance from the object target identified by the third identification unit.

9. The moving body control device according to claim 8, wherein, Even if the determination unit determines that the first dividing line and the second dividing line are consistent, when there is a dividing line within a predetermined distance from the object target identified by the third identification unit, the movement control unit controls the movement of the moving body based on the dividing line.

10. The moving body control device according to claim 8, wherein, Even when the first dividing line is within a predetermined distance from the position of the object target identified by the first identification unit, when the dividing line exists within a predetermined distance from the object target identified by the third identification unit, the movement control unit controls the movement of the moving body based on the dividing line.

11. The moving body control device according to claim 1, wherein, The movement control unit adjusts the priority of the recognition results based on the first recognition unit and the recognition results based on the third recognition unit according to the distance between the object target existing in the direction of travel of the moving body and the moving body, and controls the movement of the moving body according to the movement path determined based on the recognition result with higher priority.

12. A method for controlling a moving body, wherein, The moving body control method causes the computer to perform the following processing: The images captured by the camera unit are used to identify the first dividing line and the object target that divides the path of movement in the direction of travel of the moving body; Based on the location information of the moving body, a second dividing line is identified from the map information to divide the movement path around the moving body; Using radar devices to identify objects in the direction of travel of the moving body; Determine whether the first dividing line and the second dividing line are consistent; as well as The movement of the moving body is controlled based on the determined result. If an object target that was not identified by the image recognition process is identified using the radar device's recognition processing, the movement of the moving body is controlled according to a movement path determined based on the object target identified using the radar device.

13. A storage medium, wherein, The storage medium stores a program that causes the computer to perform the following processes: The images captured by the camera unit are used to identify the first dividing line and the object target that divides the path of movement in the direction of travel of the moving body; Based on the location information of the moving body, a second dividing line is identified from the map information to divide the movement path around the moving body; Using radar devices to identify objects in the direction of travel of the moving body; Determine whether the first dividing line and the second dividing line are consistent; as well as The movement of the moving body is controlled based on the determined result. If an object target that was not identified by the image recognition process is identified using the radar device's recognition processing, the movement of the moving body is controlled according to a movement path determined based on the object target identified using the radar device.

Images