Mobile body control device, mobile body control method, and storage medium

Abstract

Provided are a mobile body control device, a mobile body control method, and a storage medium that can improve the recognition accuracy of a division line that divides a region through which a mobile body travels. The mobile body control device of an embodiment includes a recognition unit that recognizes a surrounding situation of a mobile body based on an output of an external sensor, and a division line recognition unit that recognizes a division line that divides a region through which the mobile body travels based on the surrounding situation recognized by the recognition unit, the division line recognition unit extracting a prescribed region from the surrounding situation and extracting an edge within the extracted prescribed region in a state in which it is determined that the recognition accuracy of the division line is reduced, and recognizing the division line based on the extracted result.

Description

TECHNICAL FIELD

[0001] The present application relates to a mobile body control device, a mobile body control method, and a storage medium. BACKGROUND

[0002] In recent years, research related to automatic driving that automatically controls the travel of a vehicle is progressing. In association with this, a technology is known in which, in a case where a state in which a demarcation line of a lane in which a vehicle is traveling is detected changes to a state in which the demarcation line is not detected, a virtual demarcation line is estimated based on a position of a demarcation line that has been detected in the past, and the travel of the host vehicle is controlled so that the host vehicle is in a prescribed position with respect to the estimated virtual demarcation line (for example, International Publication No. 2018 / 012179). SUMMARY

[0003] However, the position of the demarcation line that has been detected in the past does not necessarily continue to remain unchanged, and thus a demarcation line that is different from the actual one is sometimes recognized.

[0004] The present application is completed in consideration of such a situation, and provides a mobile body control device, a mobile body control method, and a storage medium that can further improve the recognition accuracy of a demarcation line that divides an area through which a mobile body passes.

[0005] The mobile body control device, the mobile body control method, and the storage medium of the present application adopt the following structure.

[0006] (1) A mobile body control device according to an aspect of the present application, wherein the mobile body control device includes: a recognition unit that recognizes a surrounding situation of a mobile body based on an output of an external sensor; and a demarcation line recognition unit that recognizes a demarcation line that divides an area through which the mobile body passes, based on the surrounding situation recognized by the recognition unit, the demarcation line recognition unit extracting a prescribed area from the surrounding situation and extracting an edge within the extracted prescribed area in a case where it is determined that a recognition accuracy of the demarcation line is reduced, and recognizing the demarcation line based on the extracted result.

[0007] (2) In the aspect described in (1) above, a storage control unit that causes a storage unit to store information related to the demarcation line before a state in which the demarcation line recognition unit determines that the recognition accuracy of the demarcation line is reduced, the demarcation line recognition unit extracting a demarcation line candidate based on the edge extracted from the prescribed area, and recognizing a demarcation line that divides an area through which the mobile body passes based on a degree of approximation between information related to the extracted demarcation line candidate and information related to the demarcation line stored in the storage unit.

[0008] (3) In the aspect (2) above, the information related to the division line includes at least one of a position, a direction, and a type of the division line.

[0009] (4) In the aspect (2) above, the storage section further stores map information, and the division line identification section extracts the prescribed area based on information related to a division line obtained from the map information based on the position information of the mobile body or information related to the division line stored in the storage section before the state in which the recognition accuracy of the division line is determined to be reduced, in a case where the state in which the recognition accuracy of the division line is determined to be reduced.

[0010] (5) In the aspect (1) above, the prescribed area is set to the left and right of a traveling direction of the mobile body.

[0011] (6) In the aspect (1) above, the division line identification section extracts an edge in the surrounding situation, and determines the state in which the recognition accuracy of the division line is reduced in a case where at least one of a length, a reliability, and a quality of the extracted edge is less than a threshold value.

[0012] (7) A mobile body control method according to an aspect of the present application, wherein the mobile body control method causes a computer to perform the following processing: identify a surrounding situation of a mobile body based on an output of an external sensor; identify a division line that divides an area through which the mobile body travels based on the identified surrounding situation; extract a prescribed area from the surrounding situation in a case where a state in which the recognition accuracy of the division line is determined to be reduced; extract an edge within the extracted prescribed area; and identify the division line based on the extracted result.

[0013] (8) A storage medium according to an aspect of the present application stores a program, wherein the program causes a computer to perform the following processing: identify a surrounding situation of a mobile body based on an output of an external sensor; identify a division line that divides an area through which the mobile body travels based on the identified surrounding situation; extract a prescribed area from the surrounding situation in a case where a state in which the recognition accuracy of the division line is determined to be reduced; extract an edge within the extracted prescribed area; and identify the division line based on the extracted result.

[0014] According to the aspects (1) to (8) above, it is possible to further improve the recognition accuracy of a division line that divides an area through which a mobile body travels. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a configuration diagram of a vehicle system of a vehicle control device using the embodiment.

[0016] Figure 2is a functional configuration diagram of the first control section and the second control section.

[0017] Figure 3 is a diagram for explaining an example of division line recognition in the embodiment.

[0018] Figure 4 is a diagram for explaining the content of division line information recognition.

[0019] Figure 5 is a diagram for explaining extraction of a prescribed region.

[0020] Figure 6 is a diagram for explaining an example of division line candidate extraction.

[0021] Figure 7 is a diagram for explaining division line recognition of a travel lane from a division line candidate.

[0022] Figure 8 is a flowchart showing an example of a flow of processing performed by the automatic driving control device of the embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, an embodiment of a mobile body control device, a mobile body control method, and a storage medium of the present application will be described with reference to the accompanying drawings. Hereinafter, a vehicle will be used as an example of a mobile body. The vehicle is, for example, a vehicle of two wheels, three wheels, four wheels, or the like, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator coupled to the internal combustion engine, or power discharged from a secondary battery or a fuel cell. Hereinafter, an embodiment in which the mobile body control device is applied to an automatic driving vehicle will be described as an example. Automatic driving refers to, for example, performing driving control by automatically controlling one or both of the steering and the acceleration / deceleration of the vehicle. The driving control of the vehicle can include, for example, various driving support such as ACC (Adaptive Cruise Control), ALC (Auto Lane Changing), LKAS (Lane Keeping Assistance System), and TJP (Traffic Jam Pilot). The automatic driving vehicle can also control a part or all of the driving by manual driving by an occupant (driver). The mobile body can include, in addition to or instead of the vehicle, for example, a ship, a flying body (for example, including a drone, an aircraft, or the like), or the like.

[0024] [Overall Structure]

[0025] Figure 1A configuration diagram of a vehicle system 1 that is a vehicle control device using an embodiment will be described. The vehicle system 1 has, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driving operation member 80, an autonomous driving control device 100, a travel drive force output device 200, a brake device 210, and a steering device 220. These devices and apparatuses are connected to each other through a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. Figure 1 The illustrated configuration is merely an example, and a part of the configuration can be omitted, or another configuration can be further added. The camera 10, the radar device 12, the LIDAR 14, and the object recognition device 16 are combined together as an example of an "environment sensor". The environment sensor ES can also include a sonar (not illustrated), for example. The HMI 30 is an example of an "output unit". The autonomous driving control device 100 is an example of a "mobile body control device".

[0026] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to an arbitrary portion of a vehicle (hereinafter referred to as a vehicle M) on which the vehicle system 1 is mounted. In a case where the front is imaged, the camera 10 is attached to an upper portion of a front windshield glass, a back surface of a door mirror, or the like. The camera 10 repeatedly images the periphery of the vehicle M periodically, for example. The camera 10 can also be a stereo camera.

[0027] The radar device 12 radiates an electric wave such as a millimeter wave to the periphery of the vehicle M, and detects a position (distance and direction) of an object by detecting an electric wave (reflected wave) reflected by the object. The radar device 12 is attached to an arbitrary portion of the vehicle M. The radar device 12 can also detect the position and speed of the object by an FM-CW (Frequency Modulated Continuous Wave) method.

[0028] The LIDAR 14 irradiates light (or electromagnetic waves close to light in wavelength) to the periphery of the vehicle M and measures scattered light. The LIDAR 14 detects the distance to an object based on the time from light emission to light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 is installed at any position of the vehicle M. In the case where a sonar is provided on the vehicle M, the sonar is provided, for example, in a bumper or the like provided at the front end portion and the rear end portion of the vehicle M. The sonar detects an object (for example, an obstacle) present within a prescribed distance from the position of installation.

[0029] The object recognition device 16 performs sensor fusion processing on the detection results detected by some or all of the structures (the camera 10, the radar device 12, the LIDAR 14, and the sonar) of the outside sensor ES, to recognize the position, the kind, the speed, and the like of an object. The object recognition device 16 outputs the recognition result to the automatic driving control device 100. The object recognition device 16 can output the detection result of the outside sensor ES directly to the automatic driving control device 100. In this case, the object recognition device 16 can be omitted from the vehicle system 1.

[0030] The communication device 20 communicates with other vehicles present in the periphery of the vehicle M, or communicates with various server devices via a wireless base station, using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), and the like.

[0031] The HMI 30 outputs various information to the occupant of the vehicle M, and accepts input operations by the occupant. The HMI 30 includes, for example, various display devices, a speaker, a buzzer, a touch panel, a switch, a button, a microphone, and the like. The various display devices are, for example, an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display device, or the like. The display device is, for example, provided near the front of the driver seat (the seat closest to the steering wheel) in the instrument panel, and is provided at a position that the occupant can visually recognize from the gap of the steering wheel or over the steering wheel. The display device can also be provided at the center of the instrument panel. The display device can also be a HUD (Head Up Display). The HUD causes the occupant seated in the driver seat to visually recognize a virtual image by projecting an image to a part of the front windshield glass in front of the driver seat. The display device displays an image generated by the HMI control section 170 described later. The HMI 30 can also include a driving switch that switches between automatic driving and manual driving by the occupant, or the like.

[0032] The vehicle sensors 40 include a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity about the vertical axis, a direction sensor that detects the orientation of the vehicle M, and the like. The vehicle sensors 40 can include a position sensor that acquires the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude, latitude information) from a GPS (Global Positioning System) device. The position sensor can also be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50.

[0033] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route decision section 53. The navigation device 50 holds first map information 54 in a storage device such as a HDD (Hard Disk Drive) or a flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M can also be determined or supplemented by an INS (Inertial Navigation System) that utilizes the outputs of the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, a button, and the like. The navigation HMI 52 can also be partially or wholly shared with the aforementioned HMI 30. The route decision section 53 determines, for example, a route (hereinafter referred to as an on-map route) from the position of the vehicle M determined by the GNSS receiver 51 (or an arbitrary position input) to a destination input by an occupant using the navigation HMI 52, with reference to the first map information 54. The first map information 54 is, for example, information that represents the shape of a road by lines that represent the routes of the road and nodes that are connected by the lines. The first map information 54 can also include the curvatures of the road, POI (Point Of Interest) information, and the like. The on-map route is output to the MPU 60. The navigation device 50 can also perform route guidance using the navigation HMI 52 based on the on-map route. The navigation device 50 can also be implemented by the functions of a terminal device such as a smartphone or a tablet terminal held by an occupant. The navigation device 50 can also transmit the current position and the destination to a navigation server via the communication device 20 and acquire a route equivalent to the on-map route from the navigation server.

[0034] The MPU 60 includes, for example, a recommended lane determination section 61 that holds second map information 62 in a storage device such as a HDD, a flash memory, or the like. The recommended lane determination section 61 divides the on-map route provided from the navigation device 50 into a plurality of blocks (for example, divided every 100 [m] in the vehicle travel direction) and determines the recommended lane for each block with reference to the second map information 62. The recommended lane determination section 61 makes a determination to travel on the nth lane from the left. For example, in the case where there is a branch point on the on-map route, the recommended lane determination section 61 determines the recommended lane so that the vehicle M can travel on a reasonable route for traveling to the branch destination.

[0035] The second map information 62 is map information that has higher precision than the first map information 54. The second map information 62 includes, for example, position, direction, type, and the like of a road division line (hereinafter, simply referred to as "division line") that divides one or more lanes included in a road, division line information based on the division line information, information of the center of a lane or information of the boundary of a lane, or the like. The second map information 62 can also include information related to guardrails, fences, and the like that are provided along the extension direction of a road or the like, a separation sign (bun), a curb, a median strip, a shoulder, a sidewalk, or the like. The second map information 62 can include road information (road category), a legal speed (limit speed, maximum speed, minimum speed), traffic restriction information, address information (address, postal code), facility information, telephone number information, or the like. The second map information 62 can be updated at any time by communication with other devices through the communication device 20.

[0036] The driving operation section 80 includes, for example, an accelerator pedal, a brake pedal, a shift lever, and other operation sections in addition to a steering wheel. A sensor that detects the operation amount or the presence or absence of operation is installed in the driving operation section 80, and the detection result is output to some or all of the automatic driving control device 100, or the travel driving force output device 200, the brake device 210, and the steering device 220. The operation section need not necessarily be ring-shaped, and can be in the form of a shaped steering wheel, a joystick, a button, or the like.

[0037] The autonomous driving control device 100, for example, is provided with a first control section 120, a second control section 160, an HMI control section 170, a storage control section 180, and a storage section 190. The first control section 120, the second control section 160, and the HMI control section 170 are each realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components can be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or the like, and can also be realized by a combination of software and hardware. The program can be stored in advance in a storage device (a storage device provided with a non-transitory storage medium) such as an HDD, a flash memory, or the like of the autonomous driving control device 100, or can be stored in a removable storage medium such as a DVD, a CD-ROM, or the like, and installed in the HDD, the flash memory, or the like of the autonomous driving control device 100 by mounting the storage medium (non-transitory storage medium) in a drive device. The action plan generation section 140 and the second control section 160 together are an example of a "driving control section". The HMI control section 170 is an example of an "output control section".

[0038] The storage section 190 can also be realized by various storage devices described above or an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory), or the like. The storage section 190 stores, for example, recognition division line information 192, programs, and other various information, or the like. Details of the recognition division line information 192 will be described later. The storage section 190 can store the map information (first map information 54, second map information 62) described above.

[0039] Figure 2Fig. 1 is a functional configuration diagram of the first control section 120 and the second control section 160. The first control section 120 includes, for example, a recognition section 130 and a behavior plan generation section 140. The first control section 120 implements, for example, an AI (Artificial Intelligence)-based function and a function based on a model given in advance in parallel. For example, the function of "recognizing an intersection" can be implemented by "performing recognition of an intersection based on deep learning or the like and recognition based on a condition given in advance (presence of a signal capable of pattern matching, road marking, or the like), and comprehensively evaluating both by scoring them". Thus, the reliability of automated driving is ensured.

[0040] The recognition section 130 recognizes spatial information indicating the situation of the periphery of the vehicle M, for example, based on information input from the outside sensor ES. For example, the recognition section 130 recognizes the position, speed, acceleration, and the like of an object (for example, another vehicle, another obstacle) present in the periphery of the vehicle M. The position of the object is recognized as a position on an absolute coordinate with a representative point (center of gravity, center of a drive shaft, or the like) of the vehicle M as an origin, for example, and is used for control. The position of the object can also be represented by a representative point such as the center of gravity or a corner of the object, or can be represented by a region. The "state" of the object can also include the acceleration, jerk, or "behavior state" (for example, whether or not a lane change is being performed or is about to be performed) of another vehicle in the case where the object is a moving body such as another vehicle.

[0041] The recognition section 130 recognizes a lane (travel lane) in which the vehicle M is traveling, for example, from the situation of the periphery of the vehicle M. The recognition of the lane is performed by a division line recognition section 132 included in the recognition section 130. Details of the function of the division line recognition section 132 will be described later. The recognition section 130 recognizes an adjacent lane adjacent to the travel lane. The adjacent lane is, for example, a lane in which travel in the same direction as the travel lane is possible. The recognition section 130 recognizes a stop line, an obstacle, a red light, a toll gate, a road sign, and other road phenomena.

[0042] The recognition section 130 recognizes the position and posture of the vehicle M with respect to the travel lane when recognizing the travel lane. The recognition section 130 can recognize, for example, the deviation of a reference point of the vehicle M from the center of the lane and the angle of the direction of travel of the vehicle M with respect to a line connecting the center of the lane as the relative position and posture of the vehicle M with respect to the travel lane. Instead, the recognition section 130 can recognize the position of the reference point of the vehicle M with respect to an arbitrary side end portion (division line or road boundary) of the travel lane as the relative position of the vehicle M with respect to the travel lane. Here, the reference point of the vehicle M can be the center of the vehicle M or the center of gravity. The reference point can also be an end portion (front end portion, rear end portion) of the vehicle M or a position at which one of the plurality of wheels included in the vehicle M is present.

[0043] The action plan generation section 140 generates a target track along which the vehicle M is to automatically (independently of the operation of the driver) travel in the future in a manner that the vehicle M travels on the recommended lane decided by the recommended lane decision section 61 in principle and that the vehicle M can cope with the surrounding situation of the vehicle M. The target track includes, for example, a speed element. For example, the target track is expressed by arranging points (track points) at which the vehicle M should arrive in order. The track points are points at which the vehicle M should arrive at every prescribed travel distance (for example, several [m]) in terms of distance along the track, and, in addition thereto, a target speed and a target acceleration at every prescribed sampling time (for example, several [sec]) are generated as a part of the target track. The track points can also be positions at which the vehicle M should arrive at every prescribed sampling time. In this case, information of the target speed and the target acceleration is expressed by the interval of the track points.

[0044] The action plan generation section 140 can set an event (function) of automatic driving when generating the target track. The events of automatic driving include a constant speed traveling event, a low speed following traveling event, a lane changing event, a branching event, a merging event, a takeover event, and the like. The action plan generation section 140 generates a target track corresponding to the event started.

[0045] The second control section 160 controls the travel driving force output device 200, the brake device 210, and the steering device 220 so that the vehicle M passes through the target track generated by the action plan generation section 140 at a predetermined time.

[0046] The second control section 160 includes, for example, an acquisition section 162, a speed control section 164, and a steering control section 166. The acquisition section 162 acquires information of the target track (track points) generated by the action plan generation section 140 and stores it in a memory (not shown). The speed control section 164 controls the travel driving force output device 200 or the brake device 210 on the basis of a speed element attached to the target track stored in the memory. The steering control section 166 controls the steering device 220 in accordance with the curvature of the target track stored in the memory. The processing of the speed control section 164 and the steering control section 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control section 166 performs a combination of feedforward control corresponding to the curvature of the road ahead of the vehicle M and feedback control based on deviation from the target track.

[0047] The HMI control section 170 notifies the driver of the vehicle M of prescribed information through the HMI 30. The prescribed information includes, for example, driving support information. The driving support information includes, for example, the speed of the vehicle M, the engine speed, the fuel remaining amount, the radiator water temperature, the travel distance, the state of the shift lever, the information on the division line, the lane, the other vehicle, and the like that the vehicle M should travel in, which are recognized by the object recognition device 16, the automatic driving control device 100, and the like, and the future target track. The driving support information can include information indicating the switching of the driving mode described later, the driving state (for example, the kind of automatic driving in execution of LKAS, ALC, and the like) based on the driving support, and the like. For example, the HMI control section 170 can generate an image containing the prescribed information described above, and cause the generated image to be displayed on the display device of the HMI 30, or can generate a sound indicating the prescribed information, and cause the generated sound to be output from the speaker of the HMI 30. The HMI control section 170 can also output the information accepted through the HMI 30 to the communication device 20, the navigation device 50, the first control section 120, and the like.

[0048] The travel drive force output device 200 outputs a travel drive force (torque) for the travel of the vehicle M to the drive wheels. The travel drive force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, and the like, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above-described structure in accordance with the information input from the second control section 160 or the information input from the driving operation member 80.

[0049] The brake device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a brake ECU. The brake ECU controls the electric motor in accordance with the information input from the second control section 160 or the information input from the driving operation member 80, so that a brake torque corresponding to the brake operation is output to each wheel. The brake device 210 can include, as a backup, a mechanism that transmits hydraulic pressure generated by the operation of the brake pedal included in the driving operation member 80 to the hydraulic cylinder via a master hydraulic cylinder. The brake device 210 is not limited to the structure described above, and can be an electronically controlled hydraulic brake device that controls an actuator in accordance with the information input from the second control section 160 and transmits the hydraulic pressure of the master hydraulic cylinder to the hydraulic cylinder.

[0050] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, causes a force to act on a rack-and-pinion mechanism to change the orientation of the steered wheels. The steering ECU drives the electric motor in accordance with the information input from the second control section 160 or the information input from the steering wheel 82 of the driving operation member 80, so that the orientation of the steered wheels is changed.

[0051] [Regarding Division Line Recognition]

[0052] The following describes the division line recognition in the embodiment. The following describes a scenario in which LKAS control based on automatic driving is being executed. The LKAS control refers to control in which the vehicle M, for example, recognizes a division line of a travel lane while controlling at least the steering of the vehicle M to travel near the center of the travel lane to assist the lane maintenance of the vehicle M.

[0053] Figure 3 is a diagram for illustrating an example of the division line recognition in the embodiment. Figure 3 In the example of, a road RD including two lanes L1, L2 that can travel in the same direction (X-axis direction in the drawing) and a vehicle M traveling on the lane L1 at a speed VM along the extension direction (X-axis direction) of the road RD is shown. The lane L1 is set as a passing area of the vehicle M divided by division lines LL and CL. The lane L2 is an area divided by division lines CL and RL and is an adjacent lane to the lane L1. Figure 3 In the example of, objects (road structures) OB1, OB2 such as guardrails are provided on both side ends of the road RD (outside the division lines LL and RL as viewed from the center of the road RD). The object OB1 is provided along the extension direction of the division line LL, and the object OB2 is provided along the extension direction of the division line RL. Figure 3 In the example of, a range (hereinafter referred to as a recognizable range) RA in which an object can be recognized by the external sensor ES is shown. Figure 3 The recognizable range RA shown in is shown as a region in the front direction of the vehicle M for ease of explanation, but can include both the side and rear directions of the vehicle M. The recognizable range RA differs in range, for example, depending on the performance of the external sensor ES and the like.

[0054] The division line recognition unit 132, for example, recognizes the division line of the lane L1 in which the vehicle M travels, based on spatial information indicating the surrounding situation within the recognizable range RA recognized by the external sensor ES. The recognition of the division line is repeatedly performed at a prescribed timing. The prescribed timing can be, for example, a prescribed period, or a timing based on the speed or travel distance of the vehicle M.

[0055] For example, the division line recognition section 132 extracts edges from the captured image of the recognizable range RA captured by the camera 10, and recognizes the position of the division line on the basis of the extraction result. The edge refers to, for example, a pixel (or a group of pixels) whose difference in pixel value between itself and the surrounding pixels is larger than a reference, that is, a pixel of a feature. The division line recognition section 132 extracts edges using an edge extraction filter such as a prescribed differential filter, a Prewitt filter, a Sobel filter, or the like, for example, with respect to the luminance value of each pixel within the image. The edge extraction filter described above is merely an example, and the division line recognition section 132 can extract edges on the basis of other filters or algorithms.

[0056] The division line recognition section 132 recognizes a line segment (for example, a straight line or a curved line) in which the length of the edge is equal to or greater than a first threshold value as a division line on the basis of the extraction result of the edge. The division line recognition section 132 can also join line segments of edges that are similar in position and direction. The "similar" refers to a difference (discrepancy) in position and direction of the edge being within a prescribed range. The "similar" can also refer to a degree of similarity being equal to or greater than a prescribed value.

[0057] It can also be that even if the line segment is equal to or greater than the first threshold value, the division line recognition section 132 does not recognize the line segment as a division line if the line segment is a curved line and the curvature thereof is equal to or greater than a prescribed value (the radius of curvature thereof is equal to or less than a prescribed value). Thus, it is possible to exclude line segments that are obviously not division lines, and to improve the recognition accuracy of the division line.

[0058] The division line recognition section 132 can also derive one or both of the reliability and the quality of the edge in addition to (or instead of) the length of the edge. For example, the division line recognition section 132 derives the reliability of the division line as the edge on the basis of the degree of continuity and the degree of dispersion of the extracted edge. For example, the more continuity there is in the line segment of the edge, or the less dispersion there is in the direction of extension of the edge, the more the division line recognition section 132 increases the reliability. The division line recognition section 132 can also compare the edges extracted from the left and right with the position of the vehicle M as a reference, and the more the degree of continuity and the degree of dispersion of the edges are similar, the more the division line recognition section 132 increases the reliability of the division line as each edge. The more the number of edges extracted, the more the division line recognition section 132 improves the quality of the division line obtained from the edges. The quality can also be, for example, an index value (quality value) in which the higher the quality, the larger the value.

[0059] The division line recognition unit 132 determines whether or not the state in which the recognition accuracy of the division line is reduced. For example, the recognition accuracy of the division line is reduced in a case where actual division lines are worn, stained, or the like, in a case where the recognition accuracy is reduced due to road surface reflection caused by external light near the exit of the tunnel, due to performance degradation of the external sensor ES caused by heavy rain or the like, or the like. The division line recognition unit 132 determines that the state in which the recognition accuracy of the division line is reduced, for example, in a case where the length of the line segment of the edge extracted by the edge extraction is less than a first threshold value.

[0060] The division line recognition unit 132 can also determine that the state in which the recognition accuracy of the division line is reduced, in a case where the reliability of the edge is derived, in a case where the reliability is less than a second threshold value, or in a case where the quality value is less than a third threshold value. That is, the division line recognition unit 132 can also determine that the state in which the recognition accuracy of the division line is reduced, for example, in a case where at least one of the length, the reliability, and the quality of the edge based on the result of the edge extraction process is less than the respective threshold values. Thus, it is possible to more accurately determine whether or not the state in which the recognition accuracy of the division line is reduced, using a plurality of conditions. The division line recognition unit 132 can also determine whether or not the division line is normally recognized, using the same or similar criteria as the above-described determination criteria, instead of determining whether or not the state in which the recognition accuracy of the division line is reduced.

[0061] In a case where it is determined by the division line recognition unit 132 that the state in which the recognition accuracy of the division line is not reduced, the storage control unit 180 saves the recognition result of the division line (information related to the division line) recognized by the division line recognition unit 132 in the recognized division line information 192. The information related to the division line includes, for example, information related to the state of the division line.

[0062] Figure 4is a diagram for explaining the content of the recognized division line information 192. The recognized division line information 192 is, for example, information in which a vehicle position and recognized division line state information are associated with each other. The vehicle position is position information of the vehicle M acquired from the vehicle sensor 40. The division line state information includes, for example, the position, direction, and kind of the recognized division line. The position is, for example, the position of the division line with the position of the recognized vehicle M as a reference. The direction is, for example, the direction of extension of the division line with the position of the vehicle M as a reference. The kind includes, for example, the kind of line (solid line, broken line), width, and color of the division line. The kind can also include, for example, the presence or absence of a road button, the presence or absence of a center divider, and the like. The storage control portion 180 saves the position, direction, and kind of the division line on both the left and right of the vehicle M in a case where the division lines on both the left and right are recognized. The storage control portion 180 can also save information related to the length of the edge, reliability, and quality in the recognized division line information 192. The storage control portion 180 saves, for example, information related to the division line for a short period (for example, several seconds to several minutes) before a state in which the recognition accuracy of the division line is determined to be reduced. Thus, the amount of data can be reduced compared to saving data for a long period.

[0063] The division line recognition portion 132 extracts a prescribed region from the surrounding situation recognized by the recognition portion 130 in a case where the recognition accuracy of the division line is determined to be reduced, and extracts an edge from the extracted prescribed region, and recognizes the division line of the lane LI in which the vehicle M travels on the basis of the extraction result. Figure 3 In the scene of FIG. 10, there is a wear Wl of the division line LL and a stain Dl on the division line CL within the recognizable range RA. Therefore, the division line recognition portion 132 determines that the recognition accuracy of the division lines LL and CL of the lane LI in which the vehicle M travels is reduced. In this case, the division line recognition portion 132 first extracts a prescribed region from the surrounding situation.

[0064] Figure 5 is a diagram for explaining the extraction of a prescribed region. Figure 5 In the example of FIG. 11, for ease of explanation, a case where the division line recognition portion 132 extracts a prescribed region from the surrounding situation is briefly shown. Figure 3The area of the lane LI in which the host vehicle M is mainly traveling in the road RD shown. The division line recognition section 132 extracts a division line existence area in which the existence possibility of the division line is high (the existence probability is a prescribed value or more) as an example of a prescribed area in the recognizable range RA recognized by the external sensor ES of the vehicle M. For example, the division line recognition section 132 extracts the division line existence area based on the position of the division line before the recognition accuracy of the division line becomes low. For example, the division line recognition section 132 refers to the vehicle position of the recognized division line information 192 stored in the storage section 190 based on the position information of the vehicle M acquired from the vehicle sensor 40, extracts the position and direction of the division line state information to which the correspondence relation is established with the vehicle position included in the prescribed distance range from the position information of the vehicle M, and extracts the division line existence area based on the extracted position and direction. For example, the division line recognition section 132 extracts the division line existence area based on the dispersion degree of the extracted position and direction. The division line recognition section 132 can also extract a division line existence area in which the existence possibility of the division line in the future is predicted based on the displacement of the extracted position and direction.

[0065] The division line recognition section 132 can also extract a road on which the vehicle M is traveling from the position information of the vehicle M based on the position information of the vehicle M and referring to the second map information 62, for example, instead of (or in addition to) the recognized division line information 192, and extract an area in which a division line that divides the travel lane exists (a division line existence area) from the division line information of the extracted road. The division line recognition section 132 can also extract a final division line existence area based on the division line existence area extracted from the recognized division line information 192 and the division line existence area extracted from the second map information 62.

[0066] The division line recognition section 132 sets the division line existence area with respect to the traveling direction (the front direction) of the vehicle M, for example, on the left and right of the vehicle M. The division line recognition section 132 can also set the size and shape of the division line existence area according to the position of other vehicles existing in the surroundings, the presence or absence, shape, and the like of the object OB such as a guardrail. Figure 5 In the example of FIG. 11, two division line existence areas LLA and CLA are extracted on the left and right from the vehicle M within the recognizable range RA of the external sensor ES. The division line recognition section 132 can set the division line existence areas LLA and CLA to be the same size and shape, or to be different sizes and shapes. For example, the division line recognition section 132 makes the shape and size different according to the difference in the curvature (or the curvature radius) of the division line on the right and left immediately before in the case where the vehicle M has been traveling on a curve up to that point. By this, it is possible to further set the optimum division line existence area.

[0067] The division line recognition unit 132 extracts an edge in the division line existence region LLA, CLA. In this case, the division line recognition unit 132 extracts an edge based on the above-described various edge extraction filters, other filters, or algorithms. The division line existence region LLA, CLA has a high possibility of existence of a division line compared to other regions of the recognizable range RA. Therefore, the division line recognition unit 132 extracts an edge using a filter or an algorithm that easily extracts an edge compared to the edge extraction processing by the division line recognition unit 132. Thus, an edge can be more reliably extracted within the division line existence region.

[0068] The division line recognition unit 132 extracts a line segment of an edge included in the division line existence region LLA, CLA, whose length is fourth threshold or more, as a division line candidate. The fourth threshold can be the first threshold, or can be smaller than the first threshold. By being smaller than the first threshold, more division line candidates can be extracted. The division line recognition unit 132 can also link line segments of edges that are similar in position and direction.

[0069] Figure 6 Fig. 10 is a diagram for explaining an example of extraction of a division line candidate. Figure 6 In the example of Fig. 10, three division line candidates C1 to C3 are extracted in the division line existence region LLA by edge extraction by the division line recognition unit 132, and one division line candidate C4 is extracted in the division line existence region CLA. The division line recognition unit 132 derives division line candidate information of each of the extracted division line candidates C1 to C4. In the division line candidate information, as information related to a division line candidate, information related to a state of the division line candidate is included. For example, in the division line candidate information, position information and an extension direction of each of the division line candidates C1 to C4 are included, with the position of the vehicle M as a reference. In the division line candidate information, information related to a kind of division line can also be included.

[0070] Next, the division line recognition unit 132 uses the position information of the vehicle M, refers to the vehicle positions of the recognition division line information 192, acquires division line state information for which a correspondence relation is established with the vehicle position closest to the position information (in other words, division line state information last recognized in a state in which recognition accuracy is not reduced in the division line recognition), and compares the acquired division line state information with the division line candidate information, to recognize a division line of a travel lane from the division line candidates.

[0071] Figure 7 Fig. 11 is a diagram for explaining recognition of a division line of a travel lane from division line candidates. Figure 7 In the example of Fig. 11, the division line candidates C1 to C4 in the division line existence regions LLA, CLA, and the division lines LLp, CLp last recognized in a state in which recognition accuracy is not reduced, which are acquired from the recognition division line information 192, are shown.Figure 7 In the example, the dividing lines LLp and CLp are placed in the dividing line area based on the position of vehicle M so that they can be easily compared with the dividing line candidates C1 to C4 in the dividing line area.

[0072] Figure 7 In the example, the lane divider identification unit 132 compares at least one of the position, direction, and type of the candidate lane dividers included in the candidate lane divider information with the corresponding data (at least one of the position, direction, and type) of the lane divider included in the lane divider status information, and identifies it as the lane divider based on the comparison result. Specifically, the lane divider identification unit 132 compares at least one of the position, direction, and type of the candidate lane dividers C1 to C3 with the lane divider LLp, and extracts the approximation of the candidate lane dividers C1 to C3 with the lane divider LLp. For example, the smaller the difference in position, the smaller the difference in direction, and the closer the line types, the more the lane divider identification unit 132 improves the approximation. The lane divider identification unit 132 compares the candidate lane divider C4 with the lane divider CLp, and similarly extracts the approximation of the candidate lane divider C4 with the lane divider LLp. Moreover, the lane divider identification unit 132 extracts the candidate lane divider C1 to C4 with the highest approximation as the lane divider L1. Figure 7 In the example, candidate lane dividers C1 and C2 are located at different positions from lane divider LLp, and candidate lane divider C4 differs from lane divider CLp in extension direction and line type. Therefore, the lane divider recognition unit 132 recognizes candidate lane divider C3 among candidate lane dividers C1 to C4 as a lane divider for the driving lane (lane L1). Based on these recognition processes, misidentification of lane dividers can be suppressed.

[0073] The dividing line recognition unit 132 may also not recognize a dividing line if the approximation is less than a predetermined value. The dividing line recognition unit 132 may also recognize dividing lines in the regions LLA and CLA where dividing lines exist.

[0074] The driving control unit (action plan generation unit 140, second control unit 160) performs LKAS control based on the dividing lines identified by the dividing line recognition unit 132.

[0075] Thus, in this implementation, even if the accuracy of identifying the dividing line decreases, the accuracy and reliability of the dividing line identification can still be improved by extracting the edges of regions where the dividing line is highly likely to exist and identifying the dividing line based on the extracted edges. By limiting the region for identifying the dividing line, processing resources can be reduced.

[0076] The HMI control section 170 can also cause the display device of the HMI 30 to display an image related to the division line recognized by the division line recognition section 132, for example, in a case where the HMI 30 is caused to output the running state of the vehicle M based on driving support or the like. In this case, the HMI control section 170 can also cause the division line recognized in a case where the recognition accuracy of the division line is reduced and a case where the recognition accuracy is not reduced to be displayed in different display modes (for example, change of color, flickering display, change of pattern, or the like), respectively. The HMI control section 170 can also cause information indicating the state where the recognition accuracy of the division line is reduced to be output from the HMI 30. Thereby, the state of the vehicle M can be more accurately notified to the occupant.

[0077] [Processing flow]

[0078] Figure 8 is a flowchart of an example of a flow of processing performed by the automatic driving control device 100 of the embodiment. Figure 8 In the example of, the division line recognition processing in the processing performed by the automatic driving control device 100 is mainly explained. Figure 8 The processing of, for example, can be repeatedly performed during execution of automatic driving control such as LKAS.

[0079] Figure 8 In the example of, the recognition section 130 recognizes the surrounding situation of the vehicle M based on the detection result of the outside sensor ES (step S100). Next, the division line recognition section 132 recognizes the division line of the travel lane of the vehicle M from the spatial information representing the surrounding situation of the vehicle M (step S102).

[0080] Next, the division line recognition section 132 determines whether it is the state where the recognition accuracy of the division line is reduced (step S104). In a case where it is determined that it is the state where the recognition accuracy of the division line is reduced, the division line recognition section 132 extracts a division line existence area where the possibility that the division line exists is high as an example of the prescribed area in the recognizable range RA recognized by the outside sensor ES (step S106). In the processing of step S106, the division line recognition section 132 can also extract the division line existence area based on the position of the division line before the state where the recognition accuracy of the division line is reduced, for example, and can refer to the high-precision map (second map information 62) to extract the division line existence area. The final division line existence area can be extracted based on the division line existence areas extracted from them, respectively.

[0081] Next, the camera 10 captures an area including the division line existence region, and the division line recognition unit 132 performs edge extraction in the division line existence region from the captured image (step S108). Next, the division line recognition unit 132 extracts division line candidates based on the edge extraction result (step S110), and recognizes the division line based on the degree of approximation between the extracted division line candidates and the recognition result of the division line obtained before the recognition accuracy of the division line decreased (step S112).

[0082] After the processing of step S112 or in the case where it is determined in the processing of step S104 that the recognition accuracy of the division line has not decreased, the driving control unit (the action plan generation unit 140, the second control unit 160) performs driving control such as LKAS based on the recognized division line (step S114). Thereby, the processing of the present flowchart ends.

[0083] [Modified Example]

[0084] In the above-described embodiment, the division line recognition unit 132 can also perform the above-described recognition of the division line when performing driving control other than LKAS. For example, in the case where ALC control is performed, the division line recognition unit 132 can recognize not only the division line that divides the travel lane but also the division line that divides the adjacent lane to the travel lane. The division line recognition unit 132 can also extract the division line existence region with respect to the travel lane based on, for example, the position, direction, and the like of the division line of the lane (lane L2) other than the travel lane (lane L1) of the vehicle M as shown in FIG. 12, in addition to or instead of the above-described method. Further, the division line recognition unit 132 can also extract the division line existence region based on the position and direction of the object OB1, OB2 such as a guardrail provided on the road RD including the travel lane L1. Figure 3

[0085] In the above-described example, the edge included in the image captured by the camera 10 is extracted, and the division line is recognized based on the extraction result, but the edge extraction can also be performed based on the detection result (LIDAR data) of the LIDAR 14 included in the outside sensor ES in addition to or instead of this. In the case where there is an object such as a guardrail, a barrier, a curbstone, a median strip, or the like having a concave-convex shape, the division line can also be recognized based on the detection result of the radar device 12 or the sonar, and the division line existence region can be extracted. Thereby, the division line can be recognized with higher accuracy.

[0086] ​The division line recognition unit 132 can also recognize the travel lane by comparing the pattern of the division line (for example, the arrangement of solid lines and dashed lines) obtained from the second map information 62 with the pattern of the division line of the periphery of the vehicle M recognized from the image captured by the camera 10. The division line recognition unit 132 is not limited to recognizing the division line, and can also recognize the travel lane by recognizing the division line, the travel road boundary (road boundary) including the shoulder, curb, median, guardrail, and the like. In this recognition, the position of the vehicle M obtained from the navigation device 50, the processing result based on the INS can also be added.

[0087] In a case where the recognition accuracy of the division line is reduced and the division line cannot be recognized by the division line recognition unit 132, the HMI control unit 170 can cause the HMI 30 to output information indicating that the division line cannot be recognized, end the LKAS control, and cause the HMI 30 to output information prompting the occupant of the vehicle M to perform manual driving.

[0088] According to the above-described embodiment, in the automatic driving control device 100 (an example of a mobile body control device), there are provided: a recognition unit 130 that recognizes a periphery situation of a vehicle M (an example of a mobile body) based on an output of an external sensor ES; and a division line recognition unit 132 that recognizes a division line that divides a region through which the vehicle M travels, based on the periphery situation recognized by the recognition unit 130, the division line recognition unit 132 extracts a prescribed region from the periphery situation and extracts an edge within the extracted prescribed region in a case where it is determined that the recognition accuracy of the division line is reduced, and recognizes the division line based on the extracted result, whereby the recognition accuracy of the division line that divides the region through which the vehicle M travels can be further improved.

[0089] Specifically, according to the embodiment, for example, a prescribed region is extracted from a captured image of a camera only in the vicinity of a division line recognition result of the last time, in the vicinity of a boundary of a segmentation, and in the vicinity of a position where a division line is obtained by high-precision map information, and only an edge extraction is performed on the extracted region, whereby the division line can be recognized more efficiently and with higher accuracy. According to the embodiment, the information of the division line of the learning library recognized in a state where the recognition accuracy is not reduced is collated with the information of the division line recognized by the edge extraction, whereby the accuracy of the division line can be improved. According to the embodiment, regarding the state information of the division line candidate that can be obtained by the edge extraction, a division line that is similar to the division line used in the last control is selected, whereby the improvement of the accuracy and reliability of the recognition of the travel lane can be achieved, and the false detection of the division line can be suppressed.

[0090] The above-described embodiment can be expressed as follows.

[0091] A mobile body control device can be configured to have:

[0092] a storage device storing a program; and

[0093] a hardware processor,

[0094] the program stored in the storage device is executed by the hardware processor to perform the following processing:

[0095] identifying a surrounding situation of a mobile body based on an output of an external sensor;

[0096] identifying a division line that divides an area through which the mobile body travels, based on the identified surrounding situation;

[0097] in a case where it is determined that the identification accuracy of the division line is reduced, extracting a prescribed area from the surrounding situation;

[0098] extracting an edge within the extracted prescribed area;

[0099] identifying the division line based on the extracted result.

[0100] The above describes a specific embodiment of the present application using the embodiment, but the present application is not at all limited by such an embodiment, and various modifications and substitutions can be applied within a range not departing from the gist of the present application.

Claims

1. A mobile body control device, wherein, The moving body control device includes: The identification unit identifies the surrounding environment of a moving object based on the output of external sensors; and The dividing line recognition unit identifies dividing lines that divide the area through which the moving body can travel, based on the surrounding conditions identified by the recognition unit. The dividing line recognition unit extracts edges from the surrounding environment. If at least one of the length, reliability, and quality of the extracted edge is less than a threshold, it determines that the recognition accuracy of the dividing line has decreased. If the recognition accuracy of the dividing line has decreased, a specified region is extracted from the surrounding environment, and the edges within the extracted specified region are extracted. The dividing line is then recognized based on the extraction results. The moving body control device further includes a storage control unit, which enables the storage unit to store information related to the dividing line before the dividing line recognition unit determines that the recognition accuracy of the dividing line has decreased, and saves this information in the dividing line recognition information. When the dividing line recognition unit determines that the recognition accuracy of the dividing line has decreased, it extracts the position and direction of the dividing line status information that corresponds to the position of the moving body within a specified distance range from the position information of the moving body, based on the position information of the moving body and referring to the position of the moving body containing the recognized dividing line information stored in the storage unit. Based on the extracted position and direction, it extracts the area where the probability of the dividing line is estimated to be above a specified value as the specified area.

2. The moving body control device according to claim 1, wherein, The dividing line recognition unit extracts dividing line candidates based on the edges extracted from the specified area, and identifies the dividing line that divides the area through which the moving body travels based on the similarity between the information related to the extracted dividing line candidates and the information related to the dividing line stored in the storage unit.

3. The moving body control device according to claim 2, wherein, Information related to the dividing line includes at least one of the following: the position, direction, and type of the dividing line.

4. The moving body control device according to claim 2, wherein, The storage unit also stores map information. When the dividing line recognition unit determines that the recognition accuracy of the dividing line has decreased, it also extracts the specified area based on the dividing line-related information obtained from the map information based on the position information of the moving body.

5. The moving body control device according to claim 1, wherein, The designated area is set to the left and right of the moving body's direction of travel.

6. A method for controlling a moving body, wherein, The moving body control method causes the computer to perform the following processing: Identify the surrounding environment of a moving object based on the output of external sensors; Based on the identified surrounding conditions, the dividing lines for the area through which the moving body can pass are identified; If the edges in the surrounding conditions are extracted, and at least one of the length, reliability, and quality of the extracted edges is less than a threshold, it is determined that the recognition accuracy of the dividing line has decreased. If the recognition accuracy of the dividing line is determined to be decreased, a specified area is extracted from the surrounding conditions. The edges within the defined region are extracted; The dividing line is identified based on the extracted results; The storage unit stores information related to the dividing line prior to the state in which the recognition accuracy of the dividing line is determined to have decreased, and saves this information in the dividing line recognition information; If it is determined that the recognition accuracy of the dividing line has decreased, based on the position information of the moving body and referring to the position of the moving body with respect to the recognition dividing line information stored in the storage unit, the position and direction of the dividing line status information that corresponds to the position of the moving body with the recognition dividing line information within a specified distance range from the position information of the moving body are extracted, and the area where the probability of the dividing line exists is estimated to be above a specified value is extracted as the specified area based on the extracted position and direction.

7. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: Identify the surrounding environment of a moving object based on the output of external sensors; Based on the identified surrounding conditions, the dividing lines for the area through which the moving body can pass are identified; If the edges in the surrounding conditions are extracted, and at least one of the length, reliability, and quality of the extracted edges is less than a threshold, it is determined that the recognition accuracy of the dividing line has decreased. If the recognition accuracy of the dividing line is determined to be decreased, a specified area is extracted from the surrounding conditions. The edges within the defined region are extracted; The dividing line is identified based on the extracted results; The storage unit stores information related to the dividing line prior to the state in which the recognition accuracy of the dividing line is determined to have decreased, and saves this information in the dividing line recognition information; If it is determined that the recognition accuracy of the dividing line has decreased, based on the position information of the moving body and referring to the position of the moving body with respect to the recognition dividing line information stored in the storage unit, the position and direction of the dividing line status information that corresponds to the position of the moving body with the recognition dividing line information within a specified distance range from the position information of the moving body are extracted, and the area where the probability of the dividing line exists is estimated to be above a specified value is extracted as the specified area based on the extracted position and direction.

Images