Mobile body control device, mobile body control method and program

Abstract

To provide a mobile body control device, a mobile body control method and a program capable of recognizing a compartment line compartmentalizing a travel lane on which an own vehicle travels with improved accuracy.SOLUTION: A mobile body control device comprises: a candidate compartment line recognition section which recognizes a candidate compartment line compartmentalizing a travel lane where a mobile body travels thereon from an image, including surroundings of the mobile body, taken through an imaging section; a virtual line setting section which sets at least one virtual line outside the compartment line, with respect to the mobile body, recognized as the compartment line compartmentalizing the travel lane in the past; and a compartment line search section which searches the compartment line compartmentalizing the current travel lane of the mobile body on the basis of the candidate compartment line recognized with the candidate compartment line recognition section and the at least one virtual line.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a mobile object control device, a mobile object control method, and a program. [Background technology]

[0002] In recent years, research into autonomous driving, which automatically controls vehicle travel, has been progressing. In this context, a technology is known that adjusts lane arrangement based on the lanes around the vehicle recognized by a sensor that detects the situation around the vehicle and the lanes around the vehicle acquired from map data based on the position and attitude of the vehicle (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-55414 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, due to the instability of recognition by the sensors that detect the conditions around the vehicle, if a line with a similar shape to a marking line is recognized near the marking line being recognized, there is a possibility that the line will be mistakenly recognized as a marking line that defines the vehicle's driving lane.

[0005] The aspects of the present invention have been made in consideration of such circumstances, and one of the objects is to provide a mobile body control device, a mobile body control method, and a program that can more accurately recognize the dividing lines that define the driving lane in which the vehicle is traveling. [Means for solving the problem]

[0006] A mobile object control device, a mobile object control method, and a program according to the present invention employ the following configuration. (1): A mobile body control device according to one embodiment of the present invention is a mobile body control device that includes a lane candidate recognition unit that recognizes candidates for lane candidates that define the lane in which the mobile body is traveling from an image including the surroundings of the mobile body captured by an imaging unit, a virtual line setting unit that sets at least one virtual line outside, as viewed from the mobile body, a lane candidate that has previously been recognized as a lane candidate that defines the lane, and a lane search unit that searches for a lane candidate that defines the current lane of the mobile body based on the lane candidate candidates recognized by the lane candidate recognition unit and the one or more virtual lines.

[0007] (2) In the aspect (1) above, the virtual line setting unit sets the virtual line at a position where a boundary of a road including the current driving lane of the moving body is predicted to exist.

[0008] (3): In the above aspect (1) or (2), the virtual line setting unit sets a virtual line at a position a predetermined distance away from a dividing line that was previously recognized as a dividing line dividing the driving lane.

[0009] (4): In any one of the aspects (1) to (3) above, the lane marking search unit determines, among the lane marking candidates, the lane marking candidate that is closest to the lane marking that previously defined the driving lane as the lane marking that defines the current driving lane of the moving body.

[0010] (5): In any one of the above aspects (1) to (4), the lane marking search unit does not determine, among the lane marking candidates, any lane marking candidate that is within a predetermined distance from the virtual line as a lane marking that defines the current driving lane of the moving body.

[0011] (6): In any one of the above aspects (1) to (4), the lane line search unit recognizes, among the lane line candidates, those that are within a predetermined distance from the virtual line as boundary lines other than the lane line that separates the current driving lane of the moving body.

[0012] (7): In any one of the above aspects (1) to (6), a driving control unit is further provided which controls one or both of the speed and steering of the moving body so that the moving body travels along the dividing line that divides the driving lane of the moving body determined by the dividing line search unit.

[0013] (8): A driving assistance method according to one embodiment of the present invention is a mobile body control method in which a computer of a mobile body control device recognizes candidates for marking lines that define the driving lane in which the mobile body is traveling from an image including the surroundings of the mobile body captured by an imaging unit, sets at least one virtual line outside the moving body of a marking line that has previously been recognized as a marking line that defines the driving lane, and searches for a marking line that defines the current driving lane of the moving body based on the recognized candidates for marking lines and the one or more set virtual lines.

[0014] (9): A program according to one embodiment of the present invention causes a computer of a mobile body control device to recognize candidates for marking lines that define the driving lane in which the mobile body is traveling from an image including the surroundings of the mobile body captured by an imaging unit, set at least one virtual line outside the moving body of a marking line that was previously recognized as a marking line that defines the driving lane, and search for a marking line that defines the current driving lane of the mobile body based on the recognized candidates for marking lines and the one or more set virtual lines. Effect of the Invention

[0015] According to the above aspects (1) to (9), it is possible to more accurately recognize the dividing lines that define the driving lane in which the host vehicle is traveling. [Brief description of the drawings]

[0016] [Figure 1] 1 is a configuration diagram of a vehicle system 1 that uses a mobile object control device according to an embodiment. [Diagram 2] 2 is a functional configuration diagram of a first control unit 120 and a second control unit 160. FIG. [Diagram 3]11 is a diagram for explaining the function of a virtual line setting unit 132. FIG. [Figure 4] FIG. 13 is a diagram for explaining an example of the function of a lane line search unit 138. [Diagram 5] 11 is a diagram for explaining derivation of the distance between a lane marking candidate LC1 and a virtual line VL1. FIG. [Figure 6] 2 is a flowchart showing an example of a flow of processing executed by the automatic driving control device 100 of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, with reference to the drawings, an embodiment of a mobile body control device, a mobile body control method, and a program of the present invention will be described. The mobile body control device is a device that controls the movement of a mobile body. The mobile body may include a three-wheeled or four-wheeled vehicle, a two-wheeled vehicle, a micromobility, and the like, and may include any mobile body that has a person (passenger) on board and can move on a road surface or the like on which a driving lane such as a driving lane exists. In the following description, the mobile body is a four-wheeled vehicle and is referred to as a "host vehicle M". In addition, in the following, the host vehicle M will be mainly described as an autonomous driving vehicle. Autonomous driving means, for example, performing driving control that automatically controls one or both of the steering and the speed of the host vehicle M. The driving control of the host vehicle M includes, for example, a Lane Keeping Assistance System (LKAS), which is a driving assistance control that drives the host vehicle M without deviating from the host vehicle lane (driving lane) in which the host vehicle M is traveling. In addition, the driving control may include various driving assistance controls such as Auto Lane Changing (ALC) and Adaptive Cruise Control (ACC). Furthermore, the operation of an autonomous vehicle may be controlled by manual operation by an occupant (driver). Alternatively, the drive source of the vehicle M may be, for example, an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or discharged electric power from a secondary battery or a fuel cell.

[0018] [Overall configuration] FIG. 1 is a configuration diagram of a vehicle system 1 using a mobile object control device according to an embodiment. The vehicle system 1 includes, for example, a camera 10, a radar device 12, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driving operator 80, an automatic driving control device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and devices are connected to each other by multiple communication lines such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. Note that the configuration shown in FIG. 1 is merely an example, and a part of the configuration may be omitted, or another configuration may be added. The camera 10 is an example of an "imaging unit." The automatic driving control device 100 is an example of a "mobile object control device."

[0019] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to an arbitrary location of the host vehicle M in which the vehicle system 1 is mounted. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically and repeatedly captures images of the surroundings of the host vehicle M. The camera 10 may be a stereo camera.

[0020] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to an arbitrary location of the vehicle M. The radar device 12 may detect the position and speed of an object by an FM-CW (Frequency Modulated Continuous Wave) method.

[0021] The object recognition device 16 analyzes an image showing the situation ahead of the vehicle M captured by the camera 10, and extracts necessary information. Then, the object recognition device 16 performs sensor fusion processing on the detection results of the camera 10 and the radar device 12 to recognize the position, type, speed, etc. of the object, and outputs the recognition result to the automatic driving control device 100. In the present invention, the radar device 12 may be omitted, and in that case, the object recognition device 16 may only have a function of analyzing an image. Also, the detection result of the radar device 12 may be output directly to the automatic driving control device 100 without performing sensor fusion processing. Also, the function of the object recognition device 16 may be included in the automatic driving control device 100 (more specifically, the recognition unit 130 described later). In this case, the object recognition device 16 may be omitted from the vehicle system 1.

[0022] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via a wireless base station.

[0023] The HMI 30 outputs various information to the occupant of the vehicle M under the control of the HMI control unit 170. The HMI 30 may also function as a reception unit that receives input operations by the occupant. The HMI 30 includes, for example, a display device, a speaker, a microphone, a buzzer, keys, an indicator lamp, and the like. The display device is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device, and the like.

[0024] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the host vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, an orientation sensor that detects the orientation of the host vehicle M, and the like. The vehicle sensor 40 may also include a position sensor that acquires the position of the host vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. The position sensor may also be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50.

[0025] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 holds first map information 54 in a storage device such as a hard disk drive (HDD) or a flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on a signal received from a GNSS satellite. The position of the vehicle M may be identified or supplemented by an inertial navigation system (INS) using the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, and the like. The navigation HMI 52 may be partially or entirely shared with the HMI 30 described above. The route determination unit 53, for example, determines a route (hereinafter, a route on a map) from the position of the vehicle M identified by the GNSS receiver 51 (or an arbitrary position input) to a destination input by a driver using the navigation HMI 52, with reference to the first map information 54. The first map information 54 is, for example, information in which road shapes are expressed by links indicating roads and nodes connected by the links. The first map information 54 may include road curvature, POI (Point Of Interest) information, and the like. The route on the map is output to the MPU 60. The navigation device 50 may perform route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized by, for example, a function of a terminal device such as a smartphone or tablet terminal owned by a passenger. The navigation device 50 may transmit a current position and a destination to a navigation server via the communication device 20, and obtain a route equivalent to the route on the map from the navigation server.

[0026] The MPU 60 includes, for example, a recommended lane determination unit 61, and stores second map information 62 in a storage device such as an HDD or a flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of blocks (for example, by dividing it into 100 [m] blocks in the vehicle travel direction), and determines a recommended lane for each block by referring to the second map information 62. For example, when the lane on which the vehicle is currently traveling or the road on which the vehicle will travel in the near future has multiple lanes, the recommended lane determination unit 61 determines which lane from the left the vehicle should travel in. In addition, when a branch point is present on the route on the map, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel along a reasonable route to proceed to the branch point.

[0027] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries (for example, road dividing lines, road shoulders, curbs, medians, and guardrails). The second map information 62 may also include road information (road type), the number of lanes on a road, the presence or absence of branches and merging, legal speeds (speed limits, maximum speeds, minimum speeds), traffic regulation information, address information (addresses and zip codes), facility information, and telephone number information. The second map information 62 may be updated at any time by the communication device 20 communicating with another device.

[0028] The driving operators 80 include, for example, a steering wheel, an accelerator pedal, a brake pedal, a shift lever, and other operators. The driving operators 80 are fitted with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to the automatic driving control device 100, or some or all of the driving force output device 200, the brake device 210, and the steering device 220.

[0029] Next, prior to describing the automatic driving control device 100, the driving force output device 200, the brake device 210, and the steering device 220 will be described. The driving force output device 200 outputs a driving force (torque) for driving the host vehicle M to the driving wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU (Electronic Control Unit) that controls these. The ECU controls the above configuration according to information input from the automatic driving control device 100 (specifically, the second control unit 160 described later) or information input from the driving operator 80.

[0030] The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the second control unit 160 or information input from the driving operation element 80, so that a brake torque corresponding to a braking operation is output to each wheel. The brake device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of a brake pedal included in the driving operation element 80 to the cylinder via a master cylinder. Note that the brake device 210 is not limited to the configuration described above, and may be an electronically controlled hydraulic brake device that controls an actuator according to information input from the second control unit 160 to transmit hydraulic pressure in the master cylinder to the cylinder.

[0031] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by applying a force to, for example, a rack and pinion mechanism. The steering ECU drives the electric motor in accordance with information input from the second control unit 160 or information input from the steering wheel 82 of the driving operation device 80 to change the direction of the steered wheels.

[0032] Next, the automatic driving control device 100 will be described. The automatic driving control device 100 includes, for example, a first control unit 120, a second control unit 160, an HMI control unit 170, and a storage unit 180. The first control unit 120, the second control unit 160, and the HMI control unit 170 are each realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transient storage medium) such as an HDD or flash memory of the automatic driving control device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 by mounting the storage medium (non-transient storage medium) in a drive device. The combination of the action plan generation unit 140 and the second control unit 160 is an example of a "driving control unit".

[0033] The storage unit 180 may be realized by the above-mentioned various storage devices, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), or the like. The storage unit 180 stores, for example, image information 182, lane marking information 184, information necessary for executing various processes in the present embodiment, programs, and various other information. The image information 182 is information on an image of the surroundings of the vehicle M captured by the camera 10 (an image including at least the traveling direction of the vehicle M). Hereinafter, an image captured by the camera 10 is referred to as a "camera image." In addition, the image information 182 may correspond to the image capture time by the camera 10 and the position and direction of the vehicle M at the time of capture. The image information 182 stores a camera image from a predetermined time ago (a predetermined image frame ago). In this case, the predetermined time may be a fixed time or a variable time according to the speed of the vehicle M and the road shape. Hereinafter, a camera image from a predetermined time ago will be referred to as a "past camera image." A past camera image is an example of a "second image." The lane marking information 184 is information on road lane markings (hereinafter simply referred to as "lane markings") that demarcate lanes on a road recognized by the recognition unit 130 a predetermined time ago. The lane marking information 184 may include, for example, position information on the lane in which the host vehicle M is traveling (hereinafter referred to as "host vehicle lane") obtained from the analysis result of the past camera image. In this case, the memory unit 180 holds the lane markings that demarcate the host vehicle lane in the past camera image. The memory unit 180 may also store map information (first map information 54, second map information 62).

[0034] FIG. 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, a recognition unit 130 and an action plan generation unit 140. The first control unit 120 realizes, for example, a function based on AI (Artificial Intelligence) and a function based on a pre-given model in parallel. For example, the function of "recognizing an intersection" may be realized by executing in parallel recognition of an intersection by deep learning or the like and recognition based on pre-given conditions (there are signals, road markings, etc. that can be pattern matched), and by scoring both and evaluating them comprehensively. This ensures the reliability of the autonomous driving.

[0035] The recognition unit 130 recognizes the position (relative position), speed (relative speed), acceleration, and other states of objects (e.g., other vehicles, other obstacles) around the vehicle M based on information input from the camera 10 and the radar device 12 via the object recognition device 16. The position of the object is recognized as a position on absolute coordinates (vehicle-centered coordinate system) with a representative point of the vehicle M (center of gravity, center of drive shaft, etc.) as the origin, and is used for control. The position of the object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by an area. When the object is a moving body such as another vehicle, the "state" of the object may include the acceleration or jerk of the other vehicle, or the "behavior state" (e.g., whether or not the other vehicle is changing lanes or is about to change lanes).

[0036] Furthermore, the recognition unit 130 recognizes the vehicle lane, for example, based on at least information input from the camera 10. Specifically, the recognition unit 130 includes, for example, a virtual line setting unit 132, a point cloud acquisition unit 134, a lane line candidate recognition unit 136, and a lane line search unit 138. Each function recognizes the lane lines that demarcate the vehicle lane, the vehicle lane itself, or boundary lines other than the lane lines (for example, road boundary lines), etc. Details of each function will be described later.

[0037] In addition to (or instead of) recognizing the vehicle lane by the functions of the virtual line setting unit 132, the point cloud acquisition unit 134, the lane line candidate recognition unit 136, and the lane line search unit 138, the recognition unit 130 may recognize the lane lines dividing the vehicle lane and the vehicle lane itself by comparing the pattern of the lane lines (for example, an arrangement of solid lines and dashed lines) obtained from the second map information 62 with the pattern of the lane lines around the vehicle M recognized from the image captured by the camera 10. The recognition unit 130 may recognize road boundaries (road boundaries) including road shoulders, curbs, median strips, guardrails, etc., in addition to the lane lines. In this recognition, the position of the vehicle M acquired from the navigation device 50 and the processing results by the INS may be taken into consideration. The recognition unit 130 may recognize stop lines, obstacles, red lights, toll booths, road signs, and other road phenomena. The recognition unit 130 may also recognize adjacent lanes adjacent to the vehicle lane and oncoming lanes facing the vehicle lane. The adjacent lane is, for example, a lane in which the vehicle can travel in the same direction as the vehicle's own lane.

[0038] Furthermore, when the recognition unit 130 recognizes the host vehicle lane, the recognition unit 130 may recognize the position and attitude of the host vehicle M with respect to the host vehicle lane. For example, the recognition unit 130 may recognize the deviation of the reference point of the host vehicle M from the lane center and the angle with respect to a line connecting the lane centers in the traveling direction of the host vehicle M as the relative position and attitude of the host vehicle M with respect to the host vehicle lane. Alternatively, the recognition unit 130 may recognize the position of the reference point of the host vehicle M with respect to either side end of the host vehicle lane (for example, a dividing line or a road boundary) as the relative position of the host vehicle M with respect to the driving lane. Here, the reference point of the host vehicle M may be the center of the host vehicle M or the center of gravity. Furthermore, the reference point may be an end (front end, rear end) of the host vehicle M, or may be a position where one of a plurality of wheels equipped on the host vehicle M is present.

[0039] The behavior plan generating unit 140 generates a target trajectory along which the vehicle M will automatically (without the driver's operation) travel in the future based on the recognition result by the recognition unit 130 so that the vehicle M will travel along the recommended lane determined by the recommended lane determining unit 61 in principle, and furthermore, in order to respond to the surrounding conditions of the vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a sequence of points (trajectory points) to be reached by the vehicle M. The trajectory points are points to be reached by the vehicle M at each predetermined travel distance (for example, about several [m]) along the road, and separately, the target speed and target acceleration for each predetermined sampling time (for example, about a few tenths [sec]) are generated as part of the target trajectory. The trajectory points may also be positions to be reached by the vehicle M at each sampling time for each predetermined sampling time. In this case, the information on the target speed and target acceleration is expressed by the interval between the trajectory points. In addition, when a preset speed of the vehicle M is determined, the behavior plan generating unit 140 may generate a target trajectory such that the speed of the vehicle M becomes the set speed within a travelable range.

[0040] The behavior plan generating unit 140 may set an event (function) of autonomous driving when generating the target trajectory. The autonomous driving events include a constant speed driving event, a low speed following driving event, a lane change event, a branching event, a merging event, a takeover event, and the like. The behavior plan generating unit 140 generates a target trajectory according to the activated event. In addition, when executing driving control of the vehicle M or a predetermined event, the behavior plan generating unit 140 may propose (recommend) to the occupant the driving control or the execution of the event according to the driving mode of the vehicle M described later, and generate a corresponding target trajectory when the proposal is approved.

[0041] The second control unit 160 controls the traveling driving force output device 200, the braking device 210, and the steering device 220 so that the vehicle M passes the target trajectory generated by the action plan generating unit 140 at the scheduled time.

[0042] The second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires information on the target trajectory (trajectory points) generated by the action plan generation unit 140, and stores it in a memory (not shown). The speed control unit 164 controls the traveling driving force output device 200 or the brake device 210 based on a speed element associated with the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 executes a combination of feedforward control according to the curvature of the road ahead of the vehicle M and feedback control based on the deviation from the target trajectory.

[0043] The HMI control unit 170 notifies a passenger (driver) of the vehicle M of predetermined information via the HMI 30. The predetermined information includes, for example, driving assistance information. For example, the HMI control unit 170 may generate an image including the predetermined information described above and display the generated image on a display device of the HMI 30, or may generate a sound indicating the predetermined information and output the generated sound from a speaker of the HMI 30. In addition, the HMI control unit 170 may output the information received by the HMI 30 to, for example, the communication device 20, the navigation device 50, the first control unit 120, or the like.

[0044] [Virtual line setting section, point cloud acquisition section, lane line candidate recognition section, and lane line search section] The following describes in detail the functions of the virtual line setting unit 132, the point cloud acquisition unit 134, the lane line candidate recognition unit 136, and the lane line search unit 138. The virtual line setting unit 132 sets a virtual line based on the position of a lane line that was previously recognized as a lane line that divides the driving lane of the vehicle M and is included in the lane line information 184 stored in the memory unit 180. FIG. 3 is a diagram for explaining the function of the virtual line setting unit 132. In the example of FIG. 3, the vehicle M is assumed to be traveling in the X-axis direction at a speed VM. Furthermore, the lane lines LL and LR are lane lines that have been determined by the lane line search unit 138 as lane lines that divide the driving lane L1 of the vehicle M based on a past camera image (second image).

[0045] For example, on a road including a driving lane in which the vehicle M is traveling (hereinafter referred to as the vehicle lane), in addition to a dividing line dividing the vehicle lane, there are dividing lines dividing other lanes in which vehicles can travel in the same direction as the vehicle lane, and oncoming lanes in which oncoming vehicles travel, as well as road shoulders, curbs, sidewalks, etc. These often extend along the vehicle lane, so a line segment extracted by image analysis may be recognized as the vehicle lane dividing line. Therefore, the virtual line setting unit 132 sets a virtual line based on the position of a past dividing line at a position where a linear or nonlinear line segment is predicted to be extracted from a camera image due to a dividing line dividing a lane other than the vehicle lane, or an edge portion of a road boundary such as a road shoulder, curb, sidewalk, etc. For example, the virtual line setting unit 132 sets one or more virtual lines at a position a predetermined distance outside (farther away) from the dividing line as viewed from the vehicle M, based on the dividing line of the vehicle lane recognized from a past camera image. When multiple virtual lines are set, they may be set at a predetermined interval.

[0046] The above-mentioned predetermined distance may be, for example, a fixed distance, or may be a distance associated with a position where a road boundary line such as a dividing line or a road shoulder other than the vehicle lane exists by referring to map information (first map information 54, second map information 62) based on the position information of the vehicle M obtained from the vehicle sensor 40 and acquiring road information around the vehicle M. The above-mentioned predetermined distance may be the same length as the lane width of the vehicle lane L1. This allows a virtual line to be set at a position where a dividing line dividing an oncoming lane or an adjacent lane is likely to exist. The number of virtual lines may be a fixed number, or may be variably set depending on the road shape (for example, the number of lanes of the road on which the vehicle M runs, the presence or absence of a sidewalk, etc., obtained from the map information). A different number of virtual lines may be set for each of the dividing lines LL and LR on the left and right of the vehicle lane L1.

[0047] In the example of Fig. 3, using the left-hand demarcation line LL that defines the host vehicle lane L1 as a reference, a virtual line VL1 is set at a position a distance D1 away from the demarcation line LL as viewed from the host vehicle M, and a virtual line VL2 is set at a distance D2 that is further away than the distance D1. Also, in the example of Fig. 3, using the right-hand demarcation line LR that defines the host vehicle lane L1 as a reference, a virtual line VL1 is set at a position a distance D3 away from the demarcation line LR as viewed from the host vehicle M.

[0048] The point cloud acquisition unit 134 acquires a current camera image (an image showing a situation ahead of the vehicle M) from the camera 10, and acquires a point cloud of an object included in the acquired camera image. The current camera image is an example of a "first image". For example, the point cloud acquisition unit 134 extracts edge points from the camera image (first image) by existing image analysis processing, and acquires points within a predetermined interval among the extracted edge points as a point cloud of an object. The point cloud acquired at this time is, for example, a point cloud defined in an image obtained by image-converting the camera image into a two-dimensional coordinate system (bird's-eye coordinate system) in which the vehicle M is viewed from above. In this case, the reference position (origin) of the two-dimensional coordinates is, for example, a representative point of the vehicle M. The point cloud acquisition unit 134 may store the camera image acquired from the camera 10 as image information 182 in the storage unit 180.

[0049] The lane line candidate recognition unit 136 forms a point sequence from a point sequence that is arranged in the same direction (including the allowable angle range) at intervals within a predetermined distance from the point sequence acquired by the point cloud acquisition unit 134, and recognizes lane line candidates by comparing the formed point sequence with a predetermined point sequence pattern. For example, the lane line candidate recognition unit 136 recognizes a point sequence as a lane line candidate when the point sequence is linear and extends for a predetermined distance or more, or when the point sequence is nonlinear but extends for a predetermined distance or more with a predetermined curvature.

[0050] The lane line candidate recognition unit 136 acquires current lane line candidates by inputting the point cloud data acquired by the point cloud acquisition unit 134 into a trained model such as a deep neural network (DNN) that has been trained to receive point cloud data as an input and output candidates for lane lines corresponding to the point cloud data. The lane line candidate recognition unit 136 may acquire current lane line candidates by inputting a camera image into a trained model such as a deep neural network (DNN) that has been trained to receive a camera image as an input and output candidates for lane lines corresponding to the image. In this case, the point cloud acquisition unit 134 may be omitted from the recognition unit 130. The trained model described above may be stored in the storage unit 180, or may be acquired from an external device by communication via the communication device 20.

[0051] The lane marking search unit 138 determines the lane marking that demarcates the vehicle's lane from among the lane marking candidates recognized by the lane marking candidate recognition unit 136. FIG. 4 is a diagram for explaining an example of the function of the lane marking search unit 138. The example of FIG. 4 shows the lane markings LL and LR (past lane markings) on the left and right of the vehicle's lane L1 recognized from a past (predetermined time ago) camera image (second image), the virtual lines VL1 to VL3 set by the virtual line setting unit 132, and the lane marking candidates LC1 to LC3 recognized by the lane marking candidate recognition unit 136 based on the current camera image (first image). Note that the second image from which the lane markings LL, LR and the virtual lines VL1 to VL3 were acquired is a past camera image, and was captured at a different time from the first image from which the lane marking candidates LC1 to LC3 were recognized by the lane marking candidate recognition unit 136. Therefore, the lane line search unit 138 converts the first image and the second image into two-dimensional images of the vehicle M viewed from above, and overlays each image on a two-dimensional coordinate system (vehicle-centered coordinate system) based on the position of the vehicle M to obtain the positional relationship between the lane lines LL, LR and the virtual lines VL1-VL3 and the lane line candidates LC1-LC3 as shown in Fig. 4. The lane line search unit 138 may also correct the positions and shapes of the lane lines LL, LR and the virtual lines VL1-VL3, or the position and shape of the lane line candidates LC1-LC3 obtained from the second image, based on one or both of the amount and direction of movement of the vehicle M from when the first image is captured by the camera 10 to when the second image is captured, and the road shape.

[0052] For example, if the vehicle M has been turning since the previous camera image was captured, the lane line search unit 138 positions the vehicle so that its orientation (direction of movement) is the same in the current camera image (first image) and the previous camera image (second image), and corrects the positions of the lane lines LL, LR and virtual lines VL1-VL3 obtained from the second image in accordance with that orientation. Also, if the inclination angle or curvature of the road has changed since the previous camera image was captured by the vehicle M, the positions of the lane lines LL, LR and virtual lines VL1-VL3 are corrected according to the inclination angle and curvature. This makes it possible to more accurately obtain the positional relationship between the lane lines LL, LR and virtual lines VL1-VL3 and the lane line candidates LC1-LC3.

[0053] The lane line search unit 138 searches for the closest line (e.g., the nearest lane line or virtual line) to each of the lane line candidates LC1 to LC3 based on the positional relationship between the lane lines LL, LR and virtual lines VL1 to VL3 shown in Figure 4 and the lane line candidates LC1 to LC3. "Closest" means, for example, closest in distance. In addition to (or instead of) distance, it may also include closest in at least one of the following: closest in slope of the line when based on the direction of travel of the vehicle, closest in line shape, line thickness, line color, line type, etc. in a specified section. Below, as an example, an example is described in which the search is performed to find the line closest to the lane line candidate LC1 among the lane lines LL, LR and virtual lines VL1 to VL3.

[0054] For example, the lane line search unit 138 extracts a section between a reference point PLC1 on the lane line candidate LC1 that is present at a first predetermined distance from the position of the vehicle M (for example, a representative point) and reference points PLL, PLR on the lane lines LL, LR that are present at a predetermined position based on the position of the vehicle M, and reference points PV1 to PV3 on the virtual lines VL1 to VL3, respectively, and derives the distance in the extracted section. The first predetermined distance is, for example, a value determined based on camera information (parameter information of the camera 10). The parameter information includes, for example, the angle of view and the photographable distance (limit distance). For example, by setting the reference point PLC1 within a range that does not exceed at least the photographable distance based on the camera information, it is possible to suppress recognition errors and perform more accurate line comparison. In addition, the first predetermined distance may be, for example, the longest distance (the farthest position from the vehicle M) at which the camera image recognizes that the predetermined positions PLL, PLR, and the virtual lines VL1 to VL3 that are present in the road width direction are present, or may be a fixed distance (first fixed distance). Furthermore, the reference points PLL, PLR, PV1 to PV3 are, for example, points corresponding to the lateral position direction of a representative point of the host vehicle M (for example, the front end of the host vehicle), but other points may be used as references. Below, the derivation of the distance in the cut-out section between the lane line candidate LC1 and the virtual line VL1 will be described, but similar processing is also performed to derive the distance between the lane line candidate LC1 and each of the lane lines LL, LR and the virtual lines VL2, VL3.

[0055] FIG. 5 is a diagram for explaining derivation of the distance between the lane line candidate LC1 and the virtual line VL1. In the example of FIG. 5, the lane line search unit 138 sets a predetermined number of points on the lane line candidate LC1, shifted at a predetermined interval ΔI from the reference point PLC1 in the direction in which the reference point PV1 of the virtual line VL1 exists, using the reference point PLC1 of the lane line candidate LC1 as a search start point, and derives the distance from each of the set points to a point where a line extended in the horizontal direction (Y-axis direction) of the image (or a line extended in a direction perpendicular to the extension direction of the lane line candidate LC1) contacts the virtual line VL1. The predetermined interval ΔI may be, for example, a fixed interval determined in advance, or may be a variable interval based on the speed of the host vehicle M detected by the vehicle sensor 40, the road shape, and the like. The predetermined number may be, for example, a fixed interval determined in advance, or may be a variable number based on the speed of the host vehicle M, the road shape, and the like. In the example of FIG. 5, four search points PO1 to PO4 are set. The lane line search unit 138 derives the distance at each point based on the difference between the reference point PLC1 and each of the search points PO1 to PO4 and the virtual line VL1 in the lateral position direction of the image. In the example of FIG. 5, since the virtual line VL1 does not exist in the lateral direction of the reference point PLC1, the distance from this point is not derived, and the distances D11 to D14 corresponding to the search points PO1 to PO4 are derived. Then, the lane line search unit 138 derives the distance between the virtual line VL1 and the lane line candidate LC1 based on each of the distances D11 to D14. The distance is derived, for example, by calculating the mean square value of each of the distances D11 to D14. The distance may also be derived by adding or averaging the distances D11 to D14, or by substituting the values ​​of the distances D11 to D14 into a predetermined function. The distance may also be the distance on an image (vehicle-centered coordinate system image) converted into a view of the vehicle M from above, or may also be the actual distance adjusted based on the shooting information of the camera image.

[0056] The lane line search unit 138 also derives the distances to the other virtual lines VL2, VL3 and the lane lines LL, LR with respect to the lane line candidate LC1. The lane line search unit 138 then derives the closest lane line. For example, if the line closest to the lane line candidate LC1 is the past lane line LL or LR, the lane line search unit 138 determines that the lane line candidate LC1 is the lane line of the vehicle's lane L1. In addition, if the line closest to the lane line candidate LC1 is the virtual lines VL1-VL3, the lane line search unit 138 may determine that the lane line candidate LC1 is not the lane line that defines the vehicle's lane L1 (or that the lane line candidate LC1 is a boundary line other than the lane line that defines the driving lane). By determining a lane line candidate that is not the lane line of the driving lane L1, the lane line of the current vehicle's lane can be extracted more accurately.

[0057] Furthermore, when the lane marking search unit 138 determines that the lane marking candidate LC1 is a boundary line other than a lane marking that divides a driving lane, the lane marking search unit 138 may exclude the lane marking candidate LC1 from the process of searching for a lane marking of the vehicle lane L1 while the lane marking candidate LC1 is being recognized by the lane marking candidate recognition unit 136. This can reduce the processing load for determining the lane marking of the vehicle lane L1.

[0058] Furthermore, the lane line search unit 138 may not determine the closest line among the lane lines LL, LR, and the virtual lines VL1 to VL3 as a lane line if the distance between the lane line candidate LC1 and the closest line is equal to or greater than a threshold value. In this case, the lane line search unit 138 continues to use the lane line of the vehicle's lane that was previously searched for as the lane line that divides the current vehicle lane. This allows driving assistance such as LKAS to be continued based on the position of the lane line of the driving lane that is the past (e.g., most recent) search result, even if the lane line candidate cannot be temporarily recognized from the camera image due to the surrounding environment of the vehicle M or the state of the lane line drawn on the road surface. Furthermore, if the lane line search unit 138 cannot search for either of the past lane lines LL, LR from the lane lines LL, LR, and the virtual lines VL1 to VL3, it may output only the position information of the other lane line that was searched for. This allows the vehicle M to travel based on the information of one of the lane lines obtained as a search result.

[0059] 4, the line closest to the lane line candidate LC1 is the virtual line VL1, the line closest to the lane line candidate LC2 is the virtual line LR, and the line closest to the lane line candidate LC3 is the virtual line VL3. Therefore, the lane line search unit 138 determines that the lane line candidate LC2 is the lane line LR on the right side of the driving lane L1 in the direction of travel, and determines that the lane line candidates LC1 and LC3 are not lane lines of the driving lane L1.

[0060] In the above example, the lane line search unit 138 sets points for deriving distance at a predetermined interval from reference point PLC1 of the lane line candidate LC1 toward the reference point PV1 side (in other words, the front side of the vehicle M), but it may also set a predetermined number of points at a predetermined interval ΔI on the virtual line VL1 starting from reference point PV1 toward reference point PLC1. Also, the lane line search unit 138 may set the predetermined interval ΔI so that a predetermined number of points can be extracted from reference point PLC1.

[0061] Furthermore, when there are no demarcation lines LL, LR, or virtual lines VL1-VL3 horizontally of the demarcation line candidates, or when the lengths of the demarcation lines LL, LR, and virtual lines VL1-VL3 are short, the demarcation line search unit 138 may extend the demarcation line candidates LC1-LC3 or the demarcation lines LL, LR, and virtual lines VL1-VL3 in the extension direction so as to derive the distance using a predetermined number of search points. By setting a predetermined number of search points, it is possible to more accurately derive the closeness of the lines, including their shapes in a predetermined section, rather than the distance at only one point.

[0062] The marking line search unit 138 stores information regarding the marking lines of the vehicle's lane obtained by this search (e.g., position information of the marking lines based on the position of the vehicle M) and information about the vehicle's lane itself, which is marked by the marking lines, in marking line information 184 in the memory unit 180, and uses this information the next time it searches for marking lines that mark the vehicle's lane.

[0063] The recognition unit 130 outputs the recognition result including the lane marking information of the driving lane determined by the lane marking search unit 138 to the behavior plan generation unit 140. The behavior plan generation unit 140 generates a target trajectory of the host vehicle M based on the recognition result so that the host vehicle M travels toward the destination set by the navigation device 50, and executes driving control so that the host vehicle M travels along the generated target trajectory.

[0064] [Processing flow] Fig. 6 is a flowchart showing an example of a flow of processing executed by the automatic driving control device 100 of the embodiment. In the following, the processing executed by the automatic driving control device 100 will be mainly described with a focus on processing for searching for a marking line that marks the driving lane of the host vehicle M and for executing driving control of the host vehicle M based on information on the marking line etc. obtained as a result of the search. The processing in Fig. 6 may be repeatedly executed at a predetermined interval while driving control of the host vehicle M is being executed.

[0065] In the example of Figure 6, the virtual line setting unit 132 acquires information on past lane lines from the memory unit 180, etc. (step S100), and sets one or more virtual lines based on the positions of the past lane lines based on the acquired information (step S102).

[0066] Next, the point cloud acquisition unit 134 acquires a camera image including the current view ahead of the vehicle M captured by the camera 10 (step S104), and converts the acquired camera image into an image of the vehicle M viewed from above (step S106). Next, the point cloud acquisition unit 134 analyzes the coordinate-converted image and acquires point cloud data (step S108). Next, the lane marking candidate recognition unit 136 recognizes candidates for lane marks that demarcate the vehicle's lane based on the point cloud data acquired by the point cloud acquisition unit 134 (step S110).

[0067] Next, the lane line search unit 138 derives the distance between the past lane lines and virtual lines and the lane line candidates (step S112). Next, the lane line search unit 138 determines whether or not there is a lane line candidate whose distance from the past lane is less than a threshold (step S114). If it is determined that there is a lane line candidate whose distance from the past lane is less than the threshold, the lane line search unit 138 determines that the lane line candidate is the lane that divides the current vehicle lane (step S116).

[0068] Furthermore, if it is determined in the processing of step S114 that there is no lane line candidate whose distance from the past lane line is less than the threshold, the lane line search unit 138 determines whether there is a virtual line whose distance from the past lane line is less than the threshold (step S118). If it is determined that there is a virtual line whose distance from the past lane line is less than the threshold, the lane line search unit 138 recognizes that the lane line candidate is not a lane that demarcates the current vehicle lane (step S120).

[0069] After processing of step S120, or in processing of step S118, if it is determined that there is no virtual line whose distance to the previous lane line is less than the threshold value, the lane line search unit 138 continues to use information about the previously recognized lane lines and determines that the previous lane line is the lane line that divides the current vehicle lane (step S122).

[0070] After the processing of step S116 or step S122, the lane marking search unit 138 stores information about the determined lane marking in the storage unit 180 (step S124). Note that in the processing of step 124, information about lane marking candidates that are recognized as not being lane marks that demarcate the vehicle lane may be stored.

[0071] Next, the behavior plan generating unit 140 generates a target trajectory for the host vehicle M to travel along the center of the host vehicle lane defined by the determined dividing lines (step S126). Next, the second control unit 160 executes driving control to control one or both of the speed and steering so that the host vehicle M travels along the target trajectory (step S128). This ends the processing of this flowchart.

[0072] <Modification> The vehicle system 1 described above may be provided with a LIDAR (Light Detection and Ranging). The LIDAR irradiates light (or electromagnetic waves with a wavelength close to that of light) around the vehicle M and measures the scattered light. The LIDAR detects the distance to an object based on the time between light emission and light reception. The irradiated light is, for example, a pulsed laser light. The LIDAR is attached to an arbitrary location of the vehicle M. For example, the object recognition device 16 may receive input from the LIDAR in addition to the camera 10 and the radar device 12, and recognize surrounding objects (including lane markings) from the result of sensor fusion processing based on the information input from each of them.

[0073] In addition, in the above-mentioned embodiment, when a destination is not set by the navigation device 50 and the vehicle M is traveling by the driver's operation, the above-mentioned lane marking line search unit 138 may use information on the lane marking lines of the vehicle's lane that have been searched for to perform driving control that assists with some of the steering and speed of the vehicle M.

[0074] Furthermore, when searching for a line that is closest to a lane line candidate from past lane lines and virtual lines, the lane line search unit 138 may derive the closest (e.g., the line with the highest degree of match) line by using, for example, an existing matching algorithm. Furthermore, the lane line information 184 may store information on the virtual lines set by the virtual line setting unit 132 in association with information on the past lane lines. Furthermore, information on past lane lines (lane lines searched for previously) used during a search by the lane line search unit 138 may be obtained from an image stored in the image information 182 instead of the lane line information 184.

[0075] According to the above-described embodiment, an automatic driving control device (an example of a mobile body control device) is equipped with a lane marking candidate recognition unit 136 that recognizes candidates for lane marks that define the driving lane in which the host vehicle M is traveling from an image including the surroundings of the host vehicle (an example of a mobile body) M captured by a camera (an example of an imaging unit), a virtual line setting unit 132 that sets at least one virtual line outside the host vehicle M of a lane marking that has previously been recognized as a lane marking that defines the driving lane, and a lane marking search unit 138 that searches for a lane marking that defines the current driving lane of the host vehicle M based on the lane marking candidate recognized by the lane marking candidate recognition unit 136 and the one or more virtual lines, thereby making it possible to more accurately recognize the lane markings that define the driving lane in which the host vehicle is traveling.

[0076] Specifically, according to the above-described embodiment, by searching for a dividing line on the assumption that a line segment other than the vehicle lane exists outside the dividing line of the vehicle lane, it is possible to suppress erroneous recognition of a dividing line other than the vehicle lane dividing line as the vehicle lane dividing line. Furthermore, according to the above-described embodiment, it is possible to grasp road boundaries other than the dividing line dividing the vehicle lane. By driving the vehicle M or providing driving assistance based on the dividing lines on the left and right sides of the vehicle M obtained by the search process of this embodiment, it is possible to execute driving control with higher robustness. Furthermore, according to the above-described embodiment, by increasing the number of virtual lines, it is possible to search for a boundary line in a wide search range, and therefore it is also possible to apply it to determining the number of lanes of the road on which the vehicle M is traveling.

[0077] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Recognizing candidates for marking lines that mark the travel lane in which the moving object travels from an image including the periphery of the moving object captured by the imaging unit; setting at least one virtual line on the outer side, as viewed from the moving body, of a dividing line that has been previously recognized as a dividing line dividing the driving lane; searching for a lane marking that defines a current driving lane of the moving object based on the recognized lane marking candidate and the one or more virtual lines that have been set; The mobile object control device is configured as follows.

[0078] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0079] 1...vehicle system, 10...camera, 12...radar device, 16...object recognition device, 20...communication device, 30...HMI, 40...vehicle sensor, 50...navigation device, 60...MPU, 80...driving operator, 100...automatic driving control device, 120...first control unit, 130...recognition unit, 132...virtual line setting unit, 134...point cloud acquisition unit, 136...landing line candidate recognition unit, 138...landing line search unit, 140...action plan generation unit, 160...second control unit, 162...acquisition unit, 164...speed control unit, 166...steering control unit, 170...HMI control unit, 180...memory unit, 200...driving force output device, 210...brake device, 220...steering device

Claims

1. a marking line candidate recognition unit that recognizes candidates for marking lines that mark a travel lane in which the moving object travels from an image including the periphery of the moving object captured by the imaging unit; a virtual line setting unit that sets at least one virtual line on the outer side, as viewed from the moving body, of a marking line that has been previously recognized as a marking line that marks the driving lane; a lane marking search unit that searches for a lane marking that defines a current driving lane of the moving object based on the lane marking candidate recognized by the lane marking candidate recognition unit and the one or more virtual lines; A mobile object control device comprising:

2. The computer of the mobile control device Recognizing candidates for marking lines that mark the travel lane in which the moving object travels from an image including the periphery of the moving object captured by the imaging unit; setting at least one virtual line on the outer side, as viewed from the moving body, of a dividing line that has been previously recognized as a dividing line dividing the driving lane; searching for a lane marking that defines a current driving lane of the moving object based on the recognized lane marking candidate and the one or more virtual lines that have been set; A method for controlling a moving object.

3. The computer of the mobile control device Recognizing candidates for marking lines that mark a travel lane in which the moving object is traveling from an image including the periphery of the moving object captured by the imaging unit; setting at least one virtual line on the outer side, as viewed from the moving body, of a dividing line that has been previously recognized as a dividing line dividing the driving lane; a marking line that marks a current driving lane of the moving object is searched for based on the recognized marking line candidate and the one or more set virtual lines; program.

Images