Mobile body control device, mobile body control method and storage medium

Abstract

This invention provides a moving body control device, a moving body control method, and a storage medium capable of more accurately identifying lane dividing lines. The moving body control device of one embodiment includes: a lane dividing line candidate identification unit that identifies candidates for lane dividing lines dividing the moving body from a first image captured by a camera unit, including the periphery of the moving body; and a lane dividing line search unit that selects the lane dividing line candidate identified by the lane dividing line candidate identification unit that is closest to a past lane dividing line dividing the moving body's travel lane before a predetermined time as the current lane dividing line dividing the moving body's travel lane.

Description

Technical Field

[0001] This invention relates to a mobile body control device, a mobile body control method, and a storage medium. Background Technology

[0002] In recent years, research related to autonomous driving, which automatically controls the driving of vehicles, has made increasing progress. In this regard, there are known technologies that adjust the configuration of driving lanes based on the driving lanes around the vehicle detected by sensors that detect the conditions around the vehicle, and the driving lanes around the vehicle obtained from map data based on the vehicle's position and posture (e.g., Japanese Patent Application Publication No. 2018-55414). Summary of the Invention

[0003] However, due to instability in the identification process by sensors that detect the surroundings of moving objects such as vehicles, lane markings on actual roads may disappear or become interrupted, potentially resulting in the identification of more lane markings than actually exist. Therefore, the lane markings that define the vehicle's lane may sometimes be inaccurately identified.

[0004] The present invention was made in consideration of such circumstances, and one of its objectives is to provide a moving body control device, moving body control method and storage medium capable of more accurately identifying the dividing lines that divide driving lanes.

[0005] Solution for solving the problem

[0006] The mobile body control device, mobile body control method, and storage medium of the present invention adopt the following structure.

[0007] (1): A mobile body control device according to one aspect of the present invention includes: a dividing line candidate identification unit, which identifies candidates for dividing lines that divide the driving lane of the mobile body from a first image captured by a camera unit, including the periphery of the mobile body; and a dividing line search unit, which selects the dividing line candidate identified by the dividing line candidate identification unit that is closest to a past dividing line that divided the driving lane of the mobile body before a predetermined time as the dividing line for dividing the current driving lane of the mobile body.

[0008] (2): In the above (1) scheme, the past dividing line is extracted from a second image including the periphery of the moving body captured by the camera unit before the specified time.

[0009] (3): In the above (1) scheme, the dividing line search unit detects the proximity between the dividing line candidate identified by the dividing line candidate identification unit and the past dividing line in the interval locally intercepted by the dividing line candidate identification unit based on the reference position.

[0010] (4): In the above (1) scheme, when the dividing line search unit compares each of the dividing line candidates identified by the dividing line candidate identification unit with the past dividing line, it determines the dividing line that is closest to the past dividing line based on the distance from a predetermined number of points on the past dividing line to the dividing line candidate starting from a predetermined search starting point.

[0011] (5): In the above (4) scheme, the dividing line search unit transforms the first image and the second image including the periphery of the moving body captured by the camera unit before the specified time into an image obtained by observing the moving body from above, and determines the dividing line closest to the past dividing line based on the mean square value of the difference between the lateral position of each candidate dividing line contained in the transformed image and the past dividing line.

[0012] (6): In the above (4) scheme, the search starting point is set based on the parameter information of the camera unit.

[0013] (7): In the above (1) scheme, the dividing line search unit corrects the position of the past dividing line based on one or both of the amount and direction of movement of the moving body since the specified time, or the road shape.

[0014] (8): In the above (1) scheme, if the degree of similarity between the candidate dividing line identified by the candidate dividing line identified by the candidate dividing line identification unit and the candidate dividing line that is closest to the past dividing line that divided the driving lane of the moving body before the specified time is above a threshold, the dividing line search unit shall take the past driving lane dividing line that was searched in the past as the dividing line dividing line of the current driving lane.

[0015] (9): In the above (1) scheme, a driving control unit is also provided, which controls one or both of the speed and steering of the moving body so that the moving body travels along the dividing line that divides the driving lane of the moving body determined by the dividing line search unit.

[0016] (10): A mobile body control method according to one aspect of the present invention causes the computer of the mobile body control device to perform the following processing: identify candidates for dividing the driving lane of the mobile body from a first image captured by the camera unit, including the periphery of the mobile body; and select the candidate for dividing the driving lane of the mobile body that is closest to the past dividing line of the driving lane of the mobile body before a predetermined time as the dividing line of the current driving lane of the mobile body.

[0017] (11): In one aspect of the present invention, the storage medium stores a program, wherein the program causes the computer of the mobile body control device to perform the following processing: identifying candidates for dividing the driving lane of the mobile body from a first image captured by the camera unit, including the periphery of the mobile body; and selecting the candidate for dividing the driving lane of the mobile body that is closest to a past dividing line that divided the driving lane of the mobile body before a predetermined time as the dividing line for dividing the current driving lane of the mobile body.

[0018] According to the above schemes (1) to (11), the lane markings for this vehicle can be identified more accurately. Attached Figure Description

[0019] Figure 1 This is a structural diagram of a vehicle system utilizing the mobile body control device of the implementation method.

[0020] Figure 2 This is a functional structure diagram of the first control unit and the second control unit.

[0021] Figure 3 This diagram illustrates the function of the candidate identification section for dividing lines.

[0022] Figure 4 This is a diagram showing an example of the lane markings that have been in place for this vehicle.

[0023] Figure 5 This is a diagram used to illustrate the comparison between the positions of the past dividing line PL1 and the candidate dividing lines LC1~LC7.

[0024] Figure 6 This is a diagram used to illustrate the method for deriving the distance between past dividing lines and dividing line candidates.

[0025] Figure 7 This is a flowchart illustrating an example of the processing flow performed by the automatic driving control device of the embodiment. Detailed Implementation

[0026] Hereinafter, embodiments of the mobile body control device, mobile body control method, and storage medium of the present invention will be described with reference to the accompanying drawings. The mobile body control device is a device for controlling the movement of a mobile body. A mobile body includes vehicles with three or four wheels, two-wheeled vehicles, micro-mobile bodies, etc., and may also include all mobile bodies, such as those for carrying passengers and capable of moving on roads with lanes. In the following description, the mobile body is assumed to be a four-wheeled vehicle and referred to as "this vehicle M". Hereinafter, the case where this vehicle M is primarily an autonomous driving vehicle will be described. Autonomous driving refers to, for example, automatically controlling one or both of the steering and speed of this vehicle M. The driving control of this vehicle M may include, for example, LKAS (Lane Keeping Assistance System), which is a driving support control that prevents this vehicle M from leaving its lane (driving lane). Driving control may include various driving support controls such as ALC (Auto Lane Changing) and ACC (Adaptive Cruise Control). Autonomous driving vehicles can also be controlled by manual driving by a passenger (driver). Alternatively, the drive source of this vehicle M may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or electricity discharged from a secondary battery or fuel cell.

[0027] [Overall Structure]

[0028] Figure 1 This is a structural diagram of a vehicle system 1 utilizing the mobile body control device of the embodiment. The vehicle system 1 includes, for example, a camera 10, a radar device 12, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU (Map Positioning Unit) 60, driving controls 80, an automatic driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected via multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. Figure 1 The structure shown is just one example; a part of the structure may be omitted, or other structures may be added. Camera 10 is an example of a "camera unit." Automatic driving control device 100 is an example of a "movement control device."

[0029] Camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). Camera 10 is mounted anywhere on the vehicle M equipped with vehicle system 1. When taking pictures of the front, camera 10 is mounted on the upper part of the windshield, behind the rearview mirror inside the vehicle, etc. Camera 10, for example, periodically and repeatedly takes pictures of the surroundings of the vehicle M. Camera 10 can also be a stereo camera.

[0030] Radar device 12 radiates millimeter-wave or other radio waves around the vehicle M and detects the radio waves (reflected waves) reflected by objects to at least detect the position (distance and orientation) of the objects. Radar device 12 can be installed at any part of the vehicle M. Radar device 12 can also detect the position and speed of objects using FM-CW (Frequency Modulated Continuous Wave) method.

[0031] The object recognition device 16 analyzes the image of the vehicle M's frontal area captured by the camera 10 and extracts necessary information. Furthermore, the object recognition device 16 performs sensor fusion processing on the detection results from the camera 10 and the radar device 12 to identify the object's position, type, speed, etc., and outputs the identification results to the autonomous driving control device 100. In this invention, the radar device 12 can be omitted; in this case, the object recognition device 16 may only have the function of analyzing images. Alternatively, the detection results from the radar device 12 may be directly output to the autonomous driving control device 100 without sensor fusion processing. The function of the object recognition device 16 may also be included in the autonomous driving control device 100 (more specifically, the recognition unit 130 described later). In this case, the object recognition device 16 may also be omitted from the vehicle system 1.

[0032] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using cellular networks, Wi-Fi networks, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via wireless base stations.

[0033] The HMI30 outputs various information to the occupants of the vehicle M under the control of the HMI control unit 170. The HMI30 can also function as a receiving unit for input operations performed by the occupants. The HMI30 includes, for example, a display device, a speaker, a microphone, a buzzer, buttons, and indicator lights. The display device is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device.

[0034] Vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting angular velocity about a vertical axis, and an orientation sensor for detecting the orientation of the vehicle M. Vehicle sensor 40 may also include a position sensor for obtaining the position of the vehicle M. The position sensor may be, for example, a sensor that obtains position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that uses a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50 to obtain position information.

[0035] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M can also be determined or supplemented using INS (Inertial Navigation System) output from the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, buttons, etc. The navigation HMI 52 can also be partially or entirely shared with the aforementioned HMI 30. The route determination unit 53, for example, refers to the first map information 54 to determine the route (hereinafter referred to as the map path) from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52. The first map information 54 is, for example, information representing the shape of a road by indicating road segments and nodes connecting the road segments. The first map information 54 may also include road curvature, POI (Point of Interest) information, etc. The path on the map is output to the MPU 60. The navigation device 50 can also provide route guidance using the navigation HMI 52 based on the path on the map. The navigation device 50 can also be implemented, for example, through the functions of a terminal device such as a smartphone or tablet held by the occupant. The navigation device 50 can also send its current location and destination to the navigation server via the communication device 20, and obtain the path equivalent to the path on the map from the navigation server.

[0036] MPU 60 includes, for example, a lane recommendation unit 61, and stores second map information 62 in a storage device such as an HDD or flash memory. The lane recommendation unit 61 divides the path on the map provided by the navigation device 50 into multiple blocks (e.g., every 100 [m] in the vehicle's direction of travel), and determines a recommended lane for each block by referring to the second map information 62. For example, if the currently traveling lane or the road to be traveled in the near future has multiple lanes, the lane recommendation unit 61 determines which lane to travel in from the left. If the path on the map has branching points, the lane recommendation unit 61 determines a recommended lane so that the vehicle M can travel on a reasonable path to the branch destination.

[0037] The second map information 62 is map information with higher precision than the first map information 54. The second map information 62 may include, for example, information about the center of a lane or the boundaries of a lane (e.g., road markings). The second map information 62 may include road information (road type), the number of lanes, the presence or absence of branches or merging, legal speed limits (speed limit, maximum speed, minimum speed), traffic restrictions, residential information (address, postal code), facility information, telephone number information, etc. The second map information 62 can be updated in real time by communicating with other devices through the communication device 20.

[0038] The driving control unit 80 includes, for example, a steering wheel, accelerator pedal, brake pedal, gear shift lever, and other control components. Sensors are installed on the driving control unit 80 to detect the amount of operation or whether there is operation, and the detection results are output to some or all of the automatic driving control unit 100, or the driving force output device 200, the braking device 210, and the steering device 220.

[0039] Next, before describing the automatic driving control device 100, the driving force output device 200, the braking device 210, and the steering device 220 will be described. The driving force output device 200 outputs the driving force (torque) for driving the vehicle M to the drive wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU (Electronic Control Unit) that controls them. The ECU controls the above-mentioned structure according to the information input from the automatic driving control device 100 (specifically, the second control unit 160 described later) or the information input from the driving operation device 80.

[0040] The braking device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a braking ECU. The braking ECU controls the electric motor according to information input from the second control unit 160 or from the driving operation unit 80, so as to output braking torque corresponding to the braking operation to each wheel. The braking device 210 may have a backup mechanism for transmitting hydraulic pressure generated by the operation of the brake pedal included in the driving operation unit 80 via the master hydraulic cylinder to the hydraulic cylinder. The braking device 210 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that transmits hydraulic pressure from the master hydraulic cylinder to the hydraulic cylinder by controlling the actuator according to information input from the second control unit 160.

[0041] The steering system 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the second control unit 160 or from the steering wheel 82 of the driving operation unit 80.

[0042] Next, the automatic driving control device 100 will be described. The automatic driving control device 100 includes, for example, a first control unit 120, a second control unit 160, an HMI control unit 170, and a storage unit 180. The first control unit 120, the second control unit 160, and the HMI control unit 170 are each implemented by executing programs (software) via hardware processors such as CPUs (Central Processing Units). Some or all of these components can be implemented using hardware (including circuitry) such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units), or through the coordinated use of software and hardware. The program can be pre-stored in a storage device such as an HDD or flash memory (a storage device with a non-transitory storage medium) of the automatic driving control device 100, or stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 by mounting the storage medium (a non-transitory storage medium) onto the drive unit. Combining the action plan generation unit 140 and the second control unit 160 constitutes an example of a "driving control unit".

[0043] The storage unit 180 can also be implemented using various storage devices described above, or SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 180 stores, for example, image information 182, dividing line information 184, information required for performing various processes in this embodiment, programs, and other various information. Image information 182 is information of the surrounding image of the vehicle M (including at least an image of the vehicle M's direction of travel) captured by the camera 10. Hereinafter, the image captured by the camera 10 will be referred to as a "camera image." Alternatively, the image information 182 may establish a correspondence between the shooting time of the camera 10 and the position and direction of the vehicle M at the time of shooting, and the camera image. Image information 182 stores camera images from a predetermined time ago (before a predetermined image frame). In this case, the predetermined time can be a fixed time or a variable time corresponding to the speed of the vehicle M and the road shape. Hereinafter, the camera images from a predetermined time ago will be referred to as "past camera images." The past camera image is an example of a "second image". The lane divider information 184 is information about the road dividing lines (hereinafter simply referred to as "dividing lines") that were identified by the recognition unit 130 before a predetermined time. The lane divider information 184 may, for example, include the position information of the lane in which the vehicle M is traveling (hereinafter referred to as "vehicle lane"), obtained from the analysis results of the past camera image. In this case, the storage unit 180 retains the lane dividing lines from the past camera image. The storage unit 180 may also store map information (first map information 54, second map information 62).

[0044] Figure 2 This is a functional structure diagram of the first control unit 120 and the second control unit 160. The first control unit 120, for example, includes a recognition unit 130 and an action plan generation unit 140. The first control unit 120, for example, implements AI (Artificial Intelligence) based functions and functions based on pre-given models in parallel. For example, the function of "recognizing intersections" can be achieved by "parallel execution of intersection recognition based on deep learning, etc., and recognition based on pre-given conditions (the existence of signals capable of pattern matching, road signs, etc.), and comprehensively evaluating both." This ensures the reliability of autonomous driving.

[0045] The recognition unit 130 identifies the position (relative position), speed (relative speed), acceleration, and other states of objects (e.g., other vehicles, other obstacles) surrounding the vehicle M based on information input from the camera 10 and radar device 12 via the object recognition device 16. The position of an object is identified, for example, as its position in the absolute coordinate system (vehicle center coordinate system) with a representative point of the vehicle M (center of gravity, drive shaft center, etc.) as the origin, and is used for control. The position of an object can also be represented by representative points such as the object's center of gravity or corners, or by a region. The "state" of an object, in the case of a moving object such as another vehicle, can also include the acceleration, jerk, or "action state" of that other vehicle (e.g., whether a lane change is in progress or about to occur).

[0046] The recognition unit 130 identifies the vehicle's lane based, for example, at least on information input from the camera 10. Specifically, the recognition unit 130 includes, for example, a dot group acquisition unit 132, a lane dividing line candidate recognition unit 134, and a lane dividing line search unit 136, and uses each function to identify the lane dividing the vehicle's lane and the vehicle's lane itself. Details of their respective functions will be described later.

[0047] The identification unit 130 uses the functions of the point group acquisition unit 132, the lane divider candidate identification unit 134, and the lane divider search unit 136 to identify the vehicle's driving lane. In addition (or alternatively), it identifies the lane divider that divides the vehicle's driving lane by comparing the pattern of the lane divider obtained from the second map information 62 (e.g., the arrangement of solid and dashed lines) with the pattern of the lane divider around the vehicle M identified from the image captured by the camera 10. The identification unit 130 is not limited to identifying lane dividers; it can also identify driving road boundaries (road boundaries) including shoulders, curbs, median strips, guardrails, etc. In this identification, the position of the vehicle M obtained from the navigation device 50 and the processing results from the INS can also be taken into consideration. The identification unit 130 identifies temporary stop lines, obstacles, red lights, toll booths, road signs, and other road phenomena. The identification unit 130 can also identify adjacent lanes adjacent to the vehicle's driving lane and oncoming lanes opposite to the vehicle's driving lane. Adjacent lanes are, for example, lanes that allow travel in the same direction as the lane in which the vehicle is traveling.

[0048] When the identification unit 130 identifies the vehicle's driving lane, it can also identify the position and posture of the vehicle M relative to the driving lane. For example, the identification unit 130 can identify the deviation of the vehicle M's reference point from the center of the driving lane, and the angle formed by the vehicle M's direction of travel relative to the line connecting the centers of the driving lanes, as the relative position and posture of the vehicle M relative to the driving lane. Alternatively, the identification unit 130 can also identify the position of the vehicle M's reference point relative to any side end of the driving lane (e.g., a dividing line or road boundary) as the relative position of the vehicle M relative to the driving lane. Here, the reference point of the vehicle M can be the center of the vehicle M or its center of gravity. The reference point can be an end of the vehicle M (front end, rear end) or the position of one of the multiple wheels of the vehicle M.

[0049] The action plan generation unit 140 generates a target track for the future automatic (independent of driver operation) travel of the vehicle M based on the identification results identified by the identification unit 130, in a manner that allows the vehicle M to travel in principle within the recommended lane determined by the recommended lane determination unit 61 and to be able to respond to the surrounding conditions of the vehicle M. The target track may include speed elements. For example, the target track may be a track formed by sequentially arranging the locations (track points) that the vehicle M should reach. Track points are locations that the vehicle M should reach at predetermined travel distances (e.g., several meters), but target speeds and target accelerations are generated as part of the target track at predetermined sampling times (e.g., a few tenths of a second). Track points may also be positions that the vehicle M should reach at the specified sampling time. In this case, the target speed and target acceleration information are represented by the intervals of the track points. The action plan generation unit 140 may also generate a target track with the vehicle M traveling at a predetermined speed within the traversable range, provided that a preset speed for the vehicle M has been determined.

[0050] When generating a target track, the action plan generation unit 140 can set events (functions) for automatic driving. These events include constant speed driving events, low-speed following events, lane change events, branching events, merging events, and takeover events. The action plan generation unit 140 generates a target track corresponding to the initiated event. The action plan generation unit 140 can also, when executing driving control of the vehicle M or pre-defined events, propose (suggest) driving control or event execution to the occupants based on the driving mode of the vehicle M (described later), and generate a corresponding target track if the proposal is accepted.

[0051] The second control unit 160 controls the driving force output device 200, the braking device 210 and the steering device 220 so that the vehicle M passes through the target track generated by the action plan generation unit 140 at a predetermined time.

[0052] The second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires information about the target track (track point) generated by the action plan generation unit 140 and stores this information in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the braking device 210 based on the speed elements associated with the target track stored in the memory. The steering control unit 166 controls the steering device 220 based on the curvature of the target track stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is achieved, for example, through a combination of feedforward control and feedback control. As an example, the steering control unit 166 combines feedforward control corresponding to the curvature of the road ahead of the vehicle M with feedback control based on deviation from the target track.

[0053] HMI control unit 170 uses HMI 30 to notify the occupants (driver) of the vehicle M of prescribed information. This prescribed information may include, for example, driving support information. For instance, HMI control unit 170 may generate an image including the prescribed information and display the generated image on the display device of HMI 30, or generate an audio representation of the prescribed information and output the audio from the speaker of HMI 30. HMI control unit 170 may also output the information received by HMI 30 to communication device 20, navigation device 50, first control unit 120, etc.

[0054] [Point Group Acquisition Unit, Dividing Line Candidate Identification Unit, Dividing Line Search Unit]

[0055] The following describes in detail the functions of the point group acquisition unit 132, the dividing line candidate identification unit 134, and the dividing line search unit 136.

[0056] The point group acquisition unit 132 acquires the current camera image (an image showing the frontal situation of the vehicle M) from the camera 10, and acquires the point group of objects contained in the acquired camera image. The current camera image is an example of a "first image". For example, the point group acquisition unit 132 extracts edge points from the camera image (first image) using existing image analysis processing, and acquires the point group of objects by summarizing the points within a predetermined interval among the extracted edge points. The point group acquired at this time is, for example, a point group defined in an image obtained by transforming the camera image to a two-dimensional coordinate system (overlooking coordinate system) obtained by viewing the vehicle M from above. In this case, the reference position (origin) of the two-dimensional coordinate system is, for example, a representative point of the vehicle M. The point group acquisition unit 132 may also save the camera image acquired from the camera 10 as image information 182 in the storage unit 180.

[0057] The dividing line candidate identification unit 134 identifies candidates for dividing lines that divide the driving lane of the vehicle M from the point group obtained by the point group acquisition unit 132. Figure 3 This diagram illustrates the function of the dividing line candidate identification unit 134. Figure 3 In the example, the vehicle M is shown traveling at speed VM in the direction of the arrow, and the dividing line candidate is identified by the method described later.

[0058] For example, the dividing line candidate identification unit 134 forms a dot array based on the dot arrays obtained by the dot array acquisition unit 132, which are arranged at intervals within a predetermined distance along the same direction (including the allowable angle range), and identifies a dividing line candidate by comparing the formed dot array with a predetermined dot array pattern. For example, the dividing line candidate identification unit 134 identifies a dot array as a dividing line candidate if the dot array extends a predetermined distance in a linear form or if it is non-linear but extends a predetermined distance with a predetermined curvature.

[0059] The dividing line candidate recognition unit 134 obtains dividing line candidates for the current time point by inputting the point group data obtained by the point group acquisition unit 132 into a learned model such as a DNN (deep neural network) that has been trained by learning point group data as input and dividing line candidates corresponding to the point group data as output. The dividing line candidate recognition unit 134 can also obtain dividing line candidates for the current time point by inputting a camera image into a learned model such as a DNN that has been trained by learning camera images as input and dividing line candidates corresponding to the image as output. In this case, the point group acquisition unit 132 can be omitted from the recognition unit 130. The trained model described above can be stored in the storage unit 180 or obtained from an external device via communication through the communication device 20. Hereinafter, the dividing line candidate recognition unit 134 will describe the dividing line candidates identified by the dividing line candidate recognition unit 134, such as... Figure 3 The case of dividing the line into candidate lines LC1 to LC7 is shown.

[0060] The lane division search unit 136 searches for (tracks) the lane division lines that divide the vehicle's driving lanes from the lane division line candidate LC1 to LC7 identified by the lane division line candidate identification unit 134. For example, the lane division search unit 136 obtains the current lane division line of the vehicle M based on the lane division line candidates LC1 to LC7 and the information on past lane division lines contained in past camera images (second images) included in the lane division information 184 stored in the storage unit 180. Alternatively, the lane division search unit 136 may obtain the lane division line by analyzing past camera images stored in the storage unit 180, instead of obtaining the past lane division line information from the lane division information 184.

[0061] Figure 4 This is a diagram showing an example of the lane markings that have been in place for this vehicle. Figure 4 In the example shown, the left and right dividing lines PL1 and PL2 of the vehicle's driving lane were previously obtained by the dividing line search unit 136 as dividing lines of the vehicle's driving lane and stored in the dividing line information 184 of the storage unit 180. Hereinafter, dividing lines PL1 and PL2 will be referred to as "past dividing lines PL1 and PL2". For example, based on the position information of past dividing lines PL1 and PL2, the dividing line search unit 136 replaces the past dividing lines PL1 and PL2 with the dividing lines PL1 and PL2 respectively. Figure 3 The coordinate system (vehicle center coordinate system) is based on the position of the vehicle M in the two-dimensional coordinate system of the camera image at the current time point, as shown.

[0062] Here, sometimes the position and orientation of the vehicle M at the time the past camera image was taken are different from the current position and orientation of the vehicle M. Therefore, the dividing line search unit 136 corrects the positions of the past dividing lines PL1 and PL2 based on one or both of the amount and direction of movement of the vehicle M from the time the past camera image was taken to the current time (more specifically, the time point when the camera images of dividing line candidates LC1 to LC7 were identified), or the road shape. For example, if the vehicle M has turned after the past camera image was taken, the dividing line search unit 136 locates the vehicle in the current camera image (first image) and the past camera image (second image) in a way that makes the vehicle's orientation (direction of movement) the same, and corrects the positions of the past dividing lines PL1 and PL2 accordingly. If the slope angle or curvature of the road has changed after the past camera image was taken, the positions of the past dividing lines PL1 and PL2 are corrected based on the slope angle and curvature. As a result, the positional relationship between the past dividing lines PL1 and PL2 and the dividing line candidates LC1 to LC7 can be obtained more accurately.

[0063] Next, the dividing line search unit 136 searches for the dividing line candidate LC1-LC7 that is closest to each of the past dividing lines PL1 and PL2, based on the positional relationship between the past dividing lines PL1 and PL2 and the dividing line candidates LC1-LC7. "Closest" means, for example, the closest distance. It may also include, in addition to distance (or instead of distance), at least one of the following: the inclination of the line relative to the vehicle's direction of travel, the shape of the line within the specified interval, the thickness of the line, the color of the line, and the type of line.

[0064] Figure 5 This is a diagram used to illustrate the comparison between the positions of the past dividing line PL1 and the candidate dividing lines LC1~LC7. For example, the dividing line search unit 136 searches for... Figure 5 As shown, the dividing line candidates LC1 to LC7 are those closest to the reference position RP1 set on the previous dividing line PL1. The reference position RP1 is set, for example, based on the position of this vehicle M. Figure 5 In this example, the reference position RP1 is set at the point where the line obtained by extending the front end of the vehicle M laterally (Y-axis direction) on the image intersects with the previous dividing line PL1. However, a line obtained by extending laterally along the road from the front end can also be used, or other points can be used as references. Similarly, the dividing line search unit 136 searches for the dividing line candidate LC1 to LC7 that is closest to the reference position set on the previous dividing line PL2.

[0065] Figure 6 This diagram illustrates the method for deriving the distance between past dividing lines and candidate dividing lines. The following explanation describes the method for deriving the distance between past dividing line PL1 and candidate dividing line LC2, but the same process is applied to deriving the distances between past dividing line PL1 and other candidate dividing lines LC1, LC3~LC7, as well as the distances between past dividing line PL2 and candidate dividing lines LC1~LC7. Figure 6 In the example, the intervals of the past dividing line PL1 and the candidate dividing line LC2 are shown by locally truncating.

[0066] The dividing line search unit 136 sets a search starting point SP on the dividing line candidate LC2, and locally extracts the intervals of both the dividing line PL1 and the dividing line candidate LC2 from the search starting point SP to the reference position RP1. The search starting point SP is determined, for example, based on camera information (parameter information of camera 10). The camera information includes, for example, the viewing angle and the shooting distance (limit distance). For example, by setting a search starting point based on the camera information that is at least within the shooting distance range, recognition errors can be suppressed and more accurate line comparisons can be performed.

[0067] The dividing line search unit 136 sets a predetermined number of points on the dividing line candidate LC2, staggered at predetermined intervals △I in the direction from the search starting point SP towards the existing reference position RP1, and derives the distances from each of the set points to the point where the line extending laterally (Y-axis direction) along the image (or the line extending vertically relative to the extension direction of the dividing line candidate LC2) intersects with the previous dividing line PL1. The predetermined interval △I can be, for example, a fixed interval predetermined in advance, or a variable interval based on the speed of the vehicle M, road shape, etc., detected by the vehicle sensor 40. The predetermined number can be, for example, a fixed number predetermined in advance, or a variable number based on the speed of the vehicle M, road shape, etc. Figure 6 In the example, four search points PO1 to PO4 are set. The dividing line search unit 136 derives the distance at each point based on the difference between the horizontal position of the search starting point SP and each point PO1 to PO4 in the image and the previous dividing line PL1. Figure 6In the example, since there is no past dividing line PL1 in the horizontal direction of the search starting point SP, the distance to that point is not derived; instead, the distances D11 to D14 corresponding to the search points PO1 to PO4 are derived. Furthermore, the dividing line search unit 136 derives the distance between the past dividing line PL1 and the candidate dividing line LC2 based on each distance D11 to D14. The distance is derived, for example, by calculating the mean square value of each distance D11 to D14. The distance can also be derived by adding the distances D11 to D14, averaging them, or substituting the values ​​of distances D11 to D14 into a prescribed function. The distance can be the distance on an image obtained by transforming a diagram viewed from above (a vehicle center coordinate system image), or it can be the actual distance adjusted based on the camera image capture information.

[0068] The lane divider search unit 136 similarly derives distances to other lane divider candidates LC1, LC3~LC7, and extracts the lane divider candidate closest to the past lane divider PL1. This extracted lane divider candidate is then designated as the lane divider that will divide the current vehicle's lane (more specifically, the lane divider to the left of the current vehicle's lane). Similarly, the lane divider search unit 136 derives distances between the past lane divider PL2 and each of the lane divider candidates LC1~LC7, and designates the lane divider candidate among the lane divider candidates LC1~LC7 whose distance is closest to the past lane divider PL2 as the lane divider that will divide the current vehicle's lane (more specifically, the lane divider to the right of the current vehicle's lane). If either of the past lane dividers PL1 and PL2 cannot be found among the lane divider candidates LC1~LC7, the lane divider search unit 136 can output only the position information of the other lane divider that was found. Therefore, the vehicle M can be driven based on the information of the lane divider obtained as a search result.

[0069] In the example above, the dividing line search unit 136 sets points at predetermined intervals from the search starting point SP toward the reference position RP1 (in other words, the front side of the vehicle M) for deriving distance. However, it can also set a predetermined number of points at predetermined intervals △I along the past dividing line, with the reference position of the past dividing line as the starting point and toward the search starting point SP. The dividing line search unit 136 can also set the predetermined interval △I in a way that allows a predetermined number of points to be extracted from the search starting point SP.

[0070] The dividing line search unit 136 can also connect two candidate dividing lines that extend continuously along the travel direction and are close in distance by interpolating a line between them. The dividing line search unit 136 can also extend the candidate dividing line or the past dividing line along the extension direction when there is no previous dividing line in the transverse direction of the candidate dividing line, or when the previous dividing line is short, so as to be able to derive the distance based on a predetermined number of search points. By setting a predetermined number of search points, the proximity of lines that include the shape within a predetermined interval can be more accurately derived, rather than just the distance of a single point.

[0071] The dividing line search unit 136 can, for example, exclude dividing line candidates that are farther from past dividing lines than other dividing line candidates from the distance-derived objects based on their respective positional relationships. For example, in Figure 5 In this case, the candidate dividing lines LC5~LC7 are excluded from the derived objects based on their distance from the past dividing line RP1 because they exist at a position farther away from LC1~LC4 than the previous dividing line RP1 (a distance greater than the specified distance). This reduces the number of candidate dividing lines to be searched, thus improving processing efficiency.

[0072] The lane divider search unit 136 can also determine that there is no lane divider equivalent to the past lane divider among the lane divider candidates LC1 to LC7 if the distance (proximity) between the past lane divider and the closest lane divider candidate is above a threshold (not below the threshold). In this case, the lane divider search unit 136 continues to use the lane divider of the past vehicle lane found in the past as the lane divider for the current vehicle lane. Therefore, even if a lane divider candidate cannot be identified from the camera image temporarily due to the surrounding environment of the vehicle M or the state of the lane dividers drawn on the road surface, driving support such as LKAS can continue to be performed based on the position of the lane divider of the past (e.g., just now) search results.

[0073] The lane division search unit 136 stores the information related to the lane division of the vehicle obtained through this search (e.g., the position information of the lane division based on the position of the vehicle M, etc.) and the information of the lane division itself divided by the lane division in the storage unit 180 in the lane division information 184, and uses it when searching for the lane division that divides the vehicle next time.

[0074] The identification unit 130 outputs the identification results, including the lane marking information of the vehicle's driving lane determined by the lane marking search unit 136, to the action plan generation unit 140. Based on the identification results, the action plan generation unit 140 generates a target track for the vehicle M in a manner that the vehicle M travels toward a destination set by the navigation device 50, and performs driving control to make the vehicle M travel along the generated target track.

[0075] [Processing Flow]

[0076] Figure 7 This is a flowchart illustrating an example of the process executed by the automated driving control device 100 of the embodiment. Hereinafter, the description will focus on the process executed by the automated driving control device 100, which involves searching for the lane dividers that separate the driving lanes of the vehicle M and performing driving control of the vehicle M based on information such as the lane dividers obtained as search results. Figure 7 The processing can be repeated at specified intervals during the execution of automatic driving control for this vehicle M.

[0077] exist Figure 7 In this example, the point group acquisition unit 132 acquires a camera image captured by the camera 10, representing the situation in front of the vehicle M (step S100). Next, the point group acquisition unit 132 performs an image transformation on the acquired image to an image coordinate system obtained from viewing the vehicle M from above (step S102), and acquires the point groups of objects contained in the camera image obtained after the image transformation (step S104). Next, the dividing line candidate recognition unit 134 identifies dividing line candidates from the point groups of objects (step S106).

[0078] Next, the lane division search unit 136 retrieves information from the lane division information 184 stored in the storage unit 180, indicating the lane division that previously defined the vehicle M's driving lane (step S108). In step S108, the lane division search unit 136 may also perform a correction by transforming the obtained position of the past lane division to a vehicle center coordinate system based on the current position of the vehicle M. Next, the lane division search unit 136 extracts the lane division closest to the past lane division from the lane division candidates (step S110). Next, the lane division search unit 136 determines whether the distance between the extracted closest lane division candidate and the past lane division is less than a threshold (step S112). If the distance is determined to be less than the threshold, the lane division search unit 136 determines this lane division candidate as the lane division that defines the current driving lane of the vehicle M (step S114). If, during the processing in step S112, it is determined that the distance is not less than the threshold (but is greater than the threshold), the lane division search unit 136 determines the lane division line stored in the storage unit 180 (e.g., the previously searched lane division line) as the lane division line for the current vehicle (step S116). After the processing in steps 116 and S118, the information representing the determined lane division line is stored in the storage unit 180 as lane division information 184 (step S118).

[0079] Next, the action plan generation unit 140 generates a target track for the vehicle M to travel in the center of the vehicle's driving lane defined by the determined dividing line (step S120). Then, the second control unit 160 executes driving control by controlling one or both of the vehicle M's steering and speed, so that the vehicle M travels along the generated target track (step S122). Thus, the processing of this flowchart ends.

[0080] <Variation Example>

[0081] The vehicle system 1 described above may also include a LIDAR (Light Detection and Ranging) system. The LIDAR illuminates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths close to light) and measures the scattered light. The LIDAR detects the distance to an object based on the time from the emission of light to the reception of light. The emitted light may be, for example, a pulsed laser. The LIDAR can be installed at any location on the vehicle M. For example, the object recognition device 16 may also receive input from the LIDAR in addition to the camera 10 and radar device 12, and identify surrounding objects (including dividing lines) based on the sensor fusion processing results obtained from the information from their respective inputs.

[0082] In the above-described embodiment, when the vehicle M is driven by the driver without a destination set based on the navigation device 50, the information on the lane markings of the vehicle's driving lanes searched by the lane marking search unit 136 can be used to perform driving control that supports a portion of the driving, including steering and speed control of the vehicle M.

[0083] The dividing line search unit 136 can also search for the dividing line candidate that is closest to the past dividing line from the dividing line candidates, for example, by using an existing matching algorithm to derive the closest (e.g., the one with the highest degree of consistency) dividing line candidate.

[0084] The dividing line search unit 136 can also search for the line closest to a past dividing line from the dividing line candidates, for example, by using an existing matching algorithm to derive the line closest to the past dividing line (e.g., the line with the highest degree of consistency). The information related to past dividing lines (the dividing lines searched last time) used in the dividing line search unit 136 during the search can also be obtained from the image stored in the image information 182 instead of from the dividing line information 184.

[0085] According to the above-described embodiments, the automatic driving control device (an example of a mobile body control device) includes: a lane dividing line candidate recognition unit 134, which identifies candidates for lane dividing lines that divide the driving lane of the vehicle (an example of a mobile body) from a first image captured by a camera (an example of an imaging unit) 10, which includes the periphery of the mobile body; and a lane dividing line search unit 136, which selects the lane dividing line candidate identified by the lane dividing line candidate recognition unit 134 that is closest to a past lane dividing line that divided the driving lane of the vehicle M before a predetermined time as the lane dividing line that divides the current driving lane of the vehicle M, thereby enabling more accurate identification of lane dividing lines that divide the driving lane of the vehicle.

[0086] Specifically, according to the above-described implementation method, the search is performed sequentially on the candidate lane dividers identified by the camera image that are closest to the lane divider closest to the lane previously identified, thereby enabling the vehicle to maintain its lane position while driving. According to the implementation method, the search for the current lane divider considers not only past lane divider information but also the vehicle's movement, direction of movement, road shape, etc., thus suppressing the possibility of losing track of the lane divider and enabling the vehicle to track its lane even if it leaves the lane. According to the implementation method, even if a lane divider cannot be identified temporarily, past information can be used to continue driving.

[0087] The implementation methods described above can be performed as follows.

[0088] A mobile body control device, configured to include:

[0089] Storage device, which stores a program; and

[0090] Hardware processor,

[0091] The hardware processor executes the program stored in the storage device to perform the following processing:

[0092] Candidates for dividing lines that define the driving lane of the moving body are identified from a first image captured by the camera unit, which includes the periphery of the moving body.

[0093] The candidate dividing line identified is the one that is closest to a past dividing line that divided the travel lane of the moving body before a specified time, and is then used as the dividing line for dividing the current travel lane of the moving body.

[0094] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A mobile body control device, wherein, The moving body control device includes: The dividing line candidate recognition unit identifies candidates for dividing lines that divide the driving lane of the moving body from a first image captured by the camera unit, which includes the periphery of the moving body. as well as The lane division search unit selects the lane division candidate identified by the lane division candidate identification unit that is closest to a past lane division line that divided the travel lane of the moving body before a predetermined time as the lane division line for dividing the current travel lane of the moving body. The dividing line search unit performs the following processing: Set the search starting point on the candidate of the dividing line. A reference position is set on the previously defined dividing line. Search points are set at predetermined intervals along the candidate lines from the search starting point toward the reference position. By comparing each candidate dividing line with a past dividing line, the distance from the search point to the past dividing line is derived, thereby deriving the distance between the past dividing line and the candidate dividing line. The candidate dividing line that is closest to the past dividing line among the candidates is identified as the dividing line that divides the current driving lane of the moving body.

2. The moving body control device according to claim 1, wherein, The past dividing line is extracted from a second image, including the periphery of the moving object, captured by the camera unit before the specified time.

3. The moving body control device according to claim 1, wherein, The dividing line search unit detects the proximity between the candidate dividing line identified by the dividing line candidate identification unit and the past dividing line in the interval partially intercepted based on the reference position.

4. The moving body control device according to claim 1, wherein, The dividing line search unit transforms the first image and the second image, which was captured by the camera unit before the predetermined time and includes the periphery of the moving body, into an image obtained by viewing the moving body from above. Based on the mean square value of the difference between the lateral position of each candidate dividing line contained in the transformed image and the past dividing line, it determines the dividing line that is closest to the past dividing line.

5. The moving body control device according to claim 1, wherein, The search starting point is set based on the parameter information of the camera unit.

6. The moving body control device according to claim 1, wherein, The dividing line search unit corrects the position of the past dividing line based on one or both of the amount and direction of movement of the moving body since the specified time, or the road shape.

7. The moving body control device according to claim 1, wherein, If the degree of similarity between the candidate dividing line identified by the candidate dividing line identification unit and the candidate dividing line that was closest to the past dividing line that divided the driving lane of the moving body before a predetermined time is above a threshold, the dividing line search unit shall use the past dividing line that divided the driving lane of the moving body as the dividing line of the current driving lane.

8. The moving body control device according to claim 1, wherein, The mobile body control device also includes a driving control unit that controls one or both of the speed and steering of the mobile body, so that the mobile body travels along the dividing line that divides the driving lane of the mobile body as determined by the dividing line search unit.

9. A method for controlling a moving body, wherein, The movement control method causes the computer of the movement control device to perform the following processing: Candidates for dividing lines that define the driving lane of the moving body are identified from a first image captured by the camera unit, which includes the periphery of the moving body. The candidate dividing line identified is the one that is closest to the past dividing line that divided the travel lane of the moving body before a specified time, and is used as the dividing line for dividing the current travel lane of the moving body. as well as Set the search starting point on the candidate of the dividing line. A reference position is set on the previously defined dividing line. Search points are set at predetermined intervals along the candidate lines from the search starting point toward the reference position. By comparing each candidate dividing line with a past dividing line, the distance from the search point to the past dividing line is derived, thereby deriving the distance between the past dividing line and the candidate dividing line. The candidate dividing line that is closest to the past dividing line among the candidates is identified as the dividing line that divides the current driving lane of the moving body.

10. A storage medium storing a program, wherein, The program causes the computer of the mobile body control device to perform the following processing: Candidates for dividing lines that define the driving lane of the moving body are identified from a first image captured by the camera unit, which includes the periphery of the moving body. The candidate dividing line identified is the one that is closest to the past dividing line that divided the travel lane of the moving body before a specified time, and is used as the dividing line for dividing the current travel lane of the moving body. as well as Set the search starting point on the candidate of the dividing line. A reference position is set on the previously defined dividing line. Search points are set at predetermined intervals along the candidate lines from the search starting point toward the reference position. By comparing each candidate dividing line with a past dividing line, the distance from the search point to the past dividing line is derived, thereby deriving the distance between the past dividing line and the candidate dividing line. The candidate dividing line that is closest to the past dividing line among the candidates is identified as the dividing line that divides the current driving lane of the moving body.

Images