Position estimation program, vehicle control program, and position estimation device

Abstract

Accurately estimate the vehicle's own position. [Solution] The position estimation program includes: a division step which divides a detection area identified by sensor data acquired by a sensor that detects the surrounding conditions of a moving object, based on the type of object to be detected included in the detection area, and generates area information indicating the position of the divided area within the detection area; a storage step which stores past sensor data acquired in the acquisition step in a storage unit in association with the area information; a calculation step which calculates the degree of discrepancy between the current position of the divided area identified by the area information associated with the current sensor data acquired in the acquisition step and the past position of the divided area identified by the area information associated with the past sensor data; and when the calculated degree of discrepancy is less than or equal to a predetermined value, it is estimated that the current position of the moving object is the same as the position of the moving object at the time the past sensor data was acquired.

Description

Technical Field

[0004] , , , ,

[0005] , , , , ,

[0003] , , ,

[0001] The present invention relates to a position estimation program, a vehicle control program, and a position estimation device for estimating a traveling position of a host vehicle on a map.

Background Art

[0002] Conventionally, as this type of device, there is known a device that estimates the self-position of a vehicle by combining image data acquired by a camera and point cloud data acquired by a lidar so as to improve the robustness of position estimation against environmental changes (see, for example, Patent Document 1). In the device described in Patent Document 1, image data is classified pixel by pixel using semantic segmentation, and the position of the vehicle on the map is estimated by performing scan matching between the point cloud data corresponding to the region classified as a stationary object and the point cloud data constituting the map information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, if the self-position is estimated by matching point cloud data as in the device described in Patent Document 1 above, there is a possibility that a decrease in position estimation accuracy due to environmental changes cannot be sufficiently suppressed.

Means for Solving the Problems

[0006] Another aspect of the present invention is a vehicle control program which includes the above-described position estimation program and further causes a computer to perform a step of controlling the movement of a moving object based on the self-position of the moving object estimated in the estimation step.

[0007] A self-positioning device in yet another aspect of the present invention includes: an acquisition unit that acquires sensor data obtained by a sensor that detects the surrounding conditions of a moving object; a division unit that divides a detection area identified by the sensor data into different types of objects included in the detection area and generates area information indicating the position of the divided area within the detection area; a storage unit that stores past sensor data acquired by the acquisition unit in association with the area information; a calculation unit that calculates the degree of discrepancy between the current position of a divided area identified by the area information associated with the current sensor data acquired by the acquisition unit and the past position of a divided area identified by the area information associated with past sensor data stored in the storage unit; and an estimation unit that estimates the self-position of the moving object based on the degree of discrepancy calculated by the calculation unit. The estimation unit estimates that the current self-position of the moving object is the same as the self-position at the time the past sensor data was acquired when the degree of discrepancy is less than or equal to a predetermined value. [Effects of the Invention]

[0008] According to the present invention, the vehicle's own position can be estimated with high accuracy. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram schematically showing the overall configuration of a vehicle control system according to an embodiment of the present invention. [Figure 2] A block diagram showing the main components of a vehicle control device according to an embodiment of the present invention. [Figure 3A] A diagram showing an example of a camera image. [Figure 3B] A diagram showing other examples of camera images. [Figure 4] A diagram illustrating the acquisition of image sequences. [Figure 5] A flowchart showing another example of the processing performed by the CPU of the controller in Figure 2. [Modes for carrying out the invention]

[0010] Embodiments of the invention will now be described with reference to the drawings. The position estimation device according to the embodiment of the present invention can be applied to a vehicle having an autonomous driving function, i.e., an autonomous vehicle. The vehicle to which the position estimation device according to this embodiment is applied may be referred to as the "self-vehicle" to distinguish it from other vehicles. The self-vehicle may be an engine vehicle having an internal combustion engine as a driving source, an electric vehicle having a drive motor as a driving source, or a hybrid vehicle having both an engine and a drive motor as driving sources. The self-vehicle can be driven not only in an autonomous driving mode that does not require driver operation, but also in a manual driving mode with driver operation.

[0011] First, the general configuration of the vehicle involved in autonomous driving will be described. Figure 1 is a block diagram that schematically shows the overall configuration of the vehicle control system 100 of the vehicle having a position estimation device according to this embodiment. As shown in Figure 1, the vehicle control system 100 mainly consists of a controller 10, a group of external sensors 1 and 2, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and an actuator AC for driving.

[0012] External sensor group 1 is a general term for multiple sensors (external sensors) that detect external conditions, which are information about the surroundings of the vehicle. For example, external sensor group 1 includes a lidar that measures the distance from the vehicle to surrounding obstacles by measuring the reflected light from the light illuminating the vehicle in all directions, a radar that detects other vehicles and obstacles around the vehicle by irradiating electromagnetic waves and detecting the reflected waves, and a camera mounted on the vehicle that has an image sensor such as a CCD or CMOS and captures images of the area around the vehicle (front, rear, and sides).

[0013] Internal sensor group 2 is a collective term for multiple sensors (internal sensors) that detect the vehicle's driving state. For example, internal sensor group 2 includes an inertial measurement unit (IMU) that detects the rotational angular velocity and acceleration in the three axes of the vehicle's center of gravity: vertical, longitudinal (direction of travel), and lateral (width direction). Sensors that detect the driver's driving operations in manual driving mode, such as accelerator pedal operation, brake pedal operation, and steering wheel operation, are also included in internal sensor group 2.

[0014] Input / output device 3 is a general term for devices that receive commands from the driver or output information to the driver. For example, input / output device 3 includes various switches that the driver uses to input commands by operating an operating component, a microphone that the driver uses to input commands by voice, a display that provides information to the driver via an image, and a speaker that provides information to the driver by voice.

[0015] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. Positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 uses the positioning information received by the positioning sensor to measure the current position (latitude, longitude, altitude) of the vehicle.

[0016] The map database 5 is a device that stores general map information used in the navigation device 6, and is composed of, for example, a hard disk or semiconductor elements. The map information includes road location information, road shape information (curvature, etc.), and location information of intersections and junctions. Note that the map information stored in the map database 5 is different from the high-precision map information stored in the storage unit 12 of the controller 10.

[0017] The navigation device 6 is a device that searches for a target route on the road to a destination input by the driver and provides guidance along the target route. The input of the destination and the guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the host vehicle measured by the positioning unit 4 and the map information stored in the map database 5. It is also possible to measure the current position of the host vehicle using the detection values of the external sensor group 1, and calculate the target route based on this current position and the highly accurate map information stored in the storage unit 12.

[0018] The communication unit 7 communicates with various servers (not shown) via a network including a wireless communication network typified by the Internet or a mobile phone network, and periodically or at an arbitrary timing acquires map information, driving history information, traffic information, etc. from the server. Not only can it acquire driving history information, but it may also transmit the driving history information of the host vehicle to the server via the communication unit 7. The network includes not only a public wireless communication network but also a closed communication network provided for each predetermined management area, such as a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The acquired map information is output to the map database 5 and the storage unit 12, and the map information is updated.

[0019] The actuator AC is a driving actuator for controlling the driving of the host vehicle. When the driving power source is an engine, the actuator AC includes a throttle actuator for adjusting the opening degree (throttle opening) of the throttle valve of the engine. When the driving power source is a driving motor, the driving motor is included in the actuator AC. The brake actuator for operating the braking device of the host vehicle and the steering actuator for driving the steering device are also included in the actuator AC.

[0020] The controller 10 is composed of an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM and a RAM, and other peripheral circuits (not shown) such as an I / O interface. Although a plurality of ECUs with different functions, such as an engine control ECU, a driving motor control ECU, and a braking device ECU, can be provided separately, in FIG. 1, for the sake of convenience, the controller 10 is shown as an aggregation of these ECUs.

[0021] The storage unit 12 stores highly accurate and detailed map information (referred to as highly accurate map information). The highly accurate map information includes road position information, road shape (such as curvature) information, road gradient information, intersection and branch point position information, types and position information of road demarcation lines such as white lines, number of lanes information, lane width and position information for each lane (information on the center position of the lane and the boundary lines of the lane position), position information of landmarks (buildings, traffic lights, signs, etc.) as marks on the map, and road surface profile information such as road surface unevenness. In the embodiment, the center line, lane boundary line, outer lane line, etc. are collectively referred to as road demarcation lines. The highly accurate map information stored in the storage unit 12 includes map information acquired from outside the host vehicle via the communication unit 7 (referred to as external map information), and a map created by the host vehicle itself using detection values by the external sensor group 1 or detection values of the external sensor group 1 and the internal sensor group 2 (referred to as internal map information).

[0022] External map information is, for example, map information obtained via a cloud server (referred to as a cloud map), while internal map information is, for example, map information consisting of 3D point cloud data generated by mapping using technologies such as SLAM (Simultaneous Localization and Mapping) (referred to as an environmental map). External map information is shared between the vehicle and other vehicles, whereas internal map information is the vehicle's own map information (for example, map information owned solely by the vehicle). For roads not yet traveled by the vehicle, newly constructed roads, etc., the vehicle itself creates the environmental map. Internal map information may also be provided to the server device or other vehicles via the communication unit 7. In addition to the high-precision map information described above, the storage unit 12 also stores information such as the vehicle's travel trajectory, various control programs, and thresholds used in the programs.

[0023] The calculation unit 11 has the following functional configuration: a vehicle position recognition unit 13, an external environment recognition unit 14, an action plan generation unit 15, a driving control unit 16, and a map generation unit 17.

[0024] The vehicle position recognition unit 13 recognizes (or estimates) the vehicle's position on the map (vehicle position) based on the vehicle's position information obtained by the positioning unit 4 and the map information in the map database 5. The vehicle position may also be recognized (estimated) using high-precision map information stored in the memory unit 12 and surrounding information of the vehicle detected by the external sensor group 1, thereby enabling high-precision recognition of the vehicle position. The vehicle's movement information (direction of movement, distance traveled) can also be calculated based on the detection values ​​of the internal sensor group 2, and the vehicle position can be recognized accordingly. Furthermore, when the vehicle's position can be measured by sensors installed on or beside the road, the vehicle position can also be recognized by communicating with those sensors via the communication unit 7.

[0025] The external environment recognition unit 14 recognizes the external conditions around the vehicle based on signals from the external sensor group 1, including LiDAR, radar, and cameras. For example, it recognizes the position, speed, and acceleration of surrounding vehicles (vehicles in front and behind) traveling around the vehicle, the position of surrounding vehicles that are stopped or parked around the vehicle, and the position and state of other objects. Other objects include signs, traffic lights, road markings such as lane markings and stop lines, buildings, guardrails, utility poles, billboards, pedestrians, and bicycles. The state of other objects includes the color of traffic lights (red, blue, yellow), the speed and direction of pedestrians and cyclists, etc. Some of the stationary objects among the other objects constitute landmarks that serve as indicators of location on a map, and the external environment recognition unit 14 also recognizes the position and type of these landmarks.

[0026] The action plan generation unit 15 generates a driving trajectory (target trajectory) for the vehicle from the present time to a predetermined time in advance, based on, for example, the target route calculated by the navigation device 6, the high-precision map information stored in the memory unit 12, the vehicle's position recognized by the vehicle position recognition unit 13, and the external conditions recognized by the external environment recognition unit 14. If there are multiple candidate trajectories for the target trajectory on the target route, the action plan generation unit 15 selects the optimal trajectory from among them that meets criteria such as complying with laws and regulations and driving efficiently and safely, and sets the selected trajectory as the target trajectory. The action plan generation unit 15 then generates an action plan corresponding to the generated target trajectory. The action plan generation unit 15 generates various action plans corresponding to overtaking driving to overtake a preceding vehicle, lane change driving to change driving lanes, following driving to follow a preceding vehicle, lane keeping driving to maintain the driving lane without deviating from the driving lane, deceleration driving, or acceleration driving. When generating a target trajectory, the action plan generation unit 15 first determines the driving mode and generates the target trajectory based on the driving mode.

[0027] In automatic driving mode, the driving control unit 16 controls each actuator AC so that the vehicle travels along the target trajectory generated by the action plan generation unit 15. More specifically, in automatic driving mode, the driving control unit 16 calculates the required driving force to obtain the target acceleration per unit time calculated by the action plan generation unit 15, taking into account the driving resistance determined by the road gradient, etc. Then, it provides feedback control to the actuator AC so that the actual acceleration detected by, for example, the internal sensor group 2 becomes the target acceleration. In other words, it controls the actuator AC so that the vehicle travels at the target vehicle speed and target acceleration. In manual driving mode, the driving control unit 16 controls each actuator AC in accordance with driving commands (such as steering operations) from the driver acquired by the internal sensor group 2.

[0028] The map generation unit 17 generates an environmental map of the roads the vehicle has traveled on, as internal map information, using detection values ​​detected by the external sensor group 1 while the vehicle is driving in manual driving mode. For example, it extracts edges and characteristic regions (blobs) that indicate the outlines of objects based on the brightness and color information of each pixel from multiple frames of camera images acquired by the camera, and extracts feature points using the information of those edges and blobs. Feature points are, for example, the intersections of edges and correspond to the corners of buildings or road signs. The map generation unit 17 calculates the 3D position of a feature point while estimating the camera's position and orientation so that identical feature points converge to a single point across multiple frames of camera images, according to the SLAM technology algorithm. By performing this calculation process for each of the multiple feature points, it generates an environmental map consisting of 3D point cloud data. Alternatively, instead of a camera, data acquired by radar or LiDAR may be used to extract feature points of objects around the vehicle and generate an environmental map.

[0029] The vehicle position recognition unit 13 may perform vehicle position recognition processing based on the environmental map generated by the map generation unit 17 and feature points extracted from the camera image. This allows the vehicle position recognition processing to be performed based on the stored environmental map when the vehicle is traveling to a point corresponding to the stored environmental map after the environmental map has been generated. The vehicle position recognition unit 13 may also perform vehicle position recognition processing in parallel with the map creation processing by the map generation unit 17. The map creation processing and position recognition (estimation) processing are performed simultaneously according to the SLAM technology algorithm. The map generation unit 17 can generate an environmental map not only when driving in manual driving mode but also when driving in automatic driving mode. If an environmental map has already been generated and stored in the storage unit 12, the map generation unit 17 may update the environmental map based on newly extracted feature points from newly acquired camera images.

[0030] Incidentally, one method for estimating the vehicle's own position (self-position of the vehicle) is to match feature point clouds extracted from camera images with 3D point cloud data included in the environmental map to estimate the vehicle's self-position on the environmental map. However, feature points extracted from camera images are easily affected by the surrounding environment (illumination, etc.). Therefore, even if camera images are acquired at the same driving position, the extracted feature points may not match if the time of acquisition or weather conditions at the time of acquisition are different. In this case, there is a risk that the vehicle's self-position cannot be estimated with accuracy. To address this problem, the position estimation device according to this embodiment is configured as follows.

[0031] Figure 2 is a block diagram showing the main components of the vehicle control device 50 according to this embodiment. This vehicle control device 50 constitutes a part of the vehicle control system 100 shown in Figure 1. As shown in Figure 2, the vehicle control device 50 includes a controller 10, a camera 1a, a lidar 1b, and a radar 1c. The vehicle control device 50 also includes a position estimation device 60, which constitutes a part of the vehicle control device 50. The position estimation device 60 estimates the vehicle's position on a map based on the detection data (camera image) from the camera 1a.

[0032] Camera 1a is a monocular camera having an image sensor such as a CCD or CMOS, and constitutes part of the external sensor group 1 in Figure 1. Camera 1a detects the surrounding conditions of the vehicle. Camera 1a is mounted, for example, at a predetermined position on the front of the vehicle, and continuously captures images of the space in front of the vehicle at a predetermined frame rate, and sequentially outputs frame image data (camera images) as detection information to the controller 10. Note that camera 1a may be a stereo camera.

[0033] The lidar 1b is mounted on the vehicle and detects the distance from the vehicle to surrounding obstacles by measuring scattered light from the vehicle's omnidirectional illumination. The lidar 1b outputs the detected value (detection data) to the controller 10. The radar 1c is mounted on the vehicle and detects other vehicles and obstacles around the vehicle by emitting electromagnetic waves and detecting the reflected waves. The radar 1c outputs the detected value (detection data) to the controller 10.

[0034] The controller 10 includes a calculation unit 11 and a storage unit 12. The calculation unit 11 has the following functional configuration: an acquisition unit 111, a division unit 112, a deviation degree calculation unit 113, an estimation unit 114, and a driving control unit 16.

[0035] The acquisition unit 111, the division unit 112, and the deviation calculation unit 113 are composed of, for example, the vehicle position recognition unit 13 shown in Figure 1. The estimation unit 114 is composed of, for example, the vehicle position recognition unit 13 and the map generation unit 17 shown in Figure 1. The acquisition unit 111, the division unit 112, the deviation calculation unit 113, the estimation unit 114, the camera 1a, the lidar 1b, the radar 1c, the IMU 2a, and the storage unit 12 are all included in the position estimation device 60.

[0036] The acquisition unit 111 acquires camera images obtained by camera 1a. More specifically, the acquisition unit 111 continuously acquires camera images while the vehicle is moving.

[0037] The division unit 112 performs region division processing, which divides the imaging region identified by the camera image by classifying each pixel within the imaging region. Specifically, the division unit 112 classifies the imaging region identified by the camera image on a pixel-by-pixel basis, and divides the imaging region into regions (hereinafter referred to as divided regions) composed of pixels classified in the same class. Examples of classes include "vehicles," "roads," "buildings," "plantings," and "background." Semantic segmentation may be used for dividing the imaging region, or other segmentation techniques may be used.

[0038] The division unit 112 generates region information indicating the position of the divided region within the imaging region and stores it in the storage unit 12. At the same time, the division unit 112 stores the camera image used to generate the region information in the storage unit 12, in association with the region information.

[0039] The deviation calculation unit 113 compares the position of each divided region, which is identified by region information generated based on the current camera image acquired by the acquisition unit 111, with the position of each divided region, which is identified by region information associated with past camera images stored in the storage unit 12.

[0040] Figures 3A and 3B show examples of camera images acquired by camera 1a. Region CA1 in Figure 3A is the imaging region identified by the current camera image acquired by the acquisition unit 111. Region CA2 in Figure 3B is the imaging region identified by past camera images acquired from the same travel position as the camera image in Figure 3A. Regions BL11, BL12, BL13, VC1, RD1, PL11, and PL12 in Figure 3A are divided regions obtained by dividing imaging region CA1. Regions BL21, BL22, BL23, RD2, PL21, and PL22 in Figure 3B are divided regions obtained by dividing imaging region CA2.

[0041] The divided regions BL11, BL12, BL13, BL21, BL22, and BL23 are regions composed of pixels classified as "buildings," respectively. The divided regions RD1 and RD2 are regions composed of pixels classified as "roads," respectively. The divided regions PL11, PL12, PL21, and PL22 are regions composed of pixels classified as "plantings," respectively. The divided region VC1 is a region composed of pixels classified as "vehicles."

[0042] The deviation calculation unit 113 calculates the degree of overlap between a divided region included in the imaging area of ​​the current camera image (hereinafter referred to as the comparison source region) and a divided region included in the imaging area of ​​a past camera image that is classified in the same class as the comparison source region (hereinafter referred to as the comparison target region). Specifically, the deviation calculation unit 113 calculates IOU (Intersection over Union), which is an index for evaluating the degree of overlap of the regions. IOU is expressed as a numerical value between 0.0 and 1.0. 0.0 represents a state in which the comparison source region and the comparison target region do not overlap at all, and 1.0 represents a state in which both regions completely overlap.

[0043] The deviation calculation unit 113 calculates the IOU for each class. In the examples in Figures 3A and 3B, the IOU is calculated between divided regions BL11, BL12, BL13 and divided regions BL21, BL22, BL23. The IOU is also calculated between divided region RD1 and divided region RD2. Furthermore, the IOU is calculated between divided regions PL11, PL12 and divided regions PL21, PL22. Note that regions composed of pixels classified as moving objects such as "vehicles" (for example, divided region V1 in Figure 3A) are excluded from the calculation of IOUs. The exclusion of moving objects may be performed before region division. Alternatively, the class of a moving object may be determined to be the same as any other class.

[0044] The deviation calculation unit 113 sets (calculates) the deviation between the position of each divided region corresponding to the current camera image and the position of each divided region corresponding to past camera images, such that the deviation decreases as the IOU increases. The method for setting the deviation can be any given method; for example, the deviation calculation unit 113 may calculate the average IOU (hereinafter referred to as the average IOU) a by averaging the IOUs calculated for each class, and then calculate its complement (1-a) as the deviation.

[0045] Furthermore, the vehicle's position in the width direction may not match between past and present driving. For example, when the roads shown in Figures 3A and 3B are two-lane roads in each direction, the vehicle's position in the width direction will differ depending on whether it is driving in the left lane or the right lane. In this case, even if the vehicle's position in the direction of travel is the same, the imaging area identified by the current camera image and the imaging area identified by past camera images will be shifted in the width direction, which may prevent accurate calculation of the IOU.

[0046] Therefore, when calculating the IOU, the deviation calculation unit 113 performs a width-direction adjustment to adjust the imaging area identified by the current camera image in the vehicle width direction (horizontal axis direction in the camera image). Specifically, the deviation calculation unit 113 repeatedly calculates the average IOU value a while offsetting the imaging area identified by the current camera image by a specified amount (for example, n pixels) in the vehicle width direction relative to the imaging area identified by past camera images. Then, the deviation calculation unit 113 calculates the complement of the largest average IOU value a among the calculated average IOU values ​​a as the deviation value.

[0047] The estimation unit 114 searches among the past camera images stored in the memory unit 12 for the past camera image with the smallest deviation calculated by the deviation calculation unit 113, that is, the past camera image with the highest degree of agreement with the current camera image. If the target past camera image is identified as a result of the search, the estimation unit 114 determines whether the deviation of the identified past camera image is less than or equal to a predetermined value. If the deviation is less than or equal to the predetermined value, the estimation unit 114 estimates that the vehicle's current position, that is, its position at the time the current camera image was acquired, is the same as its position at the time the identified past camera image was acquired.

[0048] Furthermore, roads on which vehicles travel often have sections with similar scenery or locations with scenery similar to other points. When a vehicle is traveling on such a road, the above search may identify multiple past camera images corresponding to the current camera image. In this case, past camera images acquired at a location different from the vehicle's current position may be used for self-localization, potentially leading to errors in the estimation results.

[0049] Therefore, in order to suppress such errors in self-position estimation, the estimation unit 114 performs the above search using a series of camera images (sequential frame images) acquired while the vehicle travels a predetermined distance, rather than a single camera image. Now, the above search using a series of camera images will be explained.

[0050] The acquisition unit 111 acquires a series of camera images (hereinafter referred to as the image sequence) acquired by the camera 1a during the vehicle's journey when the vehicle has traveled a predetermined distance. The acquisition unit 111 also measures the vehicle's distance traveled by odometry. Specifically, the acquisition unit 111 estimates the amount and direction of movement of the vehicle based on the acceleration and angular velocity detected by the IMU 2a and calculates the vehicle's distance traveled. The acquisition unit 111 may also measure the vehicle's distance traveled based on detected values ​​from a vehicle speed sensor, yaw rate sensor, etc. (not shown).

[0051] The division unit 112 generates region information corresponding to each individual camera image included in the image sequence, associates the image sequence with the region information corresponding to each camera image, and stores it in the storage unit 12. As a result, each time the vehicle travels a predetermined distance, the image sequence and its corresponding region information are stored in the storage unit 12.

[0052] Figure 4 is a diagram illustrating the acquisition of image sequences. The triangles in the figure schematically indicate the timing at which the image sequences are stored in the storage unit 12. As shown in Figure 4, when the vehicle 101 travels a predetermined distance X (m), the camera images (image sequences) acquired by the camera 1a during that travel are stored in the storage unit 12. In this way, image sequences corresponding to the road the vehicle 101 is traveling on are stored in the storage unit 12 for each X-meter section.

[0053] The deviation calculation unit 113 compares the current image sequence acquired by the acquisition unit 111 with multiple past image sequences stored in the storage unit 12, and calculates the comparison result (deviation) for each.

[0054] When calculating the degree of discrepancy between the current image sequence and past image sequences, the discrepancy calculation unit 113 first identifies camera images with a large IOU (small degree of discrepancy) from among the individual camera images included in the current image sequence, using multiple stored past images. That is, it identifies corresponding pairs of camera images (current camera image and past camera image) between the current image sequence and multiple stored past images. At this time, if there is no inconsistency between the time series of the current camera image and the time series of the past camera image corresponding to the current camera image, it is determined that the past image sequence containing the corresponding past camera image corresponds to the current image sequence. Note that the corresponding pairs of camera images may be identified based on other criteria. The discrepancy calculation unit 113 calculates the degree of discrepancy for each identified pair of camera images and calculates the average value as the degree of discrepancy between image sequences.

[0055] Furthermore, when calculating the degree of deviation for each set of identified camera images, the deviation calculation unit 113 applies the above width-direction adjustment to each of the sets. At this time, the deviation calculation unit 113 may use the adjustment amount (offset amount in the vehicle width direction) of the set to which the width-direction adjustment was first applied for the width-direction adjustment of the other sets.

[0056] The estimation unit 114 searches among the past image sequences stored in the memory unit 12 for the past image sequence with the smallest deviation calculated by the deviation calculation unit 113, that is, the past image sequence with the highest degree of agreement with the current image sequence. By performing the above search using a series of camera images instead of a single camera image, the accuracy of the search can be improved, and errors in the estimation results in self-localization, as described above, can be suppressed.

[0057] The estimation unit 114 determines whether the degree of deviation of the identified past image sequence is less than or equal to a predetermined value. If the degree of deviation is less than or equal to the predetermined value, the estimation unit 114 estimates that the vehicle's current position, more specifically, its position at the time the current image sequence was acquired, is the same as its position at the time the identified past image sequence was acquired.

[0058] The estimation unit 116 updates the environmental map based on the current image sequence when the estimated self-position is outside the range of the environmental map stored in the memory unit 12, specifically when information (point cloud data) corresponding to the estimated self-position does not exist in the environmental map. Specifically, the estimation unit 116 updates the environmental map stored in the memory unit 12 based on the feature point cloud extracted from each camera image included in the current image sequence. As a result, when the vehicle travels on a road that does not exist in the environmental map at least twice, map information corresponding to that road is automatically added to the environmental map.

[0059] The driving control unit 16 controls the vehicle's movement based on the vehicle's position estimated by the estimation unit 116. Specifically, the driving control unit 16 controls each actuator AC so that the vehicle travels along the target trajectory generated by the action plan generation unit 15 based on the vehicle's position estimated by the estimation unit 116.

[0060] Figure 5 is a flowchart showing an example of a process executed by the CPU of controller 10 in Figure 2 according to a predetermined program.

[0061] The process shown in this flowchart starts when the vehicle control system 100 is activated and is repeated at a predetermined interval while the vehicle control system 100 is running. Specifically, it is repeated each time a camera image is input from camera 1a, that is, at a time interval determined by the frame rate of camera 1a. Note that the predetermined interval may be variable rather than constant, taking into consideration traffic safety requirements and computational load.

[0062] First, in step S1, the controller 10 acquires a camera image. In step S2, the controller 10 determines whether the vehicle has traveled a predetermined distance. Specifically, the controller 10 determines whether the measured value D of the vehicle's travel distance has reached a predetermined distance X. The initial value of the measured value D is zero. If the result in step S2 is negative, the controller 10 stores the camera image acquired in step S1 in the storage unit 12 and repeats the process in step S2 until it is determined to be positive. If the result in step S2 is positive, the controller 10 stores the camera image acquired in step S1 in the storage unit 12 and proceeds to the process in step S3. At this time, the controller 10 resets the measured value D to zero.

[0063] In step S3, the controller 10 performs region division processing on the camera images (current image sequence) stored in the storage unit 12 while the vehicle travels a predetermined distance. The controller 10 stores region information indicating the result of the region division processing in the storage unit 12, associating it with the current image sequence.

[0064] In step S4, the controller 10 compares the current image sequence with multiple past image sequences stored in the storage unit 12 and calculates the comparison result (deviation degree) for each. In step S5, the controller 10 identifies the past image sequence with the smallest deviation degree from among the multiple past image sequences and determines whether the deviation degree of the identified past image sequence is less than or equal to a predetermined value.

[0065] If the result in step S5 is negative, the controller 10 terminates the process. If the result in step S5 is positive, in step S6, the controller 10 estimates the current position of its own vehicle. Specifically, it estimates that the position of the vehicle at the time the current image sequence was acquired is the same as the position of the vehicle at the time the past image sequence identified in step S5 was acquired. In step S7, the controller 10 controls the vehicle's movement based on the position estimated in step S6.

[0066] In step S8, the controller 10 determines whether the self-position estimated in step S6 is included in the environment map stored in the memory unit 12. If it is denied in step S8, the controller 10 terminates the process. If it is affirmed in step S8, in step S9, the controller 10 updates the environment map based on the current image sequence. Specifically, it updates the environment map based on the feature point cloud extracted from each camera image included in the current image sequence.

[0067] According to the embodiments described above, the following effects and advantages can be obtained. (1) The position estimation device 60 includes an acquisition unit 111 that acquires camera images acquired by a camera 1a that detects the surrounding conditions of the vehicle, and a division unit 112 that divides the imaging area identified by the camera image according to the type of detection target included in the imaging area, generates area information indicating the position of the divided area in the imaging area, and stores past camera images acquired by the acquisition unit 111 in the storage unit 12 in association with the area information. The division unit 112 classifies each pixel constituting the camera image based on the type of detection target corresponding to each pixel, and divides the imaging area based on the result of the classification. The position estimation device 60 further includes a deviation degree calculation unit 113 that calculates the deviation degree between the current position of the divided area identified by the area information associated with the current camera image acquired by the acquisition unit 111 and the past position of the divided area identified by the area information associated with past camera images stored in the storage unit 12, and an estimation unit 114 that estimates the vehicle's own position based on the deviation degree calculated by the deviation degree calculation unit 113. The estimation unit 114 estimates that the vehicle's current position is the same as its position at the time the previous camera image was acquired, when the degree of deviation is less than or equal to a predetermined value.

[0068] In this way, instead of comparing feature points, the current camera image and past camera images are compared by comparing regions, and the self-position is estimated based on the comparison results. This enables self-position estimation that is highly robust to environmental changes such as day and night. As a result, the self-position of the vehicle can be estimated with high accuracy.

[0069] (2) The deviation calculation unit 113 calculates the deviation based on the degree of overlap between the divided regions of past camera images and the divided regions of the current camera image, which are divided as regions of the same type as the divided regions. By calculating the deviation based on the positional relationship of corresponding divided regions between camera images in this way, the robustness of self-localization to environmental changes is further improved.

[0070] (3) The acquisition unit 111 continuously acquires camera images while the vehicle is moving, the deviation calculation unit 113 searches the past camera images stored in the storage unit 12 for a series of past camera images that have the smallest deviation from a series of current camera images acquired while the vehicle has traveled a predetermined distance, and the estimation unit 114 estimates that the current position of the vehicle is the same as the position of the vehicle at the time the series of past camera images were acquired, when the deviation between the series of current camera images and the series of past camera images identified by the search is less than or equal to a predetermined value. In this way, by calculating the deviation by comparing image sequences at predetermined distances, which does not depend on the vehicle's speed, the vehicle's position can be estimated with high accuracy even when the vehicle's current speed is different from the speed at which the past image sequences were acquired.

[0071] (4) The estimation unit 114 repeatedly calculates the degree of deviation by offsetting (shifting) the imaging area identified by the current camera image by a specified amount in the width direction of the vehicle with respect to the imaging area identified by past camera images, and estimates the vehicle's own position based on the smallest degree of deviation among the calculated degrees of deviation. By performing such width direction adjustment, the vehicle's own position can be estimated with high accuracy even when the vehicle's driving position in the width direction at the time the current camera image is acquired differs from the time the past camera image is acquired.

[0072] (5) Camera 1a captures images of the space located in a specific direction relative to the vehicle, specifically the space in front of the vehicle. Both the current camera image and past camera images are acquired by camera 1a. In this way, the accuracy of the self-position can be improved by estimating the self-position based on the current camera image and past camera images acquired by the same in-vehicle camera.

[0073] (6) The position estimation device 60 further includes a driving control unit 16 that performs driving control (movement control) of the vehicle. The driving control unit 16 performs driving control based on the vehicle's own position estimated by the estimation unit 114 and the stored past camera images. In this way, by reflecting the accurately estimated own position in the driving control of the vehicle, the vehicle's driving trajectory can be appropriately maintained.

[0074] (7) When the estimation unit 114 estimates its own position, it stores the current camera image if the estimated position is outside the range of past camera images stored in the memory unit 12. This allows information about roads that do not exist in the environmental map to be automatically added to the environmental map.

[0075] The above embodiment can be modified into various forms. Modifications will be described below. In the above embodiment, the acquisition unit 111 acquires sensor data (camera image) from the camera 1a as a sensor, and the division unit 112 divides the detection area (imaging area) based on the type of detection target corresponding to each pixel that makes up the camera image. However, the sensor may be something other than the camera 1a, and the acquisition unit may acquire sensor data (point cloud data) from a lidar 1b or radar 1c at a predetermined period. The division unit may classify each measurement point included in the point cloud data based on the type of detection target corresponding to each measurement point, and divide the detection area based on the result of the classification.

[0076] Furthermore, in the above embodiment, the acquisition unit 111 acquires camera images acquired by camera 1a, which detects (images) the space in front of the vehicle. However, the acquisition unit may also acquire camera images acquired by a camera mounted at a predetermined position at the rear of the vehicle, which images the space behind the vehicle, along with the camera images acquired by camera 1a. With such a configuration, in the process of step S9, not only map information corresponding to the lane the vehicle is traveling in, but also map information corresponding to the oncoming lane can be added to the environmental map. In addition, even when traveling in a direction opposite to the direction of travel when the environmental map was created, the created environmental map can be used to estimate the vehicle's own position.

[0077] Furthermore, although the above embodiment described an example where the moving object to which the position estimation device is applied is a vehicle, the moving object may be something other than a vehicle. In addition, although the above embodiment described the vehicle control device 50 and the position estimation device 60 as being applied to an autonomous vehicle, the vehicle control device 50 and the position estimation device 60 can also be applied to vehicles other than autonomous vehicles. For example, the vehicle control device 50 and the position estimation device 60 can also be applied to a manually driven vehicle equipped with ADAS (Advanced driver-assistance systems).

[0078] From another perspective, the position estimation device of the above embodiment can also be configured as a position estimation method for estimating the driving position of the vehicle on a map. That is, it can also be configured as a position estimation method that includes an acquisition step of acquiring sensor data obtained by a sensor that detects the conditions around a moving object; a division step of dividing the detection area specified by the sensor data into types of detection targets included in the detection area and generating area information indicating the position of the divided area in the detection area; a storage step of storing the past sensor data acquired in the acquisition step in a storage unit in association with the area information; a calculation step of calculating the degree of discrepancy between the current position of the divided area, specified by the area information associated with the current sensor data acquired in the acquisition step, and the past position of the divided area, specified by the area information associated with the past sensor data stored in the storage unit; and an estimation step of estimating the self-position of the moving object based on the degree of discrepancy calculated in the calculation step, wherein in the estimation step, if the degree of discrepancy is less than or equal to a predetermined value, it is estimated that the current self-position of the moving object is the same as the self-position at the time the past sensor data was acquired.

[0079] Furthermore, the present invention can be constructed by replacing the above position estimation method with a program (position estimation program) that causes a computer to perform a process to estimate the vehicle's position on a map. Even further, the present invention can be constructed by replacing this program with a computer-readable storage medium on which it is recorded.

[0080] The above description is merely an example, and the present invention is not limited by the embodiments and modifications described above, as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and to combine modifications with each other. [Explanation of symbols]

[0081] 1a Camera, 10 Controller, 11 Calculation unit, 12 Storage unit, 16 Driving control unit, 50 Vehicle control device, 60 Position estimation device, 111 Acquisition unit, 112 Segmentation unit, 113 Deviation calculation unit, 114 Estimation unit

Claims

1. An acquisition step of acquiring sensor data obtained by a sensor that detects the surrounding conditions of a moving object, A division step involves dividing the detection region identified by the sensor data based on the type of object to be detected included in the detection region, and generating region information indicating the position of the divided region within the detection region. A storage step in which the past sensor data acquired in the acquisition step is stored in the storage unit in association with the region information, A calculation step to calculate the degree of discrepancy between the current position of the divided region, which is identified by the region information associated with the current sensor data acquired in the acquisition step, and the past position of the divided region, which is identified by the region information associated with the past sensor data stored in the storage unit. The computer is instructed to perform an estimation step, which estimates the self-position of the moving object based on the degree of deviation calculated in the calculation step, The position estimation program is characterized in that, when the degree of deviation is less than or equal to a predetermined value, it is estimated that the current position of the moving object is the same as the position of the moving object at the time the past sensor data was acquired.

2. In the position estimation program described in claim 1, The aforementioned sensor data is image data acquired by a camera or point cloud data acquired by a lidar. The division step includes dividing the detection area based on the type of object to be detected corresponding to each element constituting the sensor data, A position estimation program characterized in that each of the elements is a pixel constituting the image data or a measurement point constituting the point cloud data.

3. In the position estimation program described in claim 2, A position estimation program characterized in that the calculation step includes calculating the degree of deviation based on the degree of overlap between the past divided region and the current divided region which has been divided as a region of the same type as the divided region.

4. In the position estimation program described in claim 1, The acquisition step includes continuously acquiring the sensor data while the moving object is in motion. The calculation step involves searching the storage unit for a series of past sensor data that has the smallest deviation from a series of current sensor data acquired while the moving body travels a predetermined distance, The position estimation program is characterized in that the estimation step includes estimating that the current position of the moving object is the same as the position of the moving object at the time the series of past sensor data was acquired, when the degree of deviation between the series of current sensor data and the series of past sensor data identified by the search is less than or equal to a predetermined value.

5. In the position estimation program described in claim 1, A position estimation program characterized in that the estimation step includes repeatedly calculating the degree of deviation while shifting the detection area identified by the current sensor data by a specified amount in the width direction of the moving body relative to the detection area identified by the past sensor data, and estimating the self-position of the moving body based on the smallest degree of deviation among the calculated degrees of deviation.

6. In the position estimation program described in claim 1, The sensor targets the space located in a specific direction relative to the moving object for detection. A position estimation program characterized in that the sensor from which the current sensor data was acquired is the same as the sensor from which the past sensor data was acquired.

7. Includes a position estimation program according to any one of claims 1 to 6, A vehicle control program characterized by causing the computer to further perform a step of controlling the movement of the moving body based on the self-position of the moving body estimated in the estimation step and the stored past sensor data.

8. In the vehicle control program according to claim 7, The vehicle control program is characterized in that the estimation step involves storing the current sensor data when the self-position is outside the range of stored past sensor data.

9. An acquisition unit that acquires sensor data obtained by a sensor that detects the surrounding conditions of a moving object, A division unit divides the detection region identified by the sensor data into categories according to the type of detection target included in the detection region, and generates region information indicating the position of the divided region within the detection region. A storage unit that stores past sensor data acquired by the acquisition unit in association with the region information, A calculation unit calculates the degree of discrepancy between the current position of the divided region, which is identified by the region information associated with the current sensor data acquired by the acquisition unit, and the past position of the divided region, which is identified by the region information associated with the past sensor data stored in the storage unit. The system includes an estimation unit that estimates the self-position of the moving body based on the degree of deviation calculated by the calculation unit, The position estimation device is characterized in that, when the degree of deviation is less than or equal to a predetermined value, the current position of the moving object is the same as the position of the moving object at the time the past sensor data was acquired.

Images