Road end estimation device, road end estimation method, and computer program product

Abstract

A road end estimation device, a road end estimation method, and a road end estimation computer program are provided. The road end estimation device has a travel trajectory estimation section (41) that estimates a travel trajectory of a vehicle (2), a position estimation section (42) that estimates a position of a road end portion from a plurality of images generated by a camera (11) mounted on the vehicle (2) during travel of the vehicle (2), an undetected section determination section (43) that determines an undetected section in which the vehicle travels when an image in which the road end portion is not detected is generated, a road configuration determination section (44) that determines whether a road configuration of the undetected section is a configuration having a road end portion, and a road end estimation section (45) that estimates a position of the road end portion of the undetected section from positions of road end portions of sections before and after the undetected section when the road configuration of the undetected section is the configuration having the end portion, and does not estimate the position of the road end portion of the undetected section when the road configuration of the undetected section is a configuration not having the end portion.

Description

Technical Field

[0001] The present invention relates to a road end estimation device, a road end estimation method, and a computer program for estimating the location of a road end from an image showing the road end. Background Technology

[0002] To support drivers in operating vehicles, the following technology has been proposed: using landscape image data captured by an onboard camera to obtain the road edge, thereby controlling the vehicle's direction of travel to prevent it from deviating from the road edge (see Japanese Patent Application Publication No. 2018-83563). The vehicle driving support device disclosed in Japanese Patent Application Publication No. 2018-83563, when unable to obtain the road edge, estimates the portion of the road edge that cannot be obtained within a predetermined time period based on the previous road edge obtained just before the road edge becomes unobtainable. Summary of the Invention

[0003] The road end may not always be present in locations not visible from the vehicle-mounted camera. Therefore, for locations where the road end is not shown in images generated by the vehicle-mounted camera, it is necessary to appropriately infer the location of the road end.

[0004] Therefore, the object of the present invention is to provide a road end estimation device that can accurately estimate the location of a road end in a location not visible from a camera mounted on a vehicle.

[0005] According to one embodiment, a road end estimation device is provided. The road end estimation device includes: a driving trajectory estimation unit that estimates the driving trajectory of a vehicle based on sensor signals obtained from a sensor mounted on the vehicle that detects the vehicle's behavior or the vehicle's own position obtained from a positioning device mounted on the vehicle; a position estimation unit that estimates the position of the end of the road based on an image showing the end of the road the vehicle is traveling on, which is generated from multiple images obtained during the vehicle's travel by a camera mounted on the vehicle; an interval determination unit that determines the undetected interval in which the vehicle is traveling when an image of the end of the road is not detected is generated from multiple images; a road structure determination unit that determines whether the road structure in the undetected interval has an end based on the vehicle's position when an image is generated in the undetected interval, the image generated in the undetected interval, or the vehicle's behavior in a driving trajectory corresponding to the undetected interval; and a road end estimation unit that, if the road structure in the undetected interval has an end, estimates the position of the end of the road in the undetected interval by interpolation based on the positions of the ends of the road before and after the undetected interval along the driving trajectory; on the other hand, if the road structure in the undetected interval does not have an end, it does not estimate the position of the end of the road in the undetected interval.

[0006] In this road end estimation device, the road structure determination unit preferably detects other vehicles present in the direction from the vehicle toward the end of the road before and after the undetected section from the image generated in the undetected section, and estimates the distance from the vehicle to the detected other vehicles. If the estimated distance to the other vehicles is longer than the distance from the vehicle to the end of the road before and after the undetected section, it is determined that the road structure in the undetected section is a structure without an end.

[0007] Furthermore, the road end estimation device preferably also includes a storage unit that stores map information representing the road structure of the undetected section. Preferably, when the map information shows a space where a vehicle can enter from the road where it exits from the undetected section, the road structure determination unit determines that the road structure in the undetected section is a structure without an end.

[0008] Alternatively, in this road end estimation device, it is preferable that when the vehicle is stopped for a predetermined period of time with the ignition switch on while the driving trajectory is in the undetected section or the section preceding the undetected section, the road structure determination unit determines that the road structure in the undetected section is a structure without an end.

[0009] Alternatively, in this road end estimation device, it is preferable that when the direction of travel of a vehicle in the undetected section differs from the direction of travel of a vehicle in the previous section of the undetected section by a predetermined angle or more, the road structure determination unit determines that the road structure in the undetected section is a structure without an end.

[0010] Alternatively, in the road end estimation device, if the position of the vehicle in the undetected section shown in the driving trajectory is far away from the road compared to the line connecting the ends of the road in the sections before and after the undetected section, the road structure determination unit determines that the road structure in the undetected section does not have an end.

[0011] Alternatively, in this road end estimation device, the road structure determination unit preferably determines that the road structure in the undetected section is a structure without an end when it detects a signal device, stop line or pedestrian crossing from the image generated in the undetected section.

[0012] According to other methods, a road end estimation method is provided. This road end estimation method includes: estimating the vehicle's travel trajectory based on sensor signals obtained from sensors mounted on the vehicle that detect the vehicle's behavior, or the vehicle's own position obtained from positioning devices mounted on the vehicle; estimating the position of the road end based on an image showing the end of the road traveled by the vehicle from among multiple images generated by a camera mounted on the vehicle during the vehicle's travel; determining an undetected section of the vehicle's travel when an image from among the multiple images in which the road end was not detected is generated; determining whether the road structure in the undetected section has an end based on the vehicle's position when an image was generated in the undetected section, the image generated in the undetected section, or the vehicle's behavior in a travel trajectory corresponding to the undetected section; if the road structure in the undetected section has an end, estimating the position of the road end in the undetected section along the travel trajectory by interpolation based on the positions of the road ends before and after the undetected section; conversely, if the road structure in the undetected section does not have an end, not estimating the position of the road end in the undetected section.

[0013] According to a further embodiment, a computer program for road end estimation is provided. This computer program includes commands to cause the computer to perform the following processes: estimating the vehicle's travel trajectory based on sensor signals obtained from sensors mounted on the vehicle that detect the vehicle's behavior, or the vehicle's own position obtained from a positioning device mounted on the vehicle; estimating the position of the end of the road based on an image showing the end of the road traveled by the vehicle from among multiple images generated by a camera mounted on the vehicle during the vehicle's travel; determining an undetected section of the vehicle's travel when an image from among multiple images in which the end of the road was not detected is generated; determining whether the road structure in the undetected section has an end based on the vehicle's position when an image was generated in the undetected section, the image generated in the undetected section, or the vehicle's behavior in a travel trajectory corresponding to the undetected section; if the road structure in the undetected section has an end, estimating the position of the end of the road in the undetected section along the travel trajectory by interpolation based on the positions of the ends of the road before and after the undetected section; conversely, if the road structure in the undetected section does not have an end, not estimating the position of the end of the road in the undetected section.

[0014] The road end estimation device disclosed herein has the effect of being able to accurately estimate the position of the road end in a location not visible from a camera mounted on a vehicle. Attached Figure Description

[0015] Figure 1This is a schematic diagram of a ground object data collection system that includes road-end inference devices.

[0016] Figure 2 It is a schematic diagram of the vehicle's structure.

[0017] Figure 3 This is a hardware structure diagram of the data acquisition device.

[0018] Figure 4 This is a hardware structure diagram of a server as an example of a road-side inference device.

[0019] Figure 5 This is a functional block diagram of the processor of the server related to map update processing, including road-side inference processing.

[0020] Figure 6A This is a diagram illustrating an example of the result of road end estimation when a road end exists at a location obscured by other objects.

[0021] Figure 6B This is a diagram illustrating an example of the result of road end estimation when there is no road end at a location obscured by other objects.

[0022] Figure 7 This is a flowchart of the road-side inference processing. Detailed Implementation

[0023] Hereinafter, with reference to the accompanying drawings, a road end estimation device, a road end estimation method executed by the road end estimation device, and a computer program for road end estimation will be described. The road end estimation device estimates the vehicle's trajectory over a predetermined road section in which the vehicle travels, and within this road section, detects the ends of the road from a series of images representing the vehicle's surroundings generated by a camera mounted on the vehicle, thereby estimating the position of the road ends. At this time, the road end estimation device determines the vehicle's travel section where an image of a road end that was not detected is generated as an undetected section. Furthermore, the road end estimation device determines whether the road structure in the undetected section has an end based on the image of the road end that was not detected, the vehicle's position when that image was generated, or the behavior of the vehicle in the determined undetected section. If it is determined that the road structure in the undetected section has an end, the road end estimation device estimates the position of the road end in the undetected section along the vehicle's trajectory by interpolation based on the positions of the road ends before and after the undetected section. On the other hand, if the road structure in the undetected section is determined to be a structure without an end, the road end estimation device does not estimate the position of the end of the road in the undetected section, but directly considers it as having no end.

[0024] The following describes an example of applying a road-end inference device to a ground object data collection system that collects ground object data obtained during vehicle movement. This ground object data represents road ends, various road signs, signaling equipment, and ground objects associated with the movement of other vehicles. The collected ground object data is used to generate or update map information containing information related to ground objects associated with vehicle movement. However, the road-end inference device can also be used in systems other than ground object data collection systems. For example, the road-end inference device can also be applied to vehicle control systems that perform vehicle driving support or automatic driving control.

[0025] Figure 1 This is a schematic structural diagram of a ground object data collection system equipped with a road-end estimation device. In this embodiment, the ground object data collection system 1 includes at least one vehicle 2 and a server 3, which is an example of a road-end estimation device. Each vehicle 2 connects to the server 3, for example, by accessing a wireless base station 5 connected to a communication network 4 connected to the server 3 via a gateway (not shown), etc., thereby connecting via the wireless base station 5 and the communication network 4. Furthermore, in Figure 1 For simplicity, only one vehicle 2 is illustrated, but the ground object data collection system 1 can also have multiple vehicles 2. Similarly, in Figure 1 The diagram only shows one wireless base station 5, but there could also be multiple wireless base stations 5 connected to the communication network 4.

[0026] Figure 2 This is a schematic structural diagram of vehicle 2. Vehicle 2 includes a camera 11, a GPS receiver 12, at least one vehicle motion sensor 13, a wireless communication terminal 14, and a data acquisition device 15. The camera 11, GPS receiver 12, vehicle motion sensor 13, wireless communication terminal 14, and data acquisition device 15 are communicatively connected via an in-vehicle network according to a standard controller area network (CLAN). Furthermore, the vehicle motion sensor 13 can also be connected to an electronic control unit (ECU, not shown) that controls the driving of vehicle 2. Additionally, vehicle 2 may also include a navigation device (not shown) that searches for a predetermined driving route for vehicle 2 and navigates vehicle 2 along that route.

[0027] Camera 11 is an example of an imaging unit used to photograph the area around vehicle 2. It has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as CCD or C-MOS, and an imaging optical system that images the predetermined area to be photographed onto the two-dimensional detector. Camera 11 is mounted inside the vehicle compartment, for example, facing the front, rear, or side of vehicle 2. Camera 11 photographs the predetermined area around vehicle 2 at a predetermined shooting cycle (e.g., 1 / 30 second to 1 / 10 second), generating an image of the predetermined area. The image obtained by camera 11 can be either a color image or a grayscale image. Furthermore, multiple cameras 11 with different shooting directions or focal lengths can be arranged in vehicle 2.

[0028] Whenever a camera 11 generates an image, it outputs the generated image to the data acquisition device 15 via the in-vehicle network.

[0029] GPS receiver 12 is an example of a positioning device. It receives GPS signals from GPS satellites at predetermined intervals and positions the vehicle 2 based on the received GPS signals. Furthermore, GPS receiver 12 outputs positioning information, representing the vehicle 2's position based on the GPS signals, to data acquisition device 15 via an in-vehicle network at predetermined intervals. Alternatively, vehicle 2 may have a receiver based on a satellite positioning system other than GPS receiver 12. In this case, that receiver can be used to position vehicle 2.

[0030] Vehicle motion sensor 13 is an example of a sensor that detects the behavior of vehicle 2, acquires information related to the behavior of vehicle 2 (hereinafter, sometimes referred to as vehicle behavior information), and generates sensor signals representing the vehicle behavior information. Vehicle behavior information includes, for example, wheel speed, the angular velocities of the three mutually orthogonal axes of vehicle 2, or the acceleration of vehicle 2. Vehicle motion sensor 13 includes, for example, at least one of a wheel speed sensor for measuring the wheel speed of vehicle 2, a gyroscope sensor for measuring the angular velocities of the three mutually orthogonal axes of vehicle 2, and an acceleration sensor for measuring the acceleration of vehicle 2. Vehicle motion sensor 13 generates sensor signals according to a predetermined period and outputs the generated sensor signals to data acquisition device 15 via an in-vehicle network. Furthermore, when vehicle motion sensor 13 is connected to an ECU, the sensor signals generated by vehicle motion sensor 13 are output to data acquisition device 15 via the ECU and the in-vehicle network.

[0031] The wireless communication terminal 14 is an example of a communications unit. It is a device that performs wireless communication processing according to a predetermined wireless communication standard. For example, it connects to the server 3 via the wireless base station 5 and the communication network 4 by accessing the wireless base station 5. The wireless communication terminal 14 generates an uplink wireless signal including ground object data and driving information received from the data acquisition device 15. The wireless communication terminal 14 transmits the ground object data and driving information to the server 3 by sending the uplink wireless signal to the wireless base station 5. Additionally, the wireless communication terminal 14 receives downlink wireless signals from the wireless base station 5 and transmits collection instructions from the server 3 contained in the wireless signals to the data acquisition device 15 or the ECU.

[0032] Figure 3 This is a hardware structure diagram of the data acquisition device 15. The data acquisition device 15 generates ground object data based on images generated by the camera 11. Furthermore, the data acquisition device 15 generates driving information representing the driving behavior of the vehicle 2. For this purpose, the data acquisition device 15 includes a communication interface 21, a memory 22, and a processor 23.

[0033] Communication interface 21 is an example of an in-vehicle communication unit, having an interface circuit for connecting the data acquisition device 15 to the in-vehicle network. Specifically, communication interface 21 connects to camera 11, GPS receiver 12, vehicle motion sensor 13, and wireless communication terminal 14 via the in-vehicle network. Furthermore, whenever communication interface 21 receives an image from camera 11, it transmits the received image to processor 23. Additionally, whenever communication interface 21 receives positioning information from GPS receiver 12, it transmits the received positioning information to processor 23. Moreover, whenever communication interface 21 receives a sensor signal from vehicle motion sensor 13 or ECU, it transmits the received sensor signal to processor 23. Furthermore, communication interface 21 transmits the collection instructions for ground object data received from server 3 via wireless communication terminal 14 to processor 23. Additionally, communication interface 21 outputs ground object data and driving information received from processor 23 to wireless communication terminal 14 via the in-vehicle network.

[0034] The memory 22 may include, for example, volatile semiconductor memory and non-volatile semiconductor memory. The memory 22 may also include other storage devices such as a hard disk drive. Furthermore, the memory 22 stores various data used in the processing associated with the generation of ground object data and driving information, executed by the processor 23 of the data acquisition device 15. Such data includes, for example, parameters of the camera 11 such as vehicle 2 identification information, camera 11 setting height, shooting direction, and field of view. Additionally, the memory 22 may store images received from the camera 11, positioning information received from the GPS receiver 12, and vehicle behavior information contained in sensor signals generated by the vehicle motion sensor 13 for a certain period. Moreover, the memory 22 stores information indicating the area (hereinafter sometimes referred to as the collection target area) designated by the ground object data collection instruction as the object of ground object data generation and collection. Furthermore, the memory 22 may also store computer programs for implementing the various processes executed by the processor 23.

[0035] The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuitry. The processor 23 may also include other arithmetic circuits such as logic operation units, numerical operation units, or graphics processing units. Furthermore, the processor 23 stores images received from the camera 11, positioning information received from the GPS receiver 12, and vehicle behavior information contained in sensor signals received from the vehicle motion sensor 13 or the ECU into the memory 22. Moreover, during the movement of the vehicle 2, the processor 23 performs processing related to the generation of ground object data and driving information at predetermined intervals (e.g., 0.1 seconds to 10 seconds).

[0036] In processor 23, as part of the processing associated with the generation of ground object data, it is determined, for example, whether the vehicle position of vehicle 2, as indicated by the positioning information received from GPS receiver 12, is included in the target area. Then, if the vehicle position is included in the target area, processor 23 generates ground object data based on the image received from camera 11.

[0037] Ground object data refers to data representing ground objects associated with the vehicle's movement. In this embodiment, the processor 23 makes the ground object data include images generated by the camera 11, the time when the image was generated, the vehicle 2's direction of travel at that time, and camera 11 parameters such as the camera 11's setting height, shooting direction, and field of view. Furthermore, the processor 23 can obtain information representing the vehicle 2's direction of travel from the vehicle 2's ECU. Moreover, whenever ground object data is generated, the processor 23 transmits the generated ground object data to the server 3 via the wireless communication terminal 14. Additionally, the processor 23 can also make a single ground object data set include multiple images, the generation time of each image, and the vehicle 2's direction of travel. Furthermore, the processor 23 can also transmit the camera 11 parameters and the ground object data independently to the server 3 via the wireless communication terminal 14.

[0038] Furthermore, the processor 23 generates vehicle 2's driving information after a predetermined timing (e.g., the timing when the ignition switch of vehicle 2 is turned on), and transmits this driving information to the server 3 via the wireless communication terminal 14. The processor 23 includes in the driving information a series of position measurement information obtained at a certain period after the predetermined timing, the time when the position of vehicle 2 is measured in each position measurement information, and vehicle behavior information such as wheel speed, acceleration, and angular velocity. The processor 23 can also include in the driving information information the timing of the ignition switch being turned on or off, obtained from the ECU. Furthermore, the processor 23 can include vehicle 2's identification information in the driving information and ground object data.

[0039] Next, we will describe server 3 as an example of a road-side inference device.

[0040] Figure 4 This is a hardware structure diagram of server 3, an example of a road-end prediction device. Server 3 has a communication interface 31, a storage device 32, a memory 33, and a processor 34. The communication interface 31, storage device 32, and memory 33 are connected to the processor 34 via signal lines. Server 3 may also have input devices such as a keyboard and mouse, and a display device such as an LCD.

[0041] Communication interface 31 is an example of a communication unit, having interface circuitry for connecting server 3 to communication network 4. Furthermore, communication interface 31 is configured to communicate with vehicle 2 via communication network 4 and wireless base station 5. Specifically, communication interface 31 transmits ground object data and driving information received from vehicle 2 via wireless base station 5 and communication network 4 to processor 34. Additionally, communication interface 31 transmits collection instructions received from processor 34 to vehicle 2 via communication network 4 and wireless base station 5.

[0042] Storage device 32 is an example of a storage unit, such as having a hard disk drive or an optical recording medium and its access device. Storage device 32 stores various data and information used in road end prediction processing. For example, storage device 32 stores parameter sets used to determine the identifiers for detecting road ends and ground objects from images. Additionally, storage device 32 may store navigation maps used by the navigation device to search for driving routes. Furthermore, storage device 32 stores ground object data and driving information received from each vehicle 2. Moreover, storage device 32 may also store a computer program executed on processor 34 for performing road end prediction processing.

[0043] Memory 33 is another example of a storage unit, such as a non-volatile semiconductor memory and a volatile semiconductor memory. Furthermore, memory 33 temporarily stores various data generated during the execution of the road-end speculation processing.

[0044] Processor 34 is an example of a control unit, having one or more CPUs (Central Processing Units) and their peripheral circuitry. Processor 34 may also include other computational circuitry such as logic units or numerical processing units. Furthermore, processor 34 performs map update processing, including road-side prediction processing.

[0045] Figure 5 This is a functional block diagram of the processor 34 associated with map update processing, including road end estimation processing. The processor 34 includes a trajectory estimation unit 41, a position estimation unit 42, a section determination unit 43, a road structure determination unit 44, a road end estimation unit 45, and a map update unit 46. These units of the processor 34 are functional modules implemented, for example, by a computer program operating on the processor 34. Alternatively, these units of the processor 34 may be dedicated arithmetic circuits provided on the processor 34. Furthermore, the processing of each of these units of the processor 34, except for the map update unit 46, is associated with the road end estimation processing.

[0046] The trajectory estimation unit 41 estimates the trajectory of vehicle 2 when it travels within a predetermined interval based on the vehicle 2's travel information within that interval. Hereinafter, the trajectory of vehicle 2 will sometimes be referred to simply as the travel trajectory.

[0047] For example, the trajectory estimation unit 41 arranges the positions of the vehicles 2 shown by their positioning information in order from earliest to latest, according to the time when the positions of each vehicle 2 were measured in a series of positioning information contained in the vehicle 2's driving information, thereby enabling it to estimate the driving trajectory. Furthermore, the trajectory estimation unit 41 can also include the measurement time of each vehicle 2's position in the estimated trajectory. Moreover, when the received driving information includes the timing of when the ignition switch is turned on or off, the trajectory estimation unit 41 can also include information determining the position of the vehicle 2 at the timing when the ignition switch is turned on or off.

[0048] Alternatively, the trajectory estimation unit 41 may also estimate the trajectory of vehicle 2 using the so-called Structure from Motion (SfM) method. In this case, the trajectory estimation unit 41 detects one or more ground objects around vehicle 2 from each image in a series of images generated by the camera 11 of vehicle 2 traveling in a predetermined section, which includes ground object data received from vehicle 2.

[0049] For example, the trajectory estimation unit 41 detects ground objects shown in the input image (hereinafter sometimes simply referred to as the input image) by inputting a series of images into a recognizer that has been pre-learned to detect ground objects that are the objects to be detected. As such a recognizer, the trajectory estimation unit 41 can, for example, use a deep neural network (DNN) that has been pre-learned to detect ground objects shown in the input image. Such a DNN could be, for example, a Single Shot MultiBox Detector (SSD) or a Faster R-CNN with a convolutional neural network (CNN) architecture. Alternatively, a DNN with a self-attention network architecture, such as a Vision Transformer, could also be used. In this case, the recognizer calculates the confidence level of the accuracy of the presence of a ground object in various regions of the input image for each type of ground object that is the object to be detected (e.g., traffic signal equipment, lane markings, pedestrian crossings, temporary stop lines, etc.). The recognizer determines that a ground object of a certain type has appeared in a region where the confidence level regarding any type of ground object is above a predetermined detection threshold. Furthermore, the recognizer outputs information indicating the region on the input image containing the ground object that is being detected (e.g., the bounding rectangle of the ground object being detected, hereinafter referred to as the object region), and information indicating the type of ground object appearing in the object region.

[0050] The trajectory estimation unit 41 estimates the position of the vehicle 2 when each image is generated, as well as the relative positions of the detected ground objects corresponding to these positions, using the SfM method. Furthermore, the trajectory estimation unit 41 estimates the driving trajectory based on its estimation results.

[0051] Here, the position of each pixel in the image corresponds one-to-one with the orientation from the camera 11 toward the object shown in that pixel. Therefore, the driving trajectory estimation unit 41 can estimate the relative positional relationship between the ground object and the vehicle 2 based on the orientation from the camera 11 corresponding to the feature points representing ground objects in each image, the amount of movement of the vehicle 2 between the generation times of each image, the direction of travel of the vehicle 2, and the parameters of the camera 11.

[0052] The trajectory estimation unit 41 can set the initial position of vehicle 2 in the trajectory as the vehicle's position represented by the positioning information of the GPS receiver 12 at a predetermined time, which includes the driving information. Then, the trajectory estimation unit 41 uses vehicle behavior information such as wheel speed, acceleration, and angular velocity included in the driving information to estimate the amount of movement of vehicle 2 between the time of image generation.

[0053] The trajectory estimation unit 41 correlates the same ground objects detected in multiple images obtained at different time points during the movement of the vehicle 2. At this time, the trajectory estimation unit 41, for example, by using an optical flow tracking method, can correlate feature points contained in object regions showing the same ground object of interest across multiple images. Furthermore, the trajectory estimation unit 41 can estimate the relative position of the ground object of interest corresponding to the position of the vehicle 2 at the time each image is generated, as well as the position of the vehicle 2 at the time each image is generated, through triangulation. The trajectory estimation unit 41 utilizes the vehicle 2's direction of travel at the time each image is generated, the vehicle 2's position at the time any image is generated, the amount of movement of the vehicle 2 between the time each image is generated, the parameters of the camera 11, and the positions of the corresponding feature points in each image using this triangulation.

[0054] The trajectory estimation unit 41, by repeatedly performing the above processing on multiple detected ground objects, can sequentially estimate the position of the vehicle 2 at the time each image is generated, as well as the relative positions of the ground objects surrounding the vehicle 2 corresponding to these positions. Furthermore, the trajectory estimation unit 41 can estimate the trajectory by sequentially arranging the estimated positions of the vehicle 2. In this case, the trajectory estimation unit 41 can also include the image generation time for each position of the vehicle 2 as the time when the vehicle 2 passes that position in the estimated trajectory.

[0055] The trajectory prediction unit 41 notifies the location prediction unit 42, the section determination unit 43, the road structure determination unit 44, and the road end prediction unit 45 of the information indicating the predicted trajectory.

[0056] The position estimation unit 42 detects the ends of the road within a predetermined interval from each of a series of images generated by the camera 11 during the period when the vehicle 2 travels within the predetermined interval, and estimates the position of the detected road ends in actual space. The position estimation unit 42 can perform the same processing for each image, so the processing for a single image will be described below.

[0057] The location estimation unit 42 detects the ends of the road shown in the input image by inputting the image to a recognizer that has been pre-learned to detect the ends of the road. The location estimation unit 42 can utilize the same recognizer used for ground object detection as described with respect to the driving trajectory estimation unit 41. Alternatively, the location estimation unit 42 can also use a fully convolutional network or a DNN such as U-net, which performs semantic segmentation for each pixel, recognizing the objects shown in that pixel. In this case, the recognizer is pre-learned to identify each pixel of the image as either a pixel representing the outside of the road or a pixel representing the road. The location estimation unit 42 then determines that the ends of the road are shown in the pixels located at the outer edge of the set of pixels representing the road output by the recognizer.

[0058] The position estimation unit 42 estimates the actual spatial position of the end of the road shown in each pixel based on the position of that pixel in the image, the position and direction of travel of the vehicle 2 when the image was generated, and the parameters of the camera 11. Furthermore, the position estimation unit 42 obtains the position and direction of travel of the vehicle 2 when the image was generated from the trajectory estimation unit 41.

[0059] The position estimation unit 42 can, as described above, estimate the position of the end of the road shown in each pixel, or it can estimate the position of the end of the road shown in a pixel only for a few pixels among those pixels showing the end of the road. Alternatively, the position estimation unit 42 can estimate the position of the end of one end of the road on which the vehicle 2 is traveling. For example, the position estimation unit 42 can estimate the position of the end of the road on the side where the vehicle 2 can travel within a predetermined section. That is, if the road in the predetermined section is a left-hand road relative to the vehicle 2, the position estimation unit 42 estimates the position of the end of the road on the left. In this case, the position estimation unit 42 can estimate the position of the end of the road using the pixels in the set of pixels showing the end of the road that are adjacent to the side closest to the vehicle 2 and form the road surface. Alternatively, the position estimation unit 42 can estimate the position of the end of the road using the pixels in the set of pixels showing the end of the road that are located on the left side of the pixel column corresponding to the direction of travel of the vehicle 2, as determined by the mounting position and shooting direction of the camera 11.

[0060] The location estimation unit 42 notifies the road end estimation unit 45 and the map update unit 46 of the estimated location of each image showing the end of the road. Additionally, the location estimation unit 42 notifies the interval determination unit 43 of the generation time of each image where the end of the road was not detected.

[0061] The interval determination unit 43 determines the travel interval of vehicle 2 as an undetected interval when an image of the end of the road not being detected is generated, among a series of multiple images generated by camera 11 during the period when vehicle 2 travels within a predetermined interval. To this end, the interval determination unit 43 determines the position of vehicle 2 at the generation time of each image of the end of the road not being detected, by referring to the generation time of each image of the end of the road not being detected and the travel information notified from position estimation unit 42. Furthermore, in the travel trajectory estimated by travel trajectory estimation unit 41, the interval determination unit 43 defines the interval including the set of determined vehicle 2 positions as the undetected interval.

[0062] The section determination unit 43 notifies the road structure determination unit 44 and the road end estimation unit 45 of the information indicating the undetected section.

[0063] The road structure determination unit 44 determines whether the road structure in the undetected section where vehicle 2 travels is a structure with an end. Here, a road structure with an end refers to a road structure that is divided into a drivable section and a non-drivable section, and has a boundary for the drivable section. Conversely, a road structure without an end or without an end refers to a structure that has a space, such as an intersection, railway crossing, or entrance to private land like a parking lot, allowing vehicles to enter outside the road in a direction intersecting with its extension. Furthermore, a structure without an end may include a lane for specific types of vehicles, such as a bus lane, located further from the end of the road than a lane for general vehicles. Therefore, in a section where the road structure is without an end, there may be other objects in a direction away from the road compared to the sections before and after it. Alternatively, in such a section, vehicle 2 itself may move in a direction away from the road compared to the end of the road in the sections before and after it. Furthermore, in sections of the road that lack end points, there may be spaces on navigation maps where vehicles 2 can enter and exit the road they are traveling on, such as intersections or railway crossings. Therefore, the road structure determination unit 44 uses the location of the undetected section, other objects shown in the image generated by the camera 11 in the undetected section, or the behavior of the vehicle 2 while traveling in the undetected section to determine whether there is an end point of the road. In addition, the road in which the vehicle 2 travels in the undetected section will sometimes be referred to as the road in the undetected section.

[0064] For example, the road structure determination unit 44 refers to the location of the undetected section and the navigation map. Furthermore, if the navigation map shows a space in the undetected section where the vehicle 2 can enter from outside the road, such as an intersection, the road structure determination unit 44 determines that the road structure in the undetected section is a structure without an end.

[0065] Furthermore, if the position of vehicle 2 in the undetected section, as indicated by the vehicle 2's travel trajectory, is far from the road compared to the line connecting the ends of the road in the sections before and after it, the road structure determination unit 44 determines that the road structure in the undetected section lacks an end. Alternatively, the road structure determination unit 44 may also determine the road structure in the undetected section based on the changes in the travel direction of vehicles 2 before and after the undetected section, referring to the vehicle 2's travel trajectory. For example, if the travel direction of vehicle 2 in the section preceding the undetected section changes by a predetermined angle or more (e.g., 45° or more) compared to the travel direction of vehicle 2 in the undetected section, there is a high probability that vehicle 2 will turn right or left at an intersection, or move from the previously traveled road to the roadside. Therefore, in such cases, the road structure determination unit 44 determines that the road structure in the undetected section lacks an end. Additionally, if the undetected section includes an intersection, vehicle 2 may sometimes stop very close to the intersection. Therefore, if the driving trajectory indicates that vehicle 2 has been stopped for a predetermined period of time with the ignition switch on in an undetected section or the preceding section, the road structure determination unit 44 can also determine that the road structure in the undetected section is a structure without an end. Furthermore, if multiple measurement times are associated with a single position of vehicle 2 within the driving trajectory during the predetermined period, the road structure determination unit 44 can determine that vehicle 2 has been stopped at that position for the predetermined period. Additionally, if the undetected section includes an intersection, there may be traffic signals, stop lines, or pedestrian crossings in or before / after the undetected section. Therefore, the road structure determination unit 44 can also determine that the road structure in the undetected section is a structure without an end if it detects traffic signals, stop lines, or pedestrian crossings in the image generated by camera 11 while vehicle 2 is traveling in the undetected section. In this case, the road structure determination unit 44 detects traffic signals, stop lines, or pedestrian crossings by inputting the image to the recognizer. As such a recognizer, the road structure determination unit 44 can use the same recognizer as the ground object detection recognizer described with respect to the driving trajectory estimation unit 41. Alternatively, the recognizer used by the driving trajectory estimation unit 41 or the recognizer used by the position estimation unit 42 can also be pre-learned in a way that also detects signal equipment, stop lines, or pedestrian crossings.

[0066] Furthermore, the road structure determination unit 44 can also determine the road structure in the undetected section by comparing the positions of other vehicles detected from the image generated by the camera 11 when vehicle 2 is traveling in the undetected section with the positions of the ends of the road in the sections before and after it. In this case, the road structure determination unit 44 detects other vehicles by inputting the image into the recognizer. As such a recognizer, the road structure determination unit 44 can use the same recognizer as described above. Alternatively, the aforementioned recognizer, the recognizer used by the trajectory estimation unit 41, or the recognizer used by the position estimation unit 42 can also be pre-learned in a manner that also detects other vehicles. It is envisioned that the position of the lower end of the area showing other vehicles in the image represents the position where other vehicles are in contact with the road surface. In addition, each pixel in the image corresponds one-to-one with the orientation observed from the camera 11. Therefore, the road structure determination unit 44 can estimate the orientation and distance from vehicle 2 to other vehicles based on parameters such as the position of the lower end of the area showing other vehicles in the image, the position and direction of travel of vehicle 2, and the setting height and shooting direction of the camera 11. Therefore, the road structure determination unit 44 can estimate the distance from vehicle 2 to other vehicles in a direction orthogonal to the extension direction of the road on which vehicle 2 travels (hereinafter, for ease of explanation, it is sometimes referred to as the lateral distance). Furthermore, the road structure determination unit 44 compares the lateral distance from vehicle 2 to other vehicles with the lateral distance from vehicle 2 to the end of the road in any interval before or after the undetected interval. If the lateral distance from vehicle 2 to other vehicles is longer than the lateral distance from vehicle 2 to the end of the road in any interval before or after the undetected interval, the road structure determination unit 44 determines that the road structure in the undetected interval is a structure without an end.

[0067] If none of the judgment criteria described above for determining that a road has no end are met, the road structure determination unit 44 can determine that the road structure on which vehicle 2 travels in the undetected section has an end. For example, if the lateral distance from vehicle 2 to other vehicles is shorter than the lateral distance from vehicle 2 to the end of the road in any interval before or after the undetected section, it is assumed that other vehicles are parked near the end of the road, thus obscuring the end of the road. Therefore, in such a case, the road structure determination unit 44 can determine that the road structure on which vehicle 2 travels in the undetected section has an end. Furthermore, the road structure determination unit 44 can also refer to the travel trajectory of vehicle 2, and determine that the road structure on which vehicle 2 travels in the undetected section has an end when vehicle 2 stops near other parked vehicles as described above. Furthermore, if vehicle 2 stops within a certain distance from the undetected section and the ignition switch is turned off, and the angle difference between the vehicle 2's direction of travel and the road's extension direction is less than a predetermined angle, the road structure determination unit 44 determines that vehicle 2 is parked near other parked vehicles. Additionally, if vehicle 2's trajectory shows behavior that contradicts the determination result regarding the presence or absence of the end of the road in the undetected section, the road structure determination unit 44 can also determine the presence or absence of the end of the road in the undetected section based on the aforementioned determination conditions that do not refer to the travel trajectory.

[0068] The road structure determination unit 44 notifies the road end estimation unit 45 of the determination result of the road structure in the undetected section.

[0069] When the road structure in the undetected section has an end, the road end estimation unit 45 estimates the position of the end of the road in the undetected section by interpolation based on the positions of the ends of the road in the sections before and after the undetected section, along the travel trajectory. In this case, the road end estimation unit 45 estimates the position of the end of the road in the undetected section by interpolating by connecting the positions of the ends of the road in the sections before and after the undetected section with a predetermined curve or straight line. As the predetermined curve used in the interpolation, the road end estimation unit 45 can use, for example, a clothoid curve or a spline curve. In particular, when the undetected section is a curved section, the curve is sometimes designed along a clothoid curve. Therefore, by using a clothoid curve as the predetermined curve used in the interpolation, the road end estimation unit 45 can estimate the position of the end of the road in the undetected section with high accuracy. The road end estimation unit 45 can also add a mark indicating that the position of the road end estimated by interpolation is estimated by interpolation.

[0070] On the other hand, if the road structure in the undetected section is one without an end, the road end estimation unit 45 does not estimate the position of the road end in the undetected section. Therefore, the road end estimation unit 45 can prevent incorrect estimation of the road end position in sections where there is originally no road end.

[0071] Figure 6A This is an example diagram showing the result of road end estimation when a road end exists in a location obscured by other objects. Additionally, Figure 6B This is a diagram illustrating an example of the result of road end estimation when the road end is not present at a location obscured by other objects. Figure 6A In the example shown, images are generated by camera 11 at positions corresponding to times t0 to t5 along the travel trajectory 601 of vehicle 2 traveling on road 600. Then, the positions r0 to r3 and r5 of the ends 602 of road 600 are detected from images generated at times other than t4. However, in the image generated at time t4, the ends 602 of road 600 are obscured by the parked vehicle 610, so they cannot be detected. Therefore, the position of vehicle 2 at time t4 becomes the undetected interval 603. In this example, the distance d1 from the parked vehicle 610 to the travel trajectory 601 (i.e., the distance from the parked vehicle 610 to vehicle 2) is shorter than the distance d2 from the travel trajectory 601 to the ends 602 of road 600 in the intervals before and after the undetected interval 603. Therefore, the structure of road 600 in the undetected interval 603 is determined to be a structure with ends. Therefore, the position r4 of the end 602 of the road 600 in the undetected interval 603 is inferred by interpolation of the positions r3 and r5 of the end 602 in the intervals before and after it.

[0072] exist Figure 6BIn the example shown, images are generated by camera 11 at positions corresponding to times t0 to t5 along the travel trajectory 601 of vehicle 2 traveling on road 600. Then, the positions r0 to r3 and r5 of the end 602 of road 600 are detected from images generated at times other than t4. However, the camera 11's field of view at time t4 includes intersections, so the end 602 of road 600 is not reflected in the image generated at time t4. Therefore, the position of vehicle 2 at time t4 becomes the undetected interval 604. However, in this example, other temporarily parked vehicles 620 reflected in the image generated at time t4 are parked on other roads intersecting with road 600. Therefore, the distance d3 between the other vehicles 620 detected from the image generated at time t4 and the travel trajectory 601 is longer than the distance d2 from the travel trajectory 601 to the end 602 of road 600 in the intervals before and after the undetected interval 604. As a result, the road 600 in the undetected interval 604 is determined to be a structure without ends. Therefore, in the undetected interval 604, the position of the end 602 of the road 600 is not inferred by interpolation, but remains undetected.

[0073] The road end estimation unit 45 notifies the map update unit 46 of the estimation results of the location of the road end in the undetected section or information indicating that no interpolation has been performed.

[0074] The map update unit 46 adds or rewrites the map information to be generated or updated by inferring or interpolating the positions of the ends of each road within a predetermined interval. Furthermore, for undetected intervals where the road structure is determined to be without ends, the map update unit 46 does not add the positions of the road ends to the map information. This prevents the erroneous addition of road end positions to the map information even in intervals without road ends. Moreover, the map update unit 46 can also add or rewrite the map information to the map information based on the types and locations of ground objects detected within the predetermined interval.

[0075] Figure 7 This is the flowchart of the road-side inference processing in server 3. When the processor 34 of server 3 receives ground object data and driving information from vehicle 2 in a predetermined section of the collection object area, it performs road-side inference processing according to the flowchart shown below.

[0076] The trajectory estimation unit 41 of the processor 34 estimates the trajectory of the vehicle 2 within a predetermined interval (step S101). Additionally, the position estimation unit 42 of the processor 34 detects the ends of roads within the predetermined interval from each of a series of images generated by the camera 11 during the period the vehicle 2 travels within the predetermined interval. Then, the position estimation unit 42 estimates the position of the detected ends from the images (step S102). Furthermore, the interval determination unit 43 of the processor 34 determines the travel interval of the vehicle 2 when an image without detected road ends is generated as an undetected interval (step S103).

[0077] The road construction determination unit 44 of the processor 34 determines whether the road construction in the undetected section is a construction with ends (step S104). If the road construction in the undetected section is a construction with ends (step S104: "Yes"), the road end estimation unit 45 of the processor 34 performs interpolation processing based on the position of the road ends in the sections before and after the undetected section along the driving trajectory. As a result, the road end estimation unit 45 estimates the position of the road ends in the undetected section (step S105).

[0078] On the other hand, if the road structure in the undetected section is a structure without an end (step S104: "No"), the road end estimation unit 45 does not estimate the position of the road end in the undetected section and still considers it as having no road end (step S106). After step S105 or S106, the processor 34 ends the road end estimation process.

[0079] As explained above, this road end estimation device defines the vehicle's travel section when an image of a road end that was not detected is generated as an undetected section. Furthermore, the road end estimation device determines whether the road structure in the undetected section has an end based on the image of the undetected road end, the vehicle's position when the image was generated, or the behavior of the vehicle in the undetected section. If the road structure in the undetected section has an end, the road end estimation device estimates the position of the road end in the undetected section by interpolating based on the positions of the road ends preceding and following the travel section, along the vehicle's travel trajectory. Conversely, if the road structure in the undetected section does not have an end, the road end estimation device does not estimate the position of the road end. Therefore, this road end estimation device can prevent incorrectly estimated road end positions for locations where there are no road ends. Furthermore, regarding the road ends in sections that are not shown in the image for some reason, the road end estimation device can appropriately estimate their end positions based on the positions of the road ends in sections preceding and following that section. Therefore, this road end estimation device can accurately estimate the location of the road end in places that are not visible from the camera mounted on the vehicle.

[0080] According to a variation, the processor 23 of the data acquisition device 15 can also perform road end prediction processing instead of the processor 34 of the server 3. In this case, the data acquisition device 15 can send information indicating the prediction result of the road end to the server 3 via the wireless communication terminal 14. Alternatively, the road end prediction processing can be performed by the ECU of the vehicle 2. In this case, the ECU can control the vehicle 2 to maintain a certain distance from the road end based on the predicted location of the road end.

[0081] Furthermore, the computer program that enables the computer to implement the functions of the processor of the road end prediction device according to the above embodiments or modifications can also be provided in the form of a computer-readable recording medium. In addition, the computer-readable recording medium may be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0082] As described above, those skilled in the art can make various modifications within the scope of this invention in combination with the implemented methods.

Claims

1. A road end prediction device, comprising: The trajectory estimation unit estimates the vehicle's trajectory based on sensor signals obtained from sensors mounted on the vehicle that detect the vehicle's behavior or the vehicle's own position obtained from a positioning device mounted on the vehicle. The position estimation unit estimates the position of the end of the road based on an image showing the end of the road the vehicle is traveling on, which is one of multiple images generated by a camera mounted on the vehicle during the vehicle's travel. The interval determination unit, when generating an image from among the plurality of images in which the end of the road was not detected, determines the undetected interval in which the vehicle is traveling; The road structure determination unit determines, based on the vehicle's position when the image was generated in the undetected section, the image generated in the undetected section, or the vehicle's behavior in a driving trajectory corresponding to the undetected section, whether the road structure in the undetected section has an end; and The road end estimation unit, when the road in the undetected section is constructed with ends, estimates the position of the end of the road in the undetected section along the travel trajectory by interpolation based on the positions of the ends of the road before and after the undetected section. On the other hand, when the road in the undetected section is constructed without ends, it does not estimate the position of the end of the road in the undetected section.

2. The road end prediction device according to claim 1, wherein, The road structure determination unit detects other vehicles present before and after the vehicle in the direction from the vehicle toward the end of the road from the image generated in the undetected section, and estimates the distance from the vehicle to the detected other vehicles. If the estimated distance to the other vehicles is longer than the distance from the vehicle to the end of the road before and after the undetected section, it determines that the road structure in the undetected section is a structure without an end.

3. The road end prediction device according to claim 1, wherein, The road-end estimation device also includes a storage unit that stores map information representing the road structure of the undetected section. When the map information shows a space where the vehicle can enter a location where it can detach from the road in the undetected section, the road structure determination unit determines that the road structure in the undetected section is a structure without an end.

4. The road end prediction device according to claim 1, wherein, When the driving trajectory indicates that the vehicle has been parked for a predetermined period of time with the ignition switch on in the undetected section or the section preceding the undetected section, the road structure determination unit determines that the road structure in the undetected section is a structure without an end.

5. The road end prediction device according to claim 1, wherein, If the direction of travel of the vehicle in the undetected section differs from the direction of travel of the vehicle in the previous section of the undetected section by a predetermined angle or more, the road structure determination unit determines that the road structure in the undetected section is a structure without ends.

6. The road end prediction device according to claim 1, wherein, If the position of the vehicle in the undetected section shown in the driving trajectory is far away from the road compared to the line connecting the ends of the road in the sections before and after the undetected section, the road structure determination unit determines that the road structure in the undetected section is a structure without ends.

7. The road end prediction device according to claim 1, wherein, If the road structure determination unit detects a signal device, stop line, or pedestrian crossing in the image generated in the undetected section, it determines that the road structure in the undetected section is a structure without ends.

8. A method for road end estimation, comprising: The vehicle's trajectory is inferred based on sensor signals obtained from sensors mounted on the vehicle that detect the vehicle's behavior or the vehicle's own position obtained from positioning devices mounted on the vehicle. Based on an image showing the end of the road on which the vehicle is traveling, generated by a camera mounted on the vehicle during the vehicle's operation, the position of the end of the road is inferred. When generating an image of the end of the road that was not detected among the plurality of images, the undetected section of the vehicle's travel is determined; Based on the vehicle's position when the image is generated in the undetected interval, the behavior of the vehicle in the image generated in the undetected interval or in the driving trajectory corresponding to the undetected interval, it is determined whether the road structure in the undetected interval is a structure with ends; When the road in the undetected section is constructed with ends, the position of the end of the road in the undetected section is inferred along the driving trajectory by interpolation based on the position of the end of the road before the undetected section and the position of the end of the road after the undetected section. On the other hand, when the road in the undetected section is constructed without ends, the position of the end of the road in the undetected section is not inferred.

9. A computer program product comprising a road end estimation computer program, the road end estimation computer program being used to cause a computer to perform: The vehicle's trajectory is inferred based on sensor signals obtained from sensors mounted on the vehicle that detect the vehicle's behavior or the vehicle's own position obtained from positioning devices mounted on the vehicle. Based on an image showing the end of the road on which the vehicle is traveling, generated by a camera mounted on the vehicle during the vehicle's operation, the position of the end of the road is inferred. When generating an image of the end of the road that was not detected among the plurality of images, the undetected section of the vehicle's travel is determined; Based on the vehicle's position when the image is generated in the undetected interval, the behavior of the vehicle in the image generated in the undetected interval or in the driving trajectory corresponding to the undetected interval, it is determined whether the road structure in the undetected interval is a structure with ends; When the road in the undetected section is constructed with ends, the position of the end of the road in the undetected section is inferred along the driving trajectory by interpolation based on the position of the end of the road before the undetected section and the position of the end of the road after the undetected section. On the other hand, when the road in the undetected section is constructed without ends, the position of the end of the road in the undetected section is not inferred.

Images