Position estimation device, position estimation method, and computer program for position estimation

Abstract

To provide a position estimation device capable of accurately estimating a position of a feature shown in an onboard image taken by a camera installed in a vehicle.SOLUTION: A position estimation device includes: a vehicle detection unit 41 that detects a first position of a vehicle 2, which has travelled over a predetermined road section, at a first time on the basis of a stationary sensor 3 installed on the predetermined road section or near the predetermined road section; a vehicle position estimation unit 42 that estimates a second position of the vehicle 2 at a second time, which is a time when an onboard image showing a predetermined feature is generated by an onboard camera 11 installed in the vehicle 2, on the basis of the first position of the vehicle 2 at the first time; and a feature position estimation unit 44 that estimates the position of the predetermined feature in a real space on the basis of the estimated second position.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a position estimation device, a position estimation method, and a computer program for position estimation that estimate the position of a predetermined ground feature represented in an image generated by an in-vehicle camera.

Background Art

[0002] In order to generate or update a high-precision map that an automatic driving system of a vehicle refers to for automatically driving and controlling the vehicle, it is required to accurately detect the position of a ground feature related to the running of the vehicle provided on or around the road. Therefore, a technique for detecting a ground feature based on an image generated by a camera mounted on a vehicle actually running on the road has been proposed (see Patent Document 1).

[0003] The information integration device disclosed in Patent Document 1 recognizes one or more first ground features around a moving body based on sensor information acquired from sensors provided in the moving body, and generates first ground feature information indicating the first ground features. Further, this information integration device acquires second ground feature information indicating one or more second ground features recognized to exist around the moving body from an external device, or from both the external device and map information, and identifies the same ground feature from among the one or more first ground features indicated by the first ground feature information and the one or more second ground features indicated by the second ground feature information. Then, this information integration device generates correction information used to correct at least one of the first ground feature information and the second ground feature information based on the difference in the positions of the first ground feature and the second ground feature identified as the same ground feature, and after correcting at least one of the first ground feature information and the second ground feature information using the generated correction information, generates integrated ground feature information by integrating the first ground feature information and the second ground feature information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] If the accuracy of vehicle position detection is insufficient, the accuracy of the location of features detected from on-board images obtained by the on-board camera may also be insufficient. In the above technology, in order to generate correction information, it is necessary that the first feature information and the second feature information contain information about the same feature. However, the features represented in the sensor information from sensors installed on the moving vehicle are not always shown in the feature information acquired from an external device. Therefore, it is necessary to accurately estimate the location of features represented in on-board images.

[0006] Therefore, the present invention aims to provide a position estimation device capable of accurately estimating the position of features represented in on-board images obtained by a camera mounted on a vehicle. [Means for solving the problem]

[0007] According to one embodiment, a position estimation device is provided. This position estimation device includes: a vehicle detection unit that detects a first position of a vehicle traveling on a predetermined road section at a first time based on a fixed sensor installed on or near a predetermined road section; a vehicle position estimation unit that estimates a second position of the vehicle at a second time based on the first position of the vehicle at the first time, where an on-board image showing a predetermined feature is generated by an on-board camera mounted on the vehicle; and a feature position estimation unit that estimates the position of a predetermined feature in real space based on the estimated second position.

[0008] In this position estimation device, the fixed sensor is preferably a surveillance camera, and the surveillance camera preferably generates surveillance images representing a predetermined road section at predetermined intervals. The vehicle detection unit detects the position of the vehicle from each of a plurality of surveillance images that are generated at different times, and the vehicle position estimation unit preferably estimates the second position by taking the position of the vehicle detected from the surveillance image generated at the time closest to the second time as the first position.

[0009] Furthermore, it is preferable that the vehicle position estimation unit estimates the second position by correcting the second pre-correction position of the vehicle, measured by the satellite positioning device at a second time, according to the difference between the first pre-correction position of the vehicle, measured by the satellite positioning device mounted on the vehicle at a first time, and the first position.

[0010] Furthermore, the position estimation device preferably further includes a feature detection unit that detects a predetermined feature from an in-vehicle image generated at a second time point. The feature position estimation unit preferably estimates the position of the predetermined feature in real space based on the position of the predetermined feature on the in-vehicle image and the second position.

[0011] Another embodiment provides a position estimation method. This position estimation method includes detecting a first position of a vehicle traveling on a predetermined road section at a first time based on a fixed sensor installed on or near a predetermined road section; estimating a second position of the vehicle at a second time based on the first position of the vehicle at the first time, where an onboard image showing a predetermined feature is generated by an onboard camera mounted on the vehicle; and estimating the position of a predetermined feature in real space based on the estimated second position.

[0012] In yet another embodiment, a computer program for position estimation is provided. This computer program for position estimation includes instructions to cause a computer to perform the following actions: detect a first position of a vehicle traveling on a predetermined road section at a first time based on a fixed sensor installed on or near a predetermined road section; estimate a second position of the vehicle at a second time based on the first position of the vehicle at the first time, where an on-board image showing a predetermined feature is generated by an on-board camera mounted on the vehicle; and estimate the position of a predetermined feature in real space based on the estimated second position. [Effects of the Invention]

[0013] The position estimation device described herein has the effect of accurately estimating the position of features represented in images obtained by a camera mounted on a vehicle. [Brief explanation of the drawing]

[0014] [Figure 1] This is a schematic diagram of the geographical feature data collection system in which a position estimation device is implemented. [Figure 2] This is a schematic diagram of the vehicle's configuration. [Figure 3] This is a hardware configuration diagram of the data acquisition device. [Figure 4] This is a hardware configuration diagram of a server, which is an example of a position estimation device. [Figure 5] This is a functional block diagram of the server processor related to position estimation processing. [Figure 6] This diagram illustrates the general process of estimating the location of geographical features. [Figure 7] This is a flowchart illustrating the operation of the position estimation process. [Modes for carrying out the invention]

[0015] The position estimation device, the position estimation method executed by the position estimation device, and the computer program for position estimation will be described below with reference to the figures. This position estimation device collects on-board images representing features related to the movement of vehicles from one or more vehicles capable of communication in a predetermined area, detects predetermined features related to the movement of vehicles represented in the collected on-board images, and estimates the position of those features. At that time, this position estimation device detects the first position of a vehicle that has traveled on a predetermined road section at a first time, based on fixed sensors installed on or near a predetermined road section. Then, based on the first position of the vehicle at the first time, this position estimation device estimates the second position of the vehicle at a second time when an on-board image showing the predetermined features is generated, and estimates the position of the predetermined features based on the estimated second position.

[0016] Note that the predetermined ground objects to be detected include, for example, various road signs, various road markings, traffic lights, and other ground objects related to the running of vehicles.

[0017] FIG. 1 is a schematic configuration diagram of a ground object data collection system in which a position estimation device is implemented. In the present embodiment, the ground object data collection system 1 includes at least one vehicle 2, at least one fixed sensor 3, and a server 4 which is an example of a position estimation device. Each vehicle 2 is connected to the server 4 via a wireless base station 6 and a communication network 5 by accessing, for example, the wireless base station 6 connected to a communication network 5 to which the server 4 is connected and a gateway (not shown). Also, each fixed sensor 3 is connected to the communication network 5 via a gateway or the like, and is connected to the server 4 via the communication network 5. Note that in FIG. 1, only one vehicle 2 is shown for simplicity, but the ground object data collection system 1 may have a plurality of vehicles 2. Similarly, in FIG. 1, only one fixed sensor 3 is shown, but it may have a plurality of fixed sensors 3. Further, in FIG. 1, only one wireless base station 6 is shown, but a plurality of wireless base stations 6 may be connected to the communication network 5.

[0018] FIG. 2 is a schematic configuration diagram of the vehicle 2. The vehicle 2 has a camera 11, a GPS receiver 12, a wireless communication terminal 13, and a data acquisition device 14. The camera 11, the GPS receiver 12, the wireless communication terminal 13, and the data acquisition device 14 are communicably connected via an in-vehicle network compliant with a standard such as a controller area network. Also, the vehicle 2 may further have a navigation device (not shown) that searches for a planned route of the vehicle 2 and navigates the vehicle 2 to travel along the planned route.

[0019] Camera 11 is an example of an in-vehicle camera. It includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. And the camera 11 is mounted, for example, in the passenger compartment of the vehicle 2 so as to face the front of the vehicle 2. Then, the camera 11 photographs the front area of the vehicle 2 at a predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second), and generates an image in which the front area is shown. The image obtained by the camera 11 is an example of an in-vehicle image, and may be a color image or a gray image. Note that a plurality of cameras 11 having different photographing directions or focal lengths may be provided on the vehicle 2.

[0020] Each time the camera 11 generates an image, it outputs the generated image to the data acquisition device 14 via the in-vehicle network.

[0021] The GPS receiver 12 is an example of a satellite positioning device. It receives GPS signals from GPS satellites at a predetermined cycle, and determines the self-position of the vehicle 2 based on the received GPS signals. Note that the predetermined cycle at which the GPS receiver 12 determines the self-position of the vehicle 2 may be different from the photographing cycle by the camera 11. Then, the GPS receiver 12 outputs positioning information representing the result of determining the self-position of the vehicle 2 based on the GPS signals to the data acquisition device 14 via the in-vehicle network at a predetermined cycle. The GPS receiver 12 may include an index value representing the accuracy of the determined position, such as the reception intensity of the GPS signal or the number of satellites from which the GPS signal has been received, in the positioning information. Note that the vehicle 2 may have a receiver compliant with a satellite positioning system other than the GPS receiver 12. In this case, the receiver may determine the self-position of the vehicle 2.

[0022] The wireless communication terminal 13 is an example of a communication unit and is a device that performs wireless communication processing in accordance with a predetermined wireless communication standard. For example, by accessing the wireless base station 6, it connects to the server 4 via the wireless base station 6 and the communication network 5. The wireless communication terminal 13 generates an uplink wireless signal that includes feature data, including images generated by the camera 11, received from the data acquisition device 14. The wireless communication terminal 13 then transmits the images and other data to the server 4 by sending the uplink wireless signal to the wireless base station 6. The wireless communication terminal 13 also receives a downlink wireless signal from the wireless base station 6 and passes the collection instructions from the server 4 contained in the wireless signal to the data acquisition device 14 or the electronic control unit (ECU, not shown) that controls the movement of the vehicle 2.

[0023] Figure 3 is a hardware configuration diagram of the data acquisition device. The data acquisition device 14 generates feature data based on images generated by the camera 11. Furthermore, the data acquisition device 14 generates driving information representing the driving behavior of the vehicle 2. To this end, the data acquisition device 14 has a communication interface 21, a memory 22, and a processor 23.

[0024] The communication interface 21 is an example of an in-vehicle communication unit and has an interface circuit for connecting the data acquisition device 14 to the in-vehicle network. Specifically, the communication interface 21 is connected to the camera 11, GPS receiver 12, and wireless communication terminal 13 via the in-vehicle network. Whenever the communication interface 21 receives an image from the camera 11, it passes the received image to the processor 23. Also, whenever the communication interface 21 receives positioning information from the GPS receiver 12, it passes the received positioning information to the processor 23. Furthermore, the communication interface 21 passes the instruction to collect feature data received from the server 4 via the wireless communication terminal 13 to the processor 23. Finally, the communication interface 21 outputs the feature data received from the processor 23 to the wireless communication terminal 13 via the in-vehicle network.

[0025] Memory 22 includes, for example, volatile semiconductor memory and non-volatile semiconductor memory. Memory 22 may further include other storage devices such as a hard disk drive. Memory 22 stores various data used in processing related to feature data generation performed by the processor 23 of the data acquisition device 14. Such data includes, for example, identification information of the vehicle 2, and parameters of the camera 11 such as the installation height of the camera 11, shooting direction, focal length, and field of view. Memory 22 may also store images received from the camera 11 and positioning information received from the GPS receiver 12 for a certain period of time. Furthermore, memory 22 stores information representing the area to be used for generating and collecting feature data (hereinafter sometimes referred to as the collection target area), as specified in the feature data collection instruction. In addition, memory 22 may store computer programs for implementing each process performed by the processor 23.

[0026] The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic unit, a numerical unit, or a graphics processing unit. The processor 23 stores images received from the camera 11 and positioning information received from the GPS receiver 12 in the memory 22. Furthermore, while the vehicle 2 is in motion, the processor 23 performs processing related to the generation of feature data at predetermined intervals (e.g., 0.1 seconds to 10 seconds).

[0027] The processor 23 performs a process related to generating feature data, for example, by determining whether the vehicle's position, represented by positioning information received from the GPS receiver 12, is included in the data collection area. If the vehicle's position is included in the data collection area, the processor 23 generates feature data based on the image received from the camera 11.

[0028] Feature data is data representing features related to the vehicle's movement. In this embodiment, the processor 23 includes in the feature data an image generated by the camera 11, the time the image was generated, the position and direction of travel of the vehicle 2 as indicated in the positioning information at that time, and parameters of the camera 11 such as the installation height, shooting direction, focal length, and field of view of the camera 11. The processor 23 can obtain information representing the direction of travel of the vehicle 2 from the vehicle 2's ECU or compass sensor (not shown). If the time the image was generated differs from the time the latest positioning information was obtained, the processor 23 obtains odometry information of the vehicle 2 used for dead reckoning, such as wheel speed, acceleration, and angular velocity, from the ECU from the time the latest positioning information was obtained to the time the image was generated. The processor 23 may then determine the position and direction of travel of the vehicle 2 at the time the image was generated by correcting the position and direction of travel of the vehicle 2 as indicated in the latest positioning information using this odometry information. Odometry information is an example of behavioral information representing the behavior of the vehicle 2. The processor 23 then transmits the generated feature data to the server 4 via the wireless communication terminal 13 each time it generates feature data. The processor 23 may include multiple images, the generation time of each image, and the position and direction of travel of the vehicle 2 in a single feature data. In addition, the processor 23 may transmit the parameters of the camera 11 along with the identification information of the vehicle 2 to the server 4 via the wireless communication terminal 13, separately from the feature data.

[0029] Furthermore, the processor 23 generates driving information of vehicle 2 from a predetermined timing (for example, the timing when the ignition switch of vehicle 2 is turned on) and transmits this driving information to the server 4 via the wireless communication terminal 13. The processor 23 includes in the driving information a series of positioning information obtained from the predetermined timing onward, the time at which the position of vehicle 2 was measured for each piece of positioning information, and odometry information obtained from the ECU. In addition, the processor 23 may include identification information of vehicle 2 in the driving information and feature data.

[0030] The fixed sensor 3 is a sensor installed on or near a predetermined road section and capable of detecting a vehicle 2 traveling on the predetermined road section. There are no particular restrictions on the predetermined road section, but it is preferable that it be a road section on which the vehicle 2 collecting feature data is likely to travel. The fixed sensor 3 can be, for example, a camera (hereinafter sometimes referred to as a surveillance camera to distinguish it from a camera mounted on the vehicle 2) installed so as to include the predetermined road section in its shooting range. In this case, the surveillance camera, which is the fixed sensor 3, is installed facing the predetermined road section on a support member provided to straddle the predetermined road section or on a pole installed adjacent to the predetermined road section. The surveillance camera generates an image representing the predetermined road section by photographing the predetermined road section at predetermined intervals. Each time the surveillance camera generates a surveillance image, it transmits the generated image and the time of its generation to the server 4 via the communication network 5. The image generated by the surveillance camera is an example of a fixed sensor signal that detects a vehicle 2 traveling on the predetermined road section. Hereinafter, in order to distinguish between in-vehicle images generated by the in-vehicle camera 11 and images generated by the surveillance camera, images generated by the surveillance camera may be referred to as surveillance images.

[0031] The fixed sensor 3 may be a loop coil type vehicle detection sensor that utilizes a loop coil embedded in the road surface of a predetermined road section. Alternatively, the fixed sensor 3 may be an ultrasonic type vehicle detection sensor having an ultrasonic transceiver installed on the shoulder of the road in a predetermined road section. Furthermore, the fixed sensor 3 may be a beacon device capable of bidirectional communication with an on-board unit mounted on the vehicle 2. When the fixed sensor 3 is one of these sensors, each time the fixed sensor 3 detects a vehicle passing through the predetermined road section, it transmits a vehicle detection signal to the server 4 via the communication network 5, indicating that a vehicle has been detected and the time the vehicle was detected (hereinafter simply referred to as the detection time). The vehicle detection signal is another example of a fixed sensor signal.

[0032] Next, we will describe Server 4, which is an example of a position estimation device. Figure 4 is a hardware configuration diagram of server 4, which is an example of a position estimation device. Server 4 has a communication interface 31, a storage device 32, a memory 33, and a processor 34. The communication interface 31, the storage device 32, and the memory 33 are connected to the processor 34 via signal lines. Server 4 may further have input devices such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0033] The communication interface 31 is an example of a communication unit and has an interface circuit for connecting the server 4 to the communication network 5. The communication interface 31 is configured to communicate with the vehicle 2 via the communication network 5 and the wireless base station 6. Specifically, the communication interface 31 passes feature data or driving information received from the vehicle 2 via the wireless base station 6 and the communication network 5 to the processor 34. The communication interface 31 also transmits collection instructions received from the processor 34 to the vehicle 2 via the communication network 5 and the wireless base station 6. Furthermore, the communication interface 31 passes fixed sensor signals received from the fixed sensor 3 via the communication network 5 to the processor 34.

[0034] The storage device 32 is an example of a storage unit and includes, for example, a hard disk drive or an optical recording medium and its access device. The storage device 32 stores various data and information used in the position estimation process. For example, the storage device 32 stores a parameter set for identifying a classifier for detecting features from an image, a map to be updated, and identification information for each vehicle 2. Furthermore, the storage device 32 stores feature data received from each vehicle 2, driving information, and fixed sensor signals received from each fixed sensor 3. In addition, for each fixed sensor 3, the storage device 32 stores detection position information representing the position of the vehicle 2 that the fixed sensor 3 can detect. If the fixed sensor 3 is a surveillance camera, the detection position information includes the real-space position coordinates corresponding to each of the multiple points on the image generated by the surveillance camera. Alternatively, the detection position information may include parameters related to the surveillance camera, such as the installation position, shooting direction, focal length, and field of view. Also, if the fixed sensor 3 is a vehicle detection sensor, it includes the real-space position coordinates of the position where the vehicle detection sensor can detect the vehicle 2. The detected location information only needs to be transmitted from the fixed sensor 3 to the server 4 when the fixed sensor 3 is installed. Alternatively, the detected location information may be input via the user interface of the server 4. Furthermore, the storage device 32 may store a computer program for performing location estimation processing, which is executed on the processor 34.

[0035] Memory 33 is another example of a storage unit, and may include, for example, a non-volatile semiconductor memory and a volatile semiconductor memory. Memory 33 temporarily stores various data generated during the position estimation process.

[0036] The processor 34 is an example of a control unit and has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 34 may further have other arithmetic circuits such as a logic unit or a numerical unit. The processor 34 generates a data collection instruction for feature data representing the area to be collected at predetermined intervals or at timings specified by the user, and transmits the generated collection instruction to each vehicle 2 via the communication network 5 and the wireless base station 6. Furthermore, the processor 34 performs position estimation processing.

[0037] Figure 5 is a functional block diagram of the processor 34 related to the position estimation process. The processor 34 includes a vehicle detection unit 41, a vehicle position estimation unit 42, a feature detection unit 43, and a feature position estimation unit 44. Each of these parts of the processor 34 is, for example, a functional module realized by a computer program running on the processor 34. Alternatively, each of these parts of the processor 34 may be a dedicated arithmetic circuit provided on the processor 34.

[0038] The vehicle detection unit 41 detects the first position of the vehicle 2 that traveled a predetermined road section at a first time, based on the fixed sensor signal received from the fixed sensor 3.

[0039] If the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 detects a vehicle 2 that has traveled a predetermined road section by inputting the surveillance image generated by the surveillance camera into a classifier. The vehicle detection unit 41 can use, for example, a so-called deep neural network (DNN) as such a classifier. As such a DNN, for example, a DNN with a convolutional neural network (CNN) type architecture such as Single Shot MultiBox Detector (SSD) or Faster R-CNN, or a DNN with an attention mechanism such as Vision Transformer can be used. Alternatively, the vehicle detection unit 41 may use a classifier based on other machine learning methods such as a support vector machine (SVM) or adaBoost as such a classifier. Such a classifier is pre-trained using a number of training images representing vehicles according to a predetermined learning method such as backpropagation, so as to detect vehicles from images.

[0040] The classifier may be further trained to detect license plates attached to vehicles. In this case, the vehicle detection unit 41 may identify the registration number of the detected vehicle by inputting the region on the monitoring image where the license plate is shown into a number classifier that has been pre-trained to identify the registration number. The vehicle detection unit 41 then identifies an identification number that matches the identified registration number from among the identification numbers of each vehicle 2 stored in the storage device 32 or memory 33. The vehicle detection unit 41 may use a CNN-type DNN or a DNN with an attention mechanism as the number classifier. The vehicle detection unit 41 then determines that a vehicle 2 with the identified identification number has been detected. Furthermore, the vehicle detection unit 41 sets the generation time of the monitoring image showing the detected vehicle 2 as the first time. Furthermore, the vehicle detection unit 41 identifies a point among a plurality of points whose real-space position coordinates are associated with the region where the detected vehicle 2 is shown on the image, or the point closest to that region, and sets the real-space position coordinates corresponding to the identified point as the first position of the detected vehicle 2. Furthermore, the position of each pixel on the surveillance image corresponds one-to-one with the orientation from the surveillance camera to the object represented by that pixel. Therefore, the vehicle detection unit 41 may determine the first position of the detected vehicle 2 based on the orientation from the surveillance camera corresponding to the position of the area in the surveillance image where the vehicle 2 is represented, and parameters of the surveillance camera such as the installation position and shooting direction of the surveillance camera.

[0041] Furthermore, the vehicle detection unit 41 may refer to the driving information received from each vehicle 2 and identify the vehicle that is traveling in the position closest to the first position at the first time as the vehicle 2 that was at the first position at the first time. In this case, the vehicle detection unit 41 may omit the detection of the registration number. In addition, the classifier may be pre-trained to also identify the type of vehicle of the detected vehicle. In this case, the vehicle detection unit 41 may identify the vehicle that is the same type of vehicle as the vehicle detected at the first time as the vehicle 2 that was at the first position at the first time. In this case as well, the vehicle detection unit 41 may omit the detection of the registration number.

[0042] Furthermore, if the fixed sensor 3 is a vehicle detection sensor or a beacon device, the vehicle detection unit 41 sets the detection time included in the received vehicle detection signal as the first time, and the position detectable by the vehicle detection sensor as the first position. The vehicle detection unit 41 then refers to the driving information received from each vehicle 2 and identifies the vehicle 2 that is traveling at the position closest to the first position at the first time as the vehicle 2 that was at the first position at the first time.

[0043] The vehicle detection unit 41 notifies the vehicle position estimation unit 42 of the first time and first position of the detected vehicle 2.

[0044] The vehicle position estimation unit 42 estimates the position of the vehicle 2 (second position) at the generation time (second time) of an image representing a predetermined feature included in the feature data received from the vehicle 2, based on the first position of the detected vehicle 2 at a first time.

[0045] The vehicle position estimation unit 42 calculates a position correction vector representing the difference between the position of vehicle 2 at a first time (hereinafter referred to as the first pre-correction position for convenience of explanation), which is included in the driving information received from the detected vehicle 2, and the first position at the first time detected by the vehicle detection unit 41. The vehicle position estimation unit 42 then corrects the position of vehicle 2 at the image generation time (second time) (hereinafter referred to as the second pre-correction position for convenience of explanation), which is included in the feature data, with the position correction vector (i.e., the position obtained by adding the position correction vector to the second pre-correction position), and defines this position as the second position.

[0046] If the driving information does not include the first pre-correction position, the vehicle position estimation unit 42 may determine the first pre-correction position by dividing the position of vehicle 2 at times before and after the first time, which is included in the driving information, by the difference between those times and the first time. Alternatively, if the driving information includes odometry information, the first pre-correction position may be determined by correcting the position of vehicle 2 at a reference time before or after the first time using the odometry information from the reference time to the first time.

[0047] If the server 4 has received multiple feature data from the detected vehicle 2, the vehicle position estimation unit 42 can perform the above processing for each feature data to estimate the second position of vehicle 2 at the second time the image included in that feature data was generated. Similarly, if the feature data includes multiple images and the generation times of those images, the vehicle position estimation unit 42 can perform the above processing for each image to estimate the second position of vehicle 2 at the second time the image was generated.

[0048] The vehicle position estimation unit 42 notifies the feature position estimation unit 44 of the second position it has estimated.

[0049] The feature detection unit 43 detects a predetermined feature to be detected from the image contained in the feature data received from the detected vehicle 2. To this end, the feature detection unit 43 inputs the image into a feature detection classifier to identify the object region in which the predetermined feature is represented and the type of that predetermined feature. The feature detection unit 43 can use a DNN with a CNN-type architecture or a DNN with an attention mechanism as such a classifier. Alternatively, the feature detection unit 43 may use a classifier based on other machine learning methods such as SVM or adaBoost as such a classifier. Such a classifier is pre-trained using a number of training images in which the predetermined feature is represented, according to a predetermined learning method such as backpropagation, in order to detect the predetermined feature from the image.

[0050] The feature detection unit 43 notifies the feature position estimation unit 44 of the second time when an image in which a predetermined feature was detected was generated, as well as information representing the position and size of the object region in the image in which the predetermined feature is represented, and the type of the predetermined feature.

[0051] The feature location estimation unit 44 estimates the location of the detected feature based on the second position of the vehicle 2 when the image representing the detected feature is generated.

[0052] As described above regarding the surveillance camera, the position of each pixel in the image corresponds one-to-one with the orientation from the camera 11 to the object represented by that pixel. Therefore, if the detected feature is a feature placed on the road surface, such as a road marking or a three-dimensional structure placed on the road surface, the feature position estimation unit 44 can estimate the position of the feature in the camera coordinate system with the camera 11 as the reference point. In this case, the feature position estimation unit 44 only needs to estimate the position of the feature based on the orientation from the camera 11 corresponding to the reference point within the object region, the direction of travel of the vehicle 2 at the second time when the image was generated, which is included in the feature data, and parameters of the camera 11 such as the shooting direction of the camera 11 and the installation height on the vehicle 2. Furthermore, the feature position estimation unit 44 can estimate the position of the detected feature by, for example, performing an affine transformation on the position of the feature to convert the position of the feature in the camera coordinate system to a position in a real-space coordinate system with the position of the camera 11 as the second position. The reference point within the object region can be the centroid of the object region. Furthermore, if the geographical feature is a three-dimensional structure installed on the road surface, the position of the lower end of the object's region is presumed to represent the position where the feature is in contact with the road surface. In this case, the reference point can be any position on the lower end of the object's region.

[0053] Furthermore, if the same feature is detected in multiple images generated at different times, the feature position estimation unit 44 may estimate the position of the feature according to the so-called Structure from Motion method. In this case, the feature position estimation unit 44 estimates the position of the feature by triangulation based on the second position and direction of travel of the vehicle 2 at the time of each image generation, the position of the object region in which the feature is represented on each image, and the parameters of the camera 11.

[0054] The feature location estimation unit 44 can then associate the same feature represented in multiple images generated at different times with each other, according to a tracking method such as KLT tracking.

[0055] The feature location estimation unit 44 stores information representing the type of feature and its estimated location in the storage device 32 for each detected feature. Alternatively, the feature location estimation unit 44 may output the information representing the type of feature and its estimated location to other devices via the communication interface 31.

[0056] Figure 6 is a diagram illustrating the overview of the estimation of the location of a feature according to this embodiment. As shown in Figure 6, in this example, at a first time t1 when the vehicle 2 is detected by the fixed sensor 3, the first position P1 of the vehicle 2 is detected based on the fixed sensor signal. Meanwhile, at this first time t1, the position of the vehicle 2 (first pre-correction position) Pg1 is detected by the GPS receiver 12. Therefore, the position correction vector C = |P1 - Pg1| is obtained. Using this position correction vector C, the position of the vehicle 2 (second pre-correction position) Pg2, which was determined by the GPS receiver 12 at a second time t2 when the in-vehicle image IMG was generated by the camera 11 mounted on the vehicle 2, is corrected, and the second position P2 of the vehicle 2 at time t2 is estimated. Then, based on the second position P2, the real-space position Ps of the feature S detected from the in-vehicle image IMG is estimated.

[0057] Figure 7 is an operation flowchart of the position estimation process in server 4. The processor 34 of server 4 should execute the position estimation process according to the operation flowchart shown below.

[0058] The vehicle detection unit 41 of the processor 34 detects the first position of the vehicle 2 that has traveled a predetermined road section at a first time, based on the fixed sensor signal received from the fixed sensor 3 (step S101).

[0059] Furthermore, the vehicle position estimation unit 42 of the processor 34 estimates the second position of the vehicle 2 at a second time when an in-vehicle image showing a predetermined feature to be detected is generated, based on the first position of the detected vehicle 2 at a first time (step S102).

[0060] Furthermore, the feature detection unit 43 of the processor 34 detects a predetermined feature from the in-vehicle image generated at a second time, which is included in the feature data received from the detected vehicle 2 (step S103).

[0061] Then, the feature location estimation unit 44 of the processor 34 estimates the location of the detected feature based on the second location of the vehicle 2 when the in-vehicle image showing the detected feature was generated (step S104). The processor 34 then terminates the location estimation process.

[0062] The processor 34 may generate or update a map based on the type and location of the detected features. For example, in the map to be generated, the processor 34 adds information representing the location and type of each detected feature. Also, in the map to be updated, if there are no features of the same type as the detected feature within a predetermined distance from the location of the detected feature, the processor 34 adds information representing the location and type of the detected feature to the map to be updated. Furthermore, in the map to be updated, if the location of the detected feature contains information about a feature of a different type than the detected feature, the processor 34 rewrites the type of feature at that location to the type of the detected feature. Moreover, if a feature represented at a predetermined point on the map to be updated is not detected from feature data collected from each vehicle 2 that passed through that predetermined point within a certain period, the processor 34 removes the information about the feature at that predetermined point from the map.

[0063] The processor 34 may distribute the generated or updated map to vehicles that use the map for automated driving control or driving assistance via the communication network 5 and the wireless base station 6.

[0064] As explained above, this position estimation device uses the vehicle's position, detected by a fixed sensor capable of accurately detecting the vehicle's position, to estimate the vehicle's position when generating in-vehicle images. Therefore, this position estimation device can accurately estimate the vehicle's position when generating in-vehicle images. As a result, this position estimation device can accurately estimate the positions of features detected from in-vehicle images.

[0065] In a modified version, the vehicle position estimation unit 42 may estimate the position of vehicle 2 at the time the in-vehicle image was generated, but only for in-vehicle images in which a predetermined feature has been detected by the feature detection unit 43 from a series of in-vehicle images received from vehicle 2. This reduces the amount of computation required.

[0066] In another modification, the vehicle position estimation unit 42 may use odometry information included in the driving information to determine the direction and amount of movement of the vehicle 2 from a first time when the vehicle 2 is detected by the fixed sensor 3 to a second time when an in-vehicle image showing a predetermined feature is generated. The vehicle position estimation unit 42 may then estimate the second position of the vehicle at the second time by correcting the first position with its direction and amount of movement. As a result, even if the GPS receiver 12 mounted on the vehicle 2 is unable to determine the position of the vehicle 2 at the second time, the vehicle position estimation unit 42 can estimate the second position of the vehicle 2.

[0067] In another modification, if the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 may detect the position of vehicle 2 from each of a plurality of surveillance images generated at different times. The vehicle position estimation unit 42 may then estimate the second position of vehicle 2 at the second time, using the position of vehicle 2 detected from the surveillance image generated at the time closest to the second time as the first position. In this way, since the surveillance image, which is a fixed sensor signal, generated at the time closest to the generation of the in-vehicle image showing a predetermined feature is used to estimate the position of vehicle 2, the vehicle position estimation unit 42 can further improve the accuracy of estimating the second position of vehicle 2.

[0068] Furthermore, if multiple fixed sensors 3 are installed, the vehicle detection unit 41 may perform the above processing for each fixed sensor 3 to detect the vehicle 2. In this case, if the same vehicle 2 is detected from the fixed sensor signals of two or more fixed sensors 3, the vehicle position estimation unit 42 may estimate the second position of the vehicle 2 by using the position of the vehicle 2 that is closer to the second pre-correction position included in the driving information as the first position. In this way, the vehicle position estimation unit 42 can further improve the accuracy of the second position estimation by using the position of the vehicle 2 detected by the fixed sensor 3 that is closer to the second pre-correction position of the vehicle 2 when the in-vehicle image showing a predetermined feature is generated.

[0069] In another modification, the processor 23 of the data acquisition device 14 mounted on the vehicle 2 may perform the processing of the feature detection unit 43 in the above embodiment. In this case, the processor 23 should include in the feature data the generation time of the in-vehicle image in which the predetermined feature was detected, the position of the vehicle 2 as determined by the GPS receiver 12 at that generation time, the direction of travel of the vehicle 2, the parameters of the camera 11, and information representing the position and range of the object region in which the predetermined feature is represented on the in-vehicle image. The feature position estimation unit 44 should then estimate the position of the predetermined feature detected from the in-vehicle image by referring to this information included in the feature data.

[0070] In another variation, if the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 may estimate the blind spot area of ​​vehicle 2 based on the surveillance image. The processor 34 may then determine that the location of a feature detected from the in-vehicle image is within the blind spot area, and may not use the information about that feature for map generation or updating. For example, if other vehicles (hereinafter referred to as surrounding vehicles) are detected in the surveillance image as well as vehicle 2, the vehicle detection unit 41 may identify the area that is obscured by the surrounding vehicles from the perspective of vehicle 2 as the blind spot area, based on the positional relationship between vehicle 2 and the surrounding vehicles.

[0071] Furthermore, if a beacon device is used as the fixed sensor 3, the data acquisition device 14 of the vehicle 2 may detect the position of the vehicle 2. In this case, the memory 22 of the data acquisition device 14 stores in advance the position in which a beacon signal from the beacon device can be received. The processor 23 of the data acquisition device 14 then determines the time when a beacon signal emitted from the beacon device is detected by an on-board device mounted on the vehicle 2 as the first time, and detects the position in which the beacon signal can be received as the first position. The processor 23 may then include the first time and the first position, or a position correction vector, in the driving information transmitted to the server 4. In this case, the processor 23 of the data acquisition device 14 may estimate the position of the detected feature by executing the processing of each part of the processor 34 of the server 4 as in the above embodiment. The processor 34 may then include the position of the detected feature in the feature data. In this case, the data acquisition device 14 is another example of a position estimation device.

[0072] The computer program that enables a computer to implement the functions of each part of the processor of the position estimation device according to each of the above embodiments or modifications may be provided in the form of a recording medium that can be read by a computer. The recording medium that can be read by a computer may be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0073] As described above, those skilled in the art can make various modifications within the scope of the present invention to suit the implemented form. [Explanation of symbols]

[0074] 1. Feature Data Collection System 2 vehicles 11 Cameras 12 GPS receivers 13 Wireless communication terminals 14. Data acquisition device 21 Communication Interface 22 memory 23 processors 3 Fixed Sensors 4 servers 31 Communication Interface 32 Storage devices 33 memory 34 processors 41 Vehicle Mount 42 Vehicle position estimation unit 43. Feature detection unit 44 Feature position estimation section 5. Communication Network 6 Wireless base stations

Claims

1. A vehicle detection unit that detects the first position of a vehicle that has traveled along a predetermined road section at a first time, based on a fixed sensor installed on or near a predetermined road section, A vehicle position estimation unit estimates the second position of the vehicle at a second time, based on the first position of the vehicle at the first time, and an in-vehicle image showing predetermined features is generated by an in-vehicle camera mounted on the vehicle. A feature detection unit that detects the predetermined feature from the in-vehicle image generated at the second time, A feature location estimation unit that estimates the location of the predetermined feature in real space based on the second location, It has, The fixed sensor is a surveillance camera that generates a surveillance image showing the predetermined road section. The vehicle detection unit estimates the blind spots of the vehicle based on the surveillance images generated by the surveillance camera. The feature location estimation unit determines that the feature has been falsely detected if the estimated location of the feature detected from the in-vehicle image is included in the blind spot area. Location estimation device.

2. The surveillance camera generates a surveillance image showing the predetermined road section at predetermined intervals, The vehicle detection unit detects the position of the vehicle from each of the multiple monitoring images, which are generated at different times. The position estimation device according to claim 1, wherein the vehicle position estimation unit estimates the second position by taking the position of the vehicle detected from the monitoring image generated at the time closest to the second time from among a plurality of monitoring images as the first position.

3. The position estimation device according to claim 1 or 2, wherein the vehicle position estimation unit estimates the second position by correcting the second pre-correction position of the vehicle, measured by the satellite positioning device at a second time, according to the difference between the first pre-correction position of the vehicle, measured by the satellite positioning device mounted on the vehicle at a first time, and the first position.

4. The position estimation device according to claim 1 or 2, wherein the feature position estimation unit estimates the position of the predetermined feature in real space based on the position of the predetermined feature on the in-vehicle image and the second position.

5. Based on fixed sensors installed on or near a predetermined road section, the first position of a vehicle that traveled along the predetermined road section at a first time is detected. Based on the first position of the vehicle at the first time, the second position of the vehicle at the second time is estimated, which is generated by an onboard camera mounted on the vehicle, showing an onboard image of a predetermined feature. The predetermined feature is detected from the in-vehicle image generated at the second time, Based on the second position, the position of the predetermined feature in real space is estimated. This includes, The fixed sensor is a surveillance camera that generates a surveillance image showing the predetermined road section. Detecting the first position of the vehicle includes estimating the blind spots of the vehicle based on the surveillance images generated by the surveillance camera. Estimating the position of the predetermined feature includes determining that the predetermined feature was falsely detected if the estimated position of the predetermined feature detected from the in-vehicle image falls within the blind spot area. Location estimation method.

6. Based on fixed sensors installed on or near a predetermined road section, the first position of a vehicle that traveled along the predetermined road section at a first time is detected. Based on the first position of the vehicle at the first time, the second position of the vehicle at the second time is estimated, which is generated by an onboard camera mounted on the vehicle, showing an onboard image of a predetermined feature. The predetermined feature is detected from the in-vehicle image generated at the second time, Based on the second position, the position of the predetermined feature in real space is estimated. A computer program for position estimation that causes a computer to perform the following: The fixed sensor is a surveillance camera that generates a surveillance image showing the predetermined road section. Detecting the first position of the vehicle includes estimating the blind spots of the vehicle based on the surveillance images generated by the surveillance camera. Estimating the position of the predetermined feature includes determining that the predetermined feature was falsely detected if the estimated position of the predetermined feature detected from the in-vehicle image falls within the blind spot area. A computer program for location estimation.

Images