Position estimation device, position estimation method, and position estimation computer program

Abstract

To provide a position estimation device capable of accurately estimating a position of a feature shown in an onboard image taken by a camera installed in a vehicle.SOLUTION: A position estimation device includes: a vehicle detection unit 41 that detects a first position of a vehicle 2, which has travelled over a predetermined road section, at a first time on the basis of a stationary sensor 3 installed on the predetermined road section or near the predetermined road section; a vehicle position estimation unit 42 that estimates a second position of the vehicle 2 at a second time, which is a time when an onboard image showing a predetermined feature is generated by an onboard camera 11 installed in the vehicle 2, on the basis of the first position of the vehicle 2 at the first time; and a feature position estimation unit 44 that estimates the position of the predetermined feature in a real space on the basis of the estimated second position.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a position estimation device, a position estimation method, and a position estimation computer program for estimating the position of a predetermined feature shown in an image generated by an on-vehicle camera. [Background technology]

[0002] In order to generate or update a highly accurate map that an autonomous driving system of a vehicle refers to for controlling the autonomous driving of the vehicle, it is necessary to accurately detect the positions of features related to the traveling of the vehicle that are provided on the road or around the road.Therefore, a technology has been proposed for detecting features based on images generated by a camera mounted on a vehicle actually traveling on the road (see Patent Document 1).

[0003] The information integration device disclosed in Patent Document 1 recognizes one or more first features around the mobile body based on sensor information acquired from a sensor equipped on the mobile body, and generates first feature information indicating the first features. The information integration device also acquires second feature information indicating one or more second features recognized to exist around the mobile body from an external device or from both the external device and map information, and identifies an identical feature from one or more first features indicated by the first feature information and one or more second features indicated by the second feature information. The information integration device then generates correction information used to correct at least one of the first feature information and the second feature information based on the difference in position between the first feature and the second feature identified as the identical feature, corrects at least one of the first feature information and the second feature information using the generated correction information, and then integrates the first feature information and the second feature information to generate integrated feature information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7126629 Summary of the Invention [Problem to be solved by the invention]

[0005] When the detection accuracy of the vehicle position is insufficient, the accuracy of the position of the feature detected from the vehicle-mounted image obtained by the vehicle-mounted camera may also be insufficient. In the above technology, in order to generate correction information, it is necessary that the first feature information and the second feature information contain information on the same feature. However, the feature shown in the sensor information from the sensor provided in the mobile body is not always shown in the feature information obtained from the external device. Therefore, it is required to accurately estimate the position of the feature shown in the vehicle-mounted image.

[0006] Therefore, an object of the present invention is to provide a position estimation device capable of accurately estimating the position of a feature shown in an in-vehicle image obtained by a camera mounted on a vehicle. [Means for solving the problem]

[0007] According to one embodiment, there is provided a position estimation device, which includes a vehicle detection unit that detects a first position of a vehicle that has traveled along a predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section, a vehicle position estimation unit that estimates a second position of the vehicle at a second time when an on-board image depicting a predetermined feature is generated by an on-board camera mounted on the vehicle based on the first position of the vehicle at the first time, and a feature position estimation unit that estimates a position of the predetermined feature in real space based on the estimated second position.

[0008] In this position estimation device, it is preferable that the fixed sensor is a surveillance camera that generates surveillance images depicting a predetermined road section at a predetermined cycle. It is also preferable that the vehicle detection unit detects the position of the vehicle from each of the plurality of surveillance images generated at different times, and the vehicle position estimation unit estimates the second position by taking the position of the vehicle detected from the surveillance image generated at the time closest to the second time among the plurality of surveillance images as the first position.

[0009] Furthermore, it is preferable that the vehicle position estimation unit estimates the second position by correcting a second pre-correction position of the vehicle measured by a satellite positioning device mounted on the vehicle at a second time in accordance with a difference between the first pre-correction position of the vehicle measured by the satellite positioning device mounted on the vehicle at a first time and the first position.

[0010] It is preferable that the position estimation device further includes a feature detection unit that detects a predetermined feature from the vehicle-mounted image generated at the second time. It is preferable that the feature position estimation unit estimates the position of the predetermined feature in real space based on the position of the predetermined feature on the vehicle-mounted image and the second position.

[0011] According to another embodiment, there is provided a position estimation method, which includes detecting a first position of a vehicle that has traveled along a predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section, estimating a second position of the vehicle at a second time when an on-board image depicting the predetermined feature is generated by an on-board camera mounted on the vehicle based on the first position of the vehicle at the first time, and estimating a position of the predetermined feature in real space based on the estimated second position.

[0012] According to yet another embodiment, there is provided a computer program for position estimation, the computer program for position estimation including instructions for causing a computer to execute the following: detecting a first position of a vehicle that has traveled along a predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section, estimating a second position of the vehicle at a second time when an on-board image depicting the predetermined feature is generated by an on-board camera mounted on the vehicle based on the first position of the vehicle at the first time, and estimating a position of the predetermined feature in real space based on the estimated second position. Effect of the Invention

[0013] The position estimation device according to the present disclosure advantageously provides the ability to accurately estimate the position of a feature depicted in an image acquired by a camera mounted on a vehicle. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic configuration diagram of a feature data collection system in which a position estimation device is implemented. [Diagram 2] FIG. 1 is a schematic configuration diagram of a vehicle. [Diagram 3] FIG. 2 is a hardware configuration diagram of a data acquisition device. [Figure 4] FIG. 2 is a hardware configuration diagram of a server that is an example of a position estimation device. [Diagram 5] FIG. 2 is a functional block diagram of a server processor related to a position estimation process. [Figure 6] FIG. 1 is a diagram for explaining an overview of estimation of the position of a feature. [Figure 7] 13 is an operational flowchart of a position estimation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, a position estimation device, a position estimation method executed by the position estimation device, and a computer program for position estimation will be described with reference to the drawings. The position estimation device collects on-board images showing features related to the travel of the vehicles from one or more vehicles capable of communication for a predetermined area, detects the predetermined features related to the travel of the vehicles shown in the collected on-board images, and estimates the positions of the features. In this case, the position estimation device detects a first position of the vehicle that has traveled through the predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section. Then, the position estimation device estimates a second position of the vehicle at a second time when the on-board image showing the predetermined features was generated based on the first position of the vehicle at the first time, and estimates the position of the predetermined feature based on the estimated second position.

[0016] The predetermined features to be detected include, for example, various road signs, various road markings, traffic lights, and other features related to vehicle travel.

[0017] FIG. 1 is a schematic diagram of a feature data collection system in which a position estimation device is implemented. In this embodiment, the feature data collection system 1 has at least one vehicle 2, at least one fixed sensor 3, and a server 4 which is an example of a position estimation device. Each vehicle 2 is connected to the server 4 via the wireless base station 6 and the communication network 5 by accessing, for example, a wireless base station 6 connected to the communication network 5 to which the server 4 is connected via a gateway (not shown). Each fixed sensor 3 is also connected to the communication network 5 via a gateway or the like, and is connected to the server 4 via the communication network 5. Note that, for simplification, only one vehicle 2 is illustrated in FIG. 1, but the feature data collection system 1 may have multiple vehicles 2. Similarly, only one fixed sensor 3 is illustrated in FIG. 1, but multiple fixed sensors 3 may be included. Furthermore, only one wireless base station 6 is illustrated in FIG. 1, but multiple wireless base stations 6 may be connected to the communication network 5.

[0018] 2 is a schematic diagram of the vehicle 2. The vehicle 2 has a camera 11, a GPS receiver 12, a wireless communication terminal 13, and a data acquisition device 14. The camera 11, the GPS receiver 12, the wireless communication terminal 13, and the data acquisition device 14 are communicatively connected via an in-vehicle network conforming to a standard such as a controller area network. The vehicle 2 may further have a navigation device (not shown) that searches for a planned travel route of the vehicle 2 and navigates the vehicle 2 so that the vehicle 2 travels along the planned travel route.

[0019] The camera 11 is an example of an on-board camera, and has a two-dimensional detector configured with an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The camera 11 is attached, for example, inside the cabin of the vehicle 2 so as to face the front of the vehicle 2. The camera 11 photographs the area in front of the vehicle 2 at a predetermined photographing period (for example, 1 / 30 to 1 / 10 seconds) and generates an image showing the area in front of the vehicle 2. The image obtained by the camera 11 is an example of an on-board image, and may be a color image or a gray image. Note that the vehicle 2 may be provided with a plurality of cameras 11 with different photographing directions or focal lengths.

[0020] Each time the camera 11 generates an image, it outputs the generated image to the data acquisition device 14 via the in-vehicle network.

[0021] The GPS receiver 12 is an example of a satellite positioning device, and receives GPS signals from GPS satellites at a predetermined cycle and measures the vehicle 2's own position based on the received GPS signals. The predetermined cycle at which the GPS receiver 12 measures the vehicle 2's own position may be different from the image capturing cycle of the camera 11. The GPS receiver 12 outputs positioning information representing the positioning result of the vehicle 2's own position based on the GPS signals to the data acquisition device 14 via the in-vehicle network at a predetermined cycle. The GPS receiver 12 may include an index value representing the accuracy of the measured position, such as the reception strength of the GPS signal or the number of satellites from which the GPS signal can be received, in the positioning information. The vehicle 2 may have a receiver conforming to a satellite positioning system other than the GPS receiver 12. In this case, the receiver may measure the vehicle 2's own position.

[0022] The wireless communication terminal 13 is an example of a communication unit, and is a device that executes wireless communication processing conforming to a predetermined wireless communication standard. For example, by accessing the wireless base station 6, the wireless communication terminal 13 is connected to the server 4 via the wireless base station 6 and the communication network 5. The wireless communication terminal 13 generates an uplink wireless signal including feature data including an image generated by the camera 11 received from the data acquisition device 14. The wireless communication terminal 13 transmits the uplink wireless signal to the wireless base station 6 to transmit the image and the like to the server 4. The wireless communication terminal 13 also receives a downlink wireless signal from the wireless base station 6, and passes a collection instruction from the server 4, which is included in the wireless signal, to the data acquisition device 14 or an electronic control unit (ECU, not shown) that controls the running of the vehicle 2.

[0023] 3 is a hardware configuration diagram of the data acquisition device. The data acquisition device 14 generates feature data based on the image generated by the camera 11. Furthermore, the data acquisition device 14 generates driving information representing the driving behavior of the vehicle 2. To this end, the data acquisition device 14 has a communication interface 21, a memory 22, and a processor 23.

[0024] The communication interface 21 is an example of an in-vehicle communication unit, and has an interface circuit for connecting the data acquisition device 14 to the in-vehicle network. That is, the communication interface 21 is connected to the camera 11, the GPS receiver 12, and the wireless communication terminal 13 via the in-vehicle network. Then, the communication interface 21 passes the received image to the processor 23 every time it receives an image from the camera 11. Also, the communication interface 21 passes the received positioning information to the processor 23 every time it receives positioning information from the GPS receiver 12. Furthermore, the communication interface 21 passes a collection instruction for feature data received from the server 4 via the wireless communication terminal 13 to the processor 23. Furthermore, the communication interface 21 outputs the feature data received from the processor 23 to the wireless communication terminal 13 via the in-vehicle network.

[0025] The memory 22 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 may further include other storage devices such as a hard disk device. The memory 22 stores various data used in the process related to the feature data generation executed by the processor 23 of the data acquisition device 14. Such data includes, for example, the identification information of the vehicle 2, the installation height of the camera 11, the shooting direction, the focal length, and the angle of view of the camera 11, and other parameters of the camera 11. The memory 22 may also store images received from the camera 11 and positioning information received from the GPS receiver 12 for a certain period of time. The memory 22 also stores information indicating an area to be the target of the generation and collection of feature data (hereinafter, sometimes referred to as the collection target area), which is specified in the instruction to collect feature data. The memory 22 may also store computer programs for implementing each process executed by the processor 23.

[0026] The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The processor 23 stores images received from the camera 11 and positioning information received from the GPS receiver 12 in the memory 22. Furthermore, the processor 23 executes processes related to the generation of feature data at predetermined intervals (for example, every 0.1 to 10 seconds) while the vehicle 2 is traveling.

[0027] As a process related to the generation of feature data, the processor 23 determines whether or not the vehicle position of the vehicle 2 represented by the positioning information received from the GPS receiver 12 is included in the collection target area. If the vehicle position is included in the collection target area, the processor 23 generates feature data based on the image received from the camera 11.

[0028] The feature data is data representing features related to the traveling of the vehicle. In this embodiment, the processor 23 includes, in the feature data, an image generated by the camera 11, the time when the image was generated, the position and traveling direction of the vehicle 2 indicated in the positioning information at that time, and parameters of the camera 11 such as the installation height, shooting direction, focal length, and angle of view of the camera 11. The processor 23 may acquire information representing the traveling direction of the vehicle 2 from an ECU or a direction sensor (not shown) of the vehicle 2. In addition, when the time when the image was generated is different from the time when the latest positioning information was obtained, the processor 23 acquires odometry information of the vehicle 2 used for dead reckoning, such as wheel speed, acceleration, and angular velocity, from the time when the latest positioning information was obtained to the time when the image was generated, from the ECU. The processor 23 may then correct the position and traveling direction of the vehicle 2 indicated in the latest positioning information with the odometry information, thereby obtaining the position and traveling direction of the vehicle 2 at the time when the image was generated. The odometry information is an example of behavior information representing the behavior of the vehicle 2. Then, every time the processor 23 generates feature data, it transmits the generated feature data to the server 4 via the wireless communication terminal 13. The processor 23 may include, in one piece of feature data, a plurality of images, the generation time of each image, and the position and traveling direction of the vehicle 2. The processor 23 may also transmit, separately from the feature data, the parameters of the camera 11 together with the identification information of the vehicle 2 to the server 4 via the wireless communication terminal 13.

[0029] Furthermore, the processor 23 generates driving information of the vehicle 2 from a predetermined timing (for example, the timing when the ignition switch of the vehicle 2 is turned on) onward, and transmits the driving information to the server 4 via the wireless communication terminal 13. The processor 23 includes in the driving information a series of positioning information obtained from the predetermined timing onward, the time when the position of the vehicle 2 was measured in each piece of positioning information, and odometry information obtained from the ECU. Furthermore, the processor 23 may include identification information of the vehicle 2 in the driving information and feature data.

[0030] The fixed sensor 3 is a sensor that is installed on or near a predetermined road section and is capable of detecting a vehicle 2 traveling on the predetermined road section. There is no particular restriction on the predetermined road section, but it is preferably a road section where the vehicle 2 that collects feature data is likely to travel. The fixed sensor 3 can be, for example, a camera (hereinafter, sometimes referred to as a surveillance camera to distinguish it from a camera mounted on the vehicle 2) that is installed so as to include the predetermined road section in its shooting range. In this case, the surveillance camera, which is the fixed sensor 3, is installed facing the predetermined road section on a support member that is installed so as to straddle the predetermined road section or on a pole that is installed adjacent to the predetermined road section. The surveillance camera generates an image showing the predetermined road section by photographing the predetermined road section at a predetermined cycle. Then, every time the surveillance camera generates a surveillance image, it transmits the generated image and the generation time of the image to the server 4 via the communication network 5. Note that the image generated by the surveillance camera is an example of a fixed sensor signal that detects a vehicle 2 traveling on a predetermined road section. Hereinafter, in order to distinguish between an in-vehicle image generated by the in-vehicle camera 11 and an image generated by a surveillance camera, an image generated by a surveillance camera may be referred to as a surveillance image.

[0031] The fixed sensor 3 may be a loop coil type vehicle detection sensor that uses a loop coil buried in the road surface of a predetermined road section. Alternatively, the fixed sensor 3 may be an ultrasonic vehicle detection sensor having an ultrasonic transmitter and receiver that is installed on the shoulder of a road or the like of a predetermined road section. Furthermore, the fixed sensor 3 may be a beacon device capable of bidirectional communication with an on-board device mounted on the vehicle 2. When the fixed sensor 3 is one of these sensors, the fixed sensor 3 transmits a vehicle detection signal indicating that the vehicle has been detected and the time when the vehicle was detected (hereinafter simply referred to as the detection time) to the server 4 via the communication network 5 every time the fixed sensor 3 detects a vehicle passing through the predetermined road section. The vehicle detection signal is another example of a fixed sensor signal.

[0032] Next, the server 4, which is an example of a position estimation device, will be described. 4 is a hardware configuration diagram of a server 4, which is an example of a position estimation device. The server 4 has a communication interface 31, a storage device 32, a memory 33, and a processor 34. The communication interface 31, the storage device 32, and the memory 33 are connected to the processor 34 via signal lines. The server 4 may further have input devices such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0033] The communication interface 31 is an example of a communication unit, and has an interface circuit for connecting the server 4 to the communication network 5. The communication interface 31 is configured to be able to communicate with the vehicle 2 via the communication network 5 and the wireless base station 6. That is, the communication interface 31 passes feature data or traveling information received from the vehicle 2 via the wireless base station 6 and the communication network 5 to the processor 34. The communication interface 31 also transmits a collection instruction received from the processor 34 to the vehicle 2 via the communication network 5 and the wireless base station 6. Furthermore, the communication interface 31 passes a fixed sensor signal received from the fixed sensor 3 via the communication network 5 to the processor 34.

[0034] The storage device 32 is an example of a storage unit, and includes, for example, a hard disk device or an optical recording medium and an access device therefor. The storage device 32 stores various data and information used in the position estimation process. For example, the storage device 32 stores a parameter set for identifying a classifier for detecting features from an image, a map to be updated, and identification information of each vehicle 2. Furthermore, the storage device 32 stores feature data received from each vehicle 2, travel information, and fixed sensor signals received from each fixed sensor 3. Furthermore, the storage device 32 stores, for each fixed sensor 3, detection position information indicating the position of the vehicle 2 that the fixed sensor 3 can detect. If the fixed sensor 3 is a surveillance camera, the detection position information includes real space position coordinates corresponding to each of a plurality of points on an image generated by the surveillance camera. Alternatively, the detection position information may include parameters related to the surveillance camera, such as the installation position, shooting direction, focal length, and angle of view of the surveillance camera. Furthermore, if the fixed sensor 3 is a vehicle detection sensor, the detection position information includes real space position coordinates of a position where the vehicle detection sensor can detect the vehicle 2. The detected position information may be transmitted from the fixed sensor 3 to the server 4 when the fixed sensor 3 is installed. Alternatively, the detected position information may be input via a user interface of the server 4. Furthermore, the storage device 32 may store a computer program for executing a position estimation process, which is executed on the processor 34.

[0035] The memory 33 is another example of a storage unit, and includes, for example, a non-volatile semiconductor memory and a volatile semiconductor memory. The memory 33 temporarily stores various data generated during execution of the position estimation process.

[0036] The processor 34 is an example of a control unit, and has one or more central processing units (CPUs) and their peripheral circuits. The processor 34 may further have other arithmetic circuits such as a logical operation unit or a numerical operation unit. The processor 34 generates an instruction to collect feature data representing a collection target area at predetermined intervals or at a timing designated by a user, and transmits the generated collection instruction to each vehicle 2 via the communication network 5 and the wireless base station 6. The processor 34 also executes a position estimation process.

[0037] 5 is a functional block diagram of the processor 34 related to the position estimation process. The processor 34 has a vehicle detection unit 41, a vehicle position estimation unit 42, a feature detection unit 43, and a feature position estimation unit 44. Each of these units in the processor 34 is a functional module realized by, for example, a computer program running on the processor 34. Alternatively, each of these units in the processor 34 may be a dedicated arithmetic circuit provided in the processor 34.

[0038] The vehicle detection unit 41 detects a first position of the vehicle 2 that has traveled along a predetermined road section at a first time based on the fixed sensor signal received from the fixed sensor 3.

[0039] When the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 detects the vehicle 2 that has traveled along a predetermined road section by inputting a surveillance image generated by the surveillance camera to a classifier. The vehicle detection unit 41 may use, for example, a so-called deep neural network (DNN) as such a classifier. As such a DNN, for example, a DNN having a convolutional neural network (CNN) type architecture such as Single Shot MultiBox Detector (SSD) or Faster R-CNN, or a DNN having an attention mechanism such as Vision Transformer, may be used. Alternatively, the vehicle detection unit 41 may use, as such a classifier, a classifier based on other machine learning methods such as a support vector machine (SVM) or adaBoost. Such a classifier is trained in advance according to a predetermined learning method such as backpropagation using a large number of teacher images in which vehicles are depicted, so as to detect vehicles from images.

[0040] The classifier may be further trained to detect a license plate attached to a vehicle. In this case, the vehicle detection unit 41 may identify the registration number of the detected vehicle by inputting the area in which the license plate is displayed on the surveillance image to a number classifier that has been trained in advance to identify the registration number. The vehicle detection unit 41 then identifies an identification number that matches the identified registration number among the identification numbers of each vehicle 2 stored in the storage device 32 or the memory 33. The vehicle detection unit 41 may use a CNN-type DNN or a DNN with an attention mechanism as the number classifier. The vehicle detection unit 41 then determines that the vehicle 2 having the identified identification number has been detected. Furthermore, the vehicle detection unit 41 sets the generation time of the surveillance image in which the detected vehicle 2 is displayed as the first time. Furthermore, the vehicle detection unit 41 specifies a point that is included in the area in which the vehicle 2 detected on the image is displayed or is closest to that area among the multiple points associated with the position coordinates in the real space, and sets the position coordinates in the real space corresponding to the specified point as the first position of the detected vehicle 2. Furthermore, the position of each pixel on the surveillance image has a one-to-one correspondence with the direction from the surveillance camera to the object depicted in that pixel. Therefore, the vehicle detection unit 41 may determine a first position of the detected vehicle 2 based on the direction from the surveillance camera corresponding to the position of the area in which the vehicle 2 is depicted on the surveillance image, and the surveillance camera parameters such as the installation position and shooting direction of the surveillance camera.

[0041] Furthermore, the vehicle detection unit 41 may refer to the travel information received from each vehicle 2 and identify, among each vehicle 2, a vehicle traveling at a position closest to the first position at the first time as the vehicle 2 that was at the first position at the first time. In this case, the vehicle detection unit 41 may omit detection of the registration number. Furthermore, the identifier may be trained in advance to identify the model of the detected vehicle. In this case, the vehicle detection unit 41 may identify, among each vehicle 2, a vehicle of the same model as the model detected at the first time as the vehicle 2 that was at the first position at the first time. In this case, the vehicle detection unit 41 may also omit detection of the registration number.

[0042] Furthermore, when the fixed sensor 3 is a vehicle detection sensor or a beacon device, the vehicle detection unit 41 determines the detection time included in the received vehicle detection signal as the first time, and the position detectable by the vehicle detection sensor as the first position. Then, the vehicle detection unit 41 refers to the travel information received from each vehicle 2, and identifies, among each vehicle 2, the vehicle 2 traveling at a position closest to the first position at the first time as the vehicle 2 that was at the first position at the first time.

[0043] The vehicle detection unit 41 notifies the vehicle position estimation unit 42 of the first time and the first position of the detected vehicle 2.

[0044] The vehicle position estimation unit 42 estimates the position (second position) of the vehicle 2 at the generation time (second time) of an image depicting a specified feature contained in the feature data received from the vehicle 2, based on the first position of the detected vehicle 2 at the first time.

[0045] The vehicle position estimation unit 42 calculates, as a position correction vector, a vector representing a difference between the position of the vehicle 2 at the first time included in the traveling information received from the detected vehicle 2 (hereinafter, for convenience of explanation, referred to as the first pre-correction position) toward the first position at the first time detected by the vehicle detection unit 41. Then, the vehicle position estimation unit 42 determines, as the second position, a position obtained by correcting the position of the vehicle 2 at the image generation time (second time) included in the feature data (hereinafter, for convenience of explanation, referred to as the second pre-correction position) with the position correction vector (i.e., a position obtained by adding the position correction vector to the second pre-correction position).

[0046] If the travel information does not include the first pre-correction position, the vehicle position estimation unit 42 may obtain the first pre-correction position by dividing the positions of the vehicle 2 at times before and after the first time, which are included in the travel information, by the difference between the times before and after the first time and the first time. If the travel information includes odometry information, the first pre-correction position may be obtained by correcting the position of the vehicle 2 at a reference time before or after the first time using odometry information from the reference time to the first time.

[0047] When the server 4 receives a plurality of feature data from the detected vehicle 2, the vehicle position estimation unit 42 performs the above-mentioned process for each feature data to estimate, for each feature data, the second position of the vehicle 2 at the second time when the image included in the feature data was generated. Also, when the feature data includes a plurality of images and the generation times of the images, the vehicle position estimation unit 42 performs the above-mentioned process for each image to estimate, for each image, the second position of the vehicle 2 at the second time when the image was generated.

[0048] The vehicle position estimation unit 42 notifies the feature position estimation unit 44 of the estimated second position.

[0049] The feature detection unit 43 detects a predetermined feature to be detected from an image included in the feature data received from the detected vehicle 2. To this end, the feature detection unit 43 inputs the image into a feature detection classifier to identify an object region in which a predetermined feature is shown in the image and the type of the predetermined feature. The feature detection unit 43 may use a DNN having a CNN-type architecture or a DNN having an attention mechanism as such a classifier. Alternatively, the feature detection unit 43 may use a classifier based on other machine learning methods such as SVM or adaBoost as such a classifier. Such a classifier is trained in advance according to a predetermined learning method such as an error backpropagation method using a large number of teacher images in which the predetermined feature is shown so as to detect the predetermined feature from the image.

[0050] The feature detection unit 43 notifies the feature position estimation unit 44 of the second time when the image in which the specified feature is detected was generated, information representing the position and size of the object area in which the specified feature is represented on the image, and the type of the specified feature.

[0051] The feature position estimation unit 44 estimates the position of the detected feature based on the second position of the vehicle 2 when the image showing the feature was generated.

[0052] As described above with respect to the surveillance camera, the position of each pixel on the image corresponds one-to-one to the orientation from the camera 11 to the object represented by that pixel. Therefore, when the detected feature is a feature on the road surface, such as a road marking or a three-dimensional structure placed on the road surface, the feature position estimation unit 44 can estimate the position of the feature in a camera coordinate system based on the camera 11. In this case, the feature position estimation unit 44 may estimate the position of the feature based on the orientation from the camera 11 corresponding to the reference point in the object area, the traveling direction of the vehicle 2 at the second time when the image was generated, which are included in the feature data, and the parameters of the camera 11, such as the shooting direction of the camera 11 and the installation height on the vehicle 2. Furthermore, the feature position estimation unit 44 can estimate the position of the detected feature by, for example, performing an affine transformation on the position of the feature so as to convert the position of the feature in the camera coordinate system into a position in a coordinate system of a real space in which the position of the camera 11 is the second position. The reference point in the object area can be the center of gravity of the object area. Furthermore, if the feature is a three-dimensional structure installed on the road surface, the position of the bottom edge of the object region is presumed to represent the position where the feature contacts the road surface, and therefore, in this case, the reference point can be any position on the bottom edge of the object region.

[0053] Furthermore, when the same feature is detected from a plurality of images generated at different times, the feature position estimation unit 44 may estimate the position of the feature according to a so-called Structure from Motion technique. In this case, the feature position estimation unit 44 estimates the position of the feature by triangulation based on the second position and traveling direction of the vehicle 2 at the time of generating each image, the position of the object region in which the feature is depicted on each image, and the parameters of the camera 11.

[0054] The feature position estimation unit 44 may associate the same feature depicted in each of a plurality of images generated at different times between the images according to a tracking method such as KLT tracking.

[0055] The feature position estimation unit 44 stores information representing the type and estimated position of a detected specific feature in the storage device 32. Alternatively, the feature position estimation unit 44 may output the information representing the type and estimated position of the detected feature to another device via the communication interface 31.

[0056] FIG. 6 is a diagram for explaining an outline of the estimation of the position of a feature according to this embodiment. As shown in FIG. 6, in this example, at a first time t1 when the vehicle 2 is detected by the fixed sensor 3, a first position P1 of the vehicle 2 is detected based on a fixed sensor signal. Meanwhile, at this first time t1, the position (first pre-correction position) Pg1 of the vehicle 2 is detected by the GPS receiver 12 in the vehicle 2. Therefore, a position correction vector C=|P1-Pg1| is obtained. Using this position correction vector C, the position (second pre-correction position) Pg2 of the vehicle 2 measured by the GPS receiver 12 at a second time t2 when the on-board image IMG is generated by the camera 11 mounted on the vehicle 2 is corrected, and the second position P2 of the vehicle 2 at the time t2 is estimated. Then, based on the second position P2, the position Ps in the real space of the feature S detected from the on-board image IMG is estimated.

[0057] 7 is an operational flowchart of the position estimation process in the server 4. The processor 34 of the server 4 may execute the position estimation process in accordance with the operational flowchart shown below.

[0058] The vehicle detection unit 41 of the processor 34 detects a first position of the vehicle 2 that has traveled through a predetermined road section at a first time based on the fixed sensor signal received from the fixed sensor 3 (step S101).

[0059] In addition, the vehicle position estimation unit 42 of the processor 34 estimates a second position of the vehicle 2 at a second time when an in-vehicle image depicting the specified feature to be detected is generated based on the first position of the detected vehicle 2 at the first time (step S102).

[0060] Furthermore, the feature detection unit 43 of the processor 34 detects a predetermined feature from the vehicle-mounted image generated at the second time and included in the feature data received from the detected vehicle 2 (step S103).

[0061] Then, the feature position estimation unit 44 of the processor 34 estimates the position of the detected feature based on the second position of the vehicle 2 when the vehicle-mounted image showing the detected feature was generated (step S104). Then, the processor 34 ends the position estimation process.

[0062] The processor 34 may generate or update a map based on the type and position of the detected feature. For example, the processor 34 adds information representing the position and type of each detected feature to the map to be generated. If a feature of the same type as the detected feature does not exist within a predetermined distance from the position of the detected feature in the map to be updated, the processor 34 adds information representing the position and type of the detected feature to the map to be updated. If the position of the detected feature in the map to be updated includes information on a feature of a type different from the type of the detected feature, the processor 34 rewrites the type of the feature at that position to the type of the detected feature. If a feature shown at a predetermined point on the map to be updated is not detected from the feature data from each vehicle 2 that passed the predetermined point during a certain period of time, the processor 34 deletes the information on the feature at the predetermined point from the map.

[0063] The processor 34 may distribute the generated or updated map to vehicles that use the map for autonomous driving control or driving assistance via the communication network 5 and the wireless base station 6.

[0064] As described above, this position estimation device uses the vehicle position detected by a fixed sensor capable of accurately detecting the vehicle position to estimate the vehicle position when the vehicle-mounted image is generated. Therefore, this position estimation device can accurately estimate the vehicle position when the vehicle-mounted image is generated. As a result, this position estimation device can accurately estimate the position of the feature detected from the vehicle-mounted image.

[0065] According to a modified example, the vehicle position estimation unit 42 may estimate the position of the vehicle 2 at the time when the vehicle-mounted image was generated only for an on-board image in which a predetermined feature has been detected by the feature detection unit 43 among the series of on-board images received from the vehicle 2. This reduces the amount of calculation.

[0066] According to another modification, the vehicle position estimation unit 42 may use odometry information included in the travel information to determine the moving direction and the moving amount of the vehicle 2 from the first time when the vehicle 2 is detected by the fixed sensor 3 to the second time when the in-vehicle image showing the predetermined feature is generated. The vehicle position estimation unit 42 may then estimate the second position of the vehicle at the second time by correcting the first position with the moving direction and the moving amount. In this way, even if the GPS receiver 12 mounted on the vehicle 2 cannot measure the position of the vehicle 2 at the second time, the vehicle position estimation unit 42 can estimate the second position of the vehicle 2.

[0067] According to another modification, when the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 may detect the position of the vehicle 2 from each of a plurality of surveillance images generated at different times. The vehicle position estimation unit 42 may estimate the second position of the vehicle 2 at the second time by taking the position of the vehicle 2 detected from the surveillance image generated at the time closest to the second time among the plurality of surveillance images as the first position. In this way, the surveillance image, which is a fixed sensor signal, generated at the time closest to the generation of the vehicle-mounted image showing a predetermined feature is used to estimate the position of the vehicle 2, so that the vehicle position estimation unit 42 can further improve the estimation accuracy of the second position of the vehicle 2.

[0068] Furthermore, when a plurality of fixed sensors 3 are installed, the vehicle detection unit 41 may detect the vehicle 2 by executing the above process for each fixed sensor 3. In this case, when the same vehicle 2 is detected from fixed sensor signals by two or more fixed sensors 3, the vehicle position estimation unit 42 may estimate the second position of the vehicle 2 by setting the position of the vehicle 2 closer to the second pre-correction position included in the travel information as the first position among the positions of the individual detected vehicles 2. In this way, the position of the vehicle 2 detected by the fixed sensor 3 closer to the second pre-correction position of the vehicle 2 when the vehicle-mounted image showing the predetermined feature was generated is used for estimating the second position, so that the vehicle position estimation unit 42 can further improve the estimation accuracy of the second position.

[0069] According to yet another modification, the processor 23 of the data acquisition device 14 mounted on the vehicle 2 may execute the processing of the feature detection unit 43 in the above embodiment. In this case, the processor 23 may include, in the feature data, information representing the generation time of the on-board image at which the specific feature is detected, the position of the vehicle 2 measured by the GPS receiver 12 at the generation time, the traveling direction of the vehicle 2, the parameters of the camera 11, and the position and range of the object area in which the specific feature is displayed on the on-board image. The feature position estimation unit 44 may then estimate the position of the specific feature detected from the on-board image by referring to the information included in the feature data.

[0070] According to yet another modification, when the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 may estimate a region that is a blind spot of the vehicle 2 based on the surveillance image. When the estimated position of a feature detected from the on-board image is included in the blind spot region, the processor 34 may determine that the feature has been erroneously detected, and may not use information about the feature for generating or updating the map. For example, when not only the vehicle 2 but also other vehicles (hereinafter referred to as surrounding vehicles) are detected from the surveillance image, the vehicle detection unit 41 may specify, as the blind spot region, the region that is blocked by the surrounding vehicles as viewed from the vehicle 2 based on the positional relationship between the vehicle 2 and the surrounding vehicles.

[0071] Also, when a beacon device is used as the fixed sensor 3, the data acquisition device 14 of the vehicle 2 may detect the position of the vehicle 2. In this case, the memory 22 of the data acquisition device 14 stores in advance a position where a beacon signal from the beacon device can be received. Then, the processor 23 of the data acquisition device 14 may detect the first time when the beacon signal emitted from the beacon device is detected by the vehicle-mounted device mounted on the vehicle 2, and may detect the first position where the beacon signal can be received. Then, the processor 23 may include the first time and the first position, or a position correction vector, in the traveling information transmitted to the server 4. Furthermore, in this case, the processor 23 of the data acquisition device 14 may estimate the position of the detected feature by executing the processing of each part of the processor 34 of the server 4 in the above embodiment. Then, the processor 34 may include the position of the detected feature in the feature data. In this case, the data acquisition device 14 is another example of a position estimation device.

[0072] A computer program that causes a computer to realize the functions of each unit of the processor of the position estimation device according to each of the above embodiments or modifications may be provided in a form stored in a computer-readable recording medium. The computer-readable recording medium may be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0073] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0074] 1. Geophysical data collection system 2 Vehicles 11 Camera 12 GPS receiver 13 Wireless communication terminals 14 Data acquisition device 21 Communication Interface 22 Memory 23 Processors 3 Fixed Sensor 4 Server 31 Communication Interface 32 Storage Device 33 Memory 34 processors 41 Vehicle detection unit 42 Vehicle position estimation unit 43 Feature detection unit 44 Feature position estimation section 5. Communication Network 6 Radio base stations

Claims

1. a vehicle detection unit that detects a first position of a vehicle that has traveled along a predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section; a vehicle position estimation unit that estimates a second position of the vehicle at a second time when an on-board image showing a predetermined feature is generated by an on-board camera mounted on the vehicle, based on the first position of the vehicle at the first time; a feature position estimating unit that estimates a position of the predetermined feature in real space based on the second position; A position estimation device having the following:

2. the fixed sensor is a surveillance camera, the surveillance camera generates a surveillance image showing the predetermined road section at a predetermined period; the vehicle detection unit detects a position of the vehicle from each of the plurality of surveillance images generated at different times; 2. The position estimation device according to claim 1, wherein the vehicle position estimation unit estimates the second position by taking the position of the vehicle detected from a surveillance image generated at a time closest to the second time among the plurality of surveillance images as the first position.

3. 3. The position estimation device according to claim 1, wherein the vehicle position estimation unit estimates the second position by correcting a second pre-correction position of the vehicle measured by the satellite positioning device at the second time in accordance with a difference between a first pre-correction position of the vehicle measured by a satellite positioning device mounted on the vehicle at the first time and the first position.

4. a feature detection unit that detects the predetermined feature from the vehicle-mounted image generated at the second time, The position estimation device according to claim 1 , wherein the feature position estimation unit estimates the position of the predetermined feature in real space based on the position of the predetermined feature on the vehicle-mounted image and the second position.

5. Detecting a first position of a vehicle that has traveled along a predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section; estimating a second position of the vehicle at a second time when an on-board image showing a predetermined feature was generated by an on-board camera mounted on the vehicle based on the first position of the vehicle at the first time; estimating a position of the predetermined feature in real space based on the second position; A position estimation method comprising:

6. Detecting a first position of a vehicle that has traveled along a predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section; estimating a second position of the vehicle at a second time when an on-board image showing a predetermined feature was generated by an on-board camera mounted on the vehicle based on the first position of the vehicle at the first time; estimating a position of the predetermined feature in real space based on the second position; A computer program for location estimation that causes a computer to execute the above steps.

Images