Position estimation device, position estimation method and position estimation computer program

By detecting the vehicle position by fixed sensors and combining the on-board camera and satellite positioning information, the problem of insufficient feature position detection accuracy in the on-board camera images is solved, and the map update accuracy of the autonomous driving system is improved.

CN119027636APending Publication Date: 2024-11-26WOVEN BY TOYOTA INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410632792.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-05-21
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, the detection accuracy of feature positions in the images generated by the on-board camera is insufficient, and the vehicle position cannot be effectively corrected, resulting in low map update accuracy of the autonomous driving system.

Method used

Fixed sensors are used to detect the vehicle position, and the images generated by the on-board camera are combined with satellite positioning information to estimate the characteristic position, and the correction information is used to improve the detection accuracy.

Benefits of technology

It realizes accurate estimates of feature positions in the image of the on-board camera, improves vehicle position detection accuracy, and enhances the map update capability of the autonomous driving system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027636A_ABST
    Figure CN119027636A_ABST
Patent Text Reader

Abstract

A position estimation device is provided with: a vehicle detection unit (41) that detects, on the basis of a fixed sensor (3) provided on a prescribed road section or near the prescribed road section, a first position of a vehicle (2) traveling over the prescribed road section at a first time; a vehicle position estimation unit (42) that estimates, on the basis of the first position of the vehicle (2) at the first time, a second position of the vehicle (2) at a second time at which an in-vehicle camera (11) mounted on the vehicle (2) generates an in-vehicle image representing a predetermined feature; and a feature position estimation unit (44) that estimates the position of a predetermined feature in the real space on the basis of the estimated second position.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a position estimating device, a position estimating method and a position estimating computer program for estimating the position of a predetermined feature represented in an image generated by a vehicle-mounted camera. Background Art

[0002] In order to generate or update a high-precision map that an autonomous driving system of a vehicle refers to for autonomously controlling the vehicle, it is required to accurately detect the positions of features that are located on or around a road and are associated with the travel of the vehicle. Therefore, a technology for detecting features based on an image generated by a camera mounted on a vehicle actually traveling on the road has been proposed (see Japanese Patent No. 7126629).

[0003] The information integration device disclosed in Japanese Patent No. 7126629 identifies one or more first features around the mobile body based on sensor information obtained from a sensor possessed by the mobile body, and generates first feature information representing the first feature. In addition, the information integration device obtains second feature information representing one or more second features recognized as existing around the mobile body from an external device, or from both the external device and map information, and determines the same feature from the one or more first features represented by the first feature information and the one or more second features represented by the second feature information. Then, the information integration device generates correction information used to correct at least one of the first feature information and the second feature information based on the position difference between the first feature and the second feature determined as the same feature, corrects at least one of the first feature information and the second feature information using the generated correction information, and then integrates the first feature information and the second feature information to generate integrated feature information. Summary of the invention

[0004] When the detection accuracy of the vehicle position is insufficient, the accuracy of the position of the feature detected from the vehicle-mounted image obtained by the vehicle-mounted camera may also be insufficient. In the above-mentioned technology, in order to generate correction information, information related to the same feature must be included in the first feature information and the second feature information. However, the features expressed in the sensor information obtained by the sensor possessed by the mobile body are not necessarily always expressed in the feature information obtained from the external device. Therefore, it is sought to estimate the position of the feature expressed in the vehicle-mounted image with high accuracy.

[0005] Therefore, an object of the present invention is to provide a position estimating device capable of accurately estimating the position of a feature represented in an on-vehicle image obtained by a camera mounted on a vehicle.

[0006] According to one embodiment, a position estimation device is provided. The position estimation device comprises: a vehicle detection unit that detects a first position of a vehicle traveling on a prescribed road section at a first moment based on a fixed sensor disposed on or near a prescribed road section; a vehicle position estimation unit that estimates a second position of the vehicle at a second moment when a vehicle-mounted image showing a prescribed feature is generated by a vehicle-mounted camera mounted on the vehicle based on the first position of the vehicle at the first moment; and a feature position estimation unit that estimates the position of a prescribed feature in real space based on the estimated second position.

[0007] In the position estimation device, the fixed sensor is preferably a monitoring camera, and the monitoring camera generates a monitoring image showing a predetermined road section at a predetermined period. Furthermore, the vehicle detection unit preferably detects the position of the vehicle from a plurality of monitoring images generated at different times, and the vehicle position estimation unit uses the position of the vehicle detected from a monitoring image generated at a time closest to the second time among the plurality of monitoring images as the first position, thereby estimating the second position.

[0008] In addition, the vehicle position estimating unit preferably estimates the second position by correcting the second pre-correction position of the vehicle measured by the satellite positioning device at the second time corresponding to the difference between the first pre-correction position of the vehicle measured by the satellite positioning device mounted on the vehicle at the first time and the first position.

[0009] Preferably, the position estimation device further comprises a feature detection unit for detecting a predetermined feature from the vehicle-mounted image generated at the second time. Preferably, the feature position estimation unit estimates the position of the predetermined feature in real space based on the position of the predetermined feature on the vehicle-mounted image and the second position.

[0010] According to another embodiment, a position estimation method is provided. The position estimation method includes: detecting a first position of a vehicle traveling on a predetermined road section at a first moment based on a fixed sensor installed on or near a predetermined road section; estimating a second position of the vehicle at a second moment when a vehicle-mounted image showing a predetermined feature is generated by a vehicle-mounted camera mounted on the vehicle based on the first position of the vehicle at the first moment; and estimating the position of the predetermined feature in real space based on the estimated second position.

[0011] According to another embodiment, a computer program product for position estimation is provided. The computer program product for position estimation includes instructions for causing a computer to execute the following contents: based on a fixed sensor installed on or near a specified road section, a first position of a vehicle traveling on a specified road section at a first moment is detected; based on the first position of the vehicle at the first moment, a second position of the vehicle at a second moment when a vehicle-mounted image showing a specified feature is generated by a vehicle-mounted camera mounted on the vehicle; and based on the estimated second position, a position of the specified feature in real space is estimated.

[0012] The position estimating device according to the present disclosure achieves an effect of being able to accurately estimate the position of a feature represented in an in-vehicle image obtained by a camera mounted on a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of the characteristic data collection system equipped with a position estimation device.

[0014] Figure 2 This is a schematic diagram of the vehicle.

[0015] Figure 3 This is a hardware configuration diagram of the data acquisition device.

[0016] Figure 4 This is a hardware configuration diagram of a server as an example of a position estimating device.

[0017] Figure 5 This is a functional block diagram of a server processor related to position estimation processing.

[0018] Figure 6 It is a diagram for explaining the outline of the estimated feature position.

[0019] Figure 7 This is an action flow chart of the position estimation process. DETAILED DESCRIPTION

[0020] Hereinafter, a position estimation device, a position estimation method executed in the position estimation device, and a computer program for position estimation will be described with reference to the accompanying drawings. The position estimation device collects vehicle-mounted images representing features associated with the travel of the vehicle from one or more vehicles capable of communication in a specified area, detects specified features associated with the travel of the vehicle represented in the collected vehicle-mounted images, and estimates the position of the feature. At this time, the position estimation device detects a first position of a vehicle traveling on a specified road section at a first moment based on a fixed sensor provided on or near a specified road section. Then, the position estimation device estimates a second position of the vehicle at a second moment when a vehicle-mounted image representing the specified feature is generated based on the first position of the vehicle at the first moment, and estimates the position of the specified feature based on the estimated second position.

[0021] It should be noted that the prescribed features to be detected include, for example, various road markings, various road signs, traffic lights, and other features associated with the travel of the vehicle.

[0022] Figure 1 1 is a schematic diagram of a feature data collection system equipped with a position estimation device. In this embodiment, the feature data collection system 1 has at least one vehicle 2, at least one fixed sensor 3, and a server 4 as an example of a position estimation device. Each vehicle 2 is connected to the server 4 via the wireless base station 6 and the communication network 5 by, for example, accessing a communication network 5 connected to the server 4 and a wireless base station 6 connected via a gateway (not shown). In addition, each fixed sensor 3 is also connected to the communication network 5 via a gateway, etc., and is connected to the server 4 via the communication network 5. In addition, in Figure 1 In the figure, only one vehicle 2 is shown for simplicity, but the characteristic data collection system 1 may have a plurality of vehicles 2. Figure 1 In the figure, only one fixed sensor 3 is shown, but a plurality of fixed sensors 3 may be provided. Figure 1 Although only one wireless base station 6 is shown in the figure, a plurality of wireless base stations 6 may be connected to the communication network 5 .

[0023] Figure 2 1 is a schematic diagram of a vehicle 2. The vehicle 2 includes a camera 11, a GPS receiver 12, a wireless communication terminal 13, and a data acquisition device 14. The camera 11, the GPS receiver 12, the wireless communication terminal 13, and the data acquisition device 14 can be connected to each other via an in-vehicle network based on the controller area network standard. In addition, the vehicle 2 may also include a navigation device (not shown) that searches for a planned driving route of the vehicle 2 and navigates the vehicle 2 in a manner that the vehicle 2 drives along the planned driving route.

[0024] The camera 11 is an example of a vehicle-mounted camera, and has a two-dimensional detector composed of a matrix of photoelectric conversion elements sensitive to visible light, such as CCD or C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. And the camera 11 is installed in the vehicle compartment of the vehicle 2, for example, in a manner facing the front of the vehicle 2. And the camera 11 photographs the area in front of the vehicle 2 at a predetermined shooting cycle (for example, 1 / 30 second to 1 / 10 second) to generate an image showing the area in front. The image obtained by the camera 11 is an example of a vehicle-mounted image, and can be a color image or a grayscale image. In addition, a plurality of cameras 11 with different shooting directions or focal distances can be set in the vehicle 2.

[0025] Every time the camera 11 generates an image, it outputs the generated image to the data acquisition device 14 via the in-vehicle network.

[0026] The GPS receiver 12 is an example of a satellite positioning device, which receives GPS signals from GPS satellites at predetermined intervals and locates the position of the vehicle 2 based on the received GPS signals. In addition, the predetermined period for the GPS receiver 12 to locate the position of the vehicle 2 may be different from the shooting period of the camera 11. Then, the GPS receiver 12 outputs the positioning information indicating the positioning result of the vehicle 2 based on the GPS signal to the data acquisition device 14 via the in-vehicle network at predetermined intervals. The GPS receiver 12 may include the reception strength of the GPS signal or the number of satellites that successfully received the GPS signal in the positioning information, which are index values ​​indicating the accuracy of the located position. In addition, the vehicle 2 may also have a receiver based on a satellite positioning system other than the GPS receiver 12. In this case, it is sufficient as long as the receiver can locate the position of the vehicle 2.

[0027] The wireless communication terminal 13 is an example of a communication unit, and is a device that performs wireless communication processing based on a prescribed wireless communication standard, and is connected to the server 4 via the wireless base station 6 and the communication network 5 by, for example, accessing the wireless base station 6. In addition, the wireless communication terminal 13 generates an uplink wireless signal containing feature data, etc., which contains an image generated by the camera 11 and received from the data acquisition device 14. Then, the wireless communication terminal 13 transmits the image, etc. to the server 4 by transmitting the uplink wireless signal to the wireless base station 6. In addition, the wireless communication terminal 13 receives a downlink wireless signal from the wireless base station 6, and transmits the collection instruction, etc. from the server 4 contained in the wireless signal to the data acquisition device 14 or an electronic control unit (ECU, not shown) that controls the driving of the vehicle 2.

[0028] Figure 3The data acquisition device 14 generates characteristic data based on the image generated by the camera 11. In addition, the data acquisition device 14 generates travel information indicating the travel behavior of the vehicle 2. To this end, the data acquisition device 14 has a communication interface 21, a memory 22, and a processor 23.

[0029] The communication interface 21 is an example of an in-vehicle communication unit, and has an interface circuit for connecting the data acquisition device 14 to the in-vehicle network. That is, the communication interface 21 is connected to the camera 11, the GPS receiver 12, and the wireless communication terminal 13 via the in-vehicle network. And, each time the communication interface 21 receives an image from the camera 11, it transmits the received image to the processor 23. In addition, each time the communication interface 21 receives positioning information from the GPS receiver 12, it transmits the received positioning information to the processor 23. In addition, the communication interface 21 transmits the collection instruction of the feature data received from the server 4 via the wireless communication terminal 13 to the processor 23. In addition, the communication interface 21 outputs the feature data received from the processor 23 to the wireless communication terminal 13 via the in-vehicle network.

[0030] The memory 22 includes, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 may also include other storage devices such as a hard disk device. Furthermore, the memory 22 stores various data used in the processing associated with the generation of feature data executed by the processor 23 of the data acquisition device 14. These data include, for example, identification information of the vehicle 2, the setting height of the camera 11, the shooting direction, the focal length and the viewing angle of the camera 11, and other parameters of the camera 11. In addition, the memory 22 may also store images received from the camera 11 and positioning information received from the GPS receiver 12 for a certain period of time. In addition, the memory 22 stores information indicating the generation of feature data and the area to be collected (hereinafter sometimes referred to as the collection object area) specified in the collection instruction of feature data. Furthermore, the memory 22 may also store computer programs, etc. for implementing the various processes executed by the processor 23.

[0031] The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may also have other operation circuits such as a logic operation unit, a numerical operation unit, or an image processing unit. In addition, the processor 23 stores the image received from the camera 11 and the positioning information received from the GPS receiver 12 in the memory 22. In addition, the processor 23 performs processing associated with feature data generation at predetermined cycles (e.g., 0.1 seconds to 10 seconds) while the vehicle 2 is traveling.

[0032] As a process associated with generating feature data, the processor 23 determines, for example, whether the vehicle position of the vehicle 2 indicated by the positioning information received from the GPS receiver 12 is included in the collection target area. Then, if the vehicle position is included in the collection target area, the processor 23 generates feature data based on the image received from the camera 11.

[0033] The feature data is data indicating features associated with the travel of the vehicle. In the present embodiment, the processor 23 includes in the feature data an image generated by the camera 11, the time when the image is generated, the position and travel direction of the vehicle 2 indicated by the positioning information at the time, and the parameters of the camera 11 such as the installation height, shooting direction, focal length and viewing angle of the camera 11. In addition, the processor 23 only needs to be able to obtain information indicating the travel direction of the vehicle 2 from the ECU or the azimuth sensor (not shown) of the vehicle 2. In addition, when the time when the image is generated is different from the time when the latest positioning information is obtained, the processor 23 obtains the wheel speed, acceleration and angular velocity from the time when the latest positioning information is obtained to the time when the image is generated. The distance measurement information of the vehicle 2 used in the dead reckoning is obtained from the ECU. In addition, the processor 23 can also calculate the position and travel direction of the vehicle 2 at the time when the image is generated by correcting the position and travel direction of the vehicle 2 shown in the latest positioning information according to the distance measurement information. In addition, the distance measurement information is an example of behavior information indicating the behavior of the vehicle 2. Furthermore, each time the processor 23 generates feature data, it transmits the generated feature data to the server 4 via the wireless communication terminal 13. In addition, the processor 23 may include a plurality of images, the generation time of each image, and the position and travel direction of the vehicle 2 in one feature data. In addition, the processor 23 may also transmit the parameters of the camera 11 together with the identification information of the vehicle 2 to the server 4 via the wireless communication terminal 13 independently of the feature data.

[0034] In addition, the processor 23 generates the driving information of the vehicle 2 after the predetermined timing (for example, the timing when the ignition switch of the vehicle 2 is turned on), and transmits the driving information to the server 4 via the wireless communication terminal 13. The processor 23 includes in the driving information a series of positioning information obtained after the predetermined timing, the time when the position of the vehicle 2 in each positioning information is measured, and the distance measurement information obtained from the ECU. In addition, the processor 23 may also include the identification information of the vehicle 2 in the driving information and the feature data.

[0035] The fixed sensor 3 is a sensor that is installed on or near a predetermined road section and can detect a vehicle 2 traveling on the predetermined road section. The predetermined road section is not particularly limited, but it is preferably a road section where the vehicle 2 that collects characteristic data is likely to travel. The fixed sensor 3 may be, for example, a camera that is installed so as to include the predetermined road section in the shooting range (hereinafter, sometimes referred to as a monitoring camera to distinguish it from a camera mounted on the vehicle 2). In this case, the monitoring camera as the fixed sensor 3 is installed facing the predetermined road section on a support member that is installed so as to cross the predetermined road section or a pole that is installed adjacent to the predetermined road section. The monitoring camera generates an image representing the predetermined road section by shooting the predetermined road section at a predetermined cycle. And, each time the monitoring camera generates a monitoring image, the generated image and the generation time of the image are sent to the server 4 via the communication network 5. In addition, the image generated by the monitoring camera is an example of a fixed sensor signal that detects the vehicle 2 traveling on the predetermined road section. In the following, in order to distinguish the vehicle-mounted image generated by the vehicle-mounted camera 11 from the image generated by the monitoring camera, the image generated by the monitoring camera is sometimes referred to as a monitoring image.

[0036] In addition, the fixed sensor 3 may also be a toroidal coil type vehicle detection sensor using a toroidal coil buried in the road surface of a specified road section. Alternatively, the fixed sensor 3 may also be an ultrasonic vehicle detection sensor provided on the shoulder of a specified road section or the like and having an ultrasonic transmitter and receiver. In addition, the fixed sensor 3 may also be a beacon device capable of bidirectional communication with an onboard device mounted on the vehicle 2. In the case where the fixed sensor 3 is one of these sensors, each time the fixed sensor 3 detects a vehicle passing through a specified road section, a vehicle detection signal indicating that the vehicle has been detected and the time when the vehicle has been detected (hereinafter referred to as the detection time) is sent to the server 4 via the communication network 5. The vehicle detection signal is another example of a fixed sensor signal.

[0037] Next, the server 4 as an example of the position estimating device will be described. Figure 4 : is a hardware configuration diagram of a server 4 as an example of a position estimation device. The server 4 has a communication interface 31, a storage device 32, a memory 33, and a processor 34. The communication interface 31, the storage device 32, and the memory 33 are connected to the processor 34 via a signal line. The server 4 may also have an input device such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0038] The communication interface 31 is an example of a communication unit, and has an interface circuit for connecting the server 4 to the communication network 5. Furthermore, the communication interface 31 is configured to be able to communicate with the vehicle 2 via the communication network 5 and the wireless base station 6. That is, the communication interface 31 transmits the feature data or driving information received from the vehicle 2 via the communication network 5 and the wireless base station 6 to the processor 34. In addition, the communication interface 31 transmits the collection instruction received from the processor 34 to the vehicle 2 via the communication network 5 and the wireless base station 6. In addition, the communication interface 31 transmits the fixed sensor signal received from the fixed sensor 3 via the communication network 5 to the processor 34.

[0039] The storage device 32 is an example of a storage unit, and for example, includes a hard disk device or an optical storage medium and an access device thereto. In addition, the storage device 32 stores various data and information used in the position estimation process. For example, the storage device 32 stores a parameter group for determining an identifier for detecting features from an image, a map to be updated, and identification information of each vehicle 2. In addition, the storage device 32 stores feature data received from each vehicle 2, driving information, and fixed sensor signals received from each fixed sensor 3. In addition, the storage device 32 stores detection position information indicating the position of the vehicle 2 that the fixed sensor 3 can detect for each fixed sensor 3. In the case where the fixed sensor 3 is a surveillance camera, the detection position information includes the position coordinates of the real space corresponding to each of the multiple points on the image generated by the surveillance camera. Alternatively, the detection position information may also include parameters related to the surveillance camera such as the installation position, shooting direction, focal length, and viewing angle of the surveillance camera. In addition, in the case where the fixed sensor 3 is a vehicle detection sensor, the position coordinates of the real space for the position of the vehicle 2 that the vehicle detection sensor can detect are included. In addition, the detection position information may be sent from the fixed sensor 3 to the server 4 when the fixed sensor 3 is installed. Alternatively, the detected position information may be input via a user interface of the server 4. Furthermore, the storage device 32 may store a computer program executed by the processor 34 for executing the position estimation process.

[0040] The memory 33 is another example of a storage unit, and includes, for example, a nonvolatile semiconductor memory and a volatile semiconductor memory, and temporarily stores various data generated during the execution of the position estimation process.

[0041] The processor 34 is an example of a control unit, and has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 34 may also have other operation circuits such as a logic operation unit or a numerical operation unit. In addition, the processor 34 generates a collection instruction representing the characteristic data of the collection target area at a predetermined period or at a timing specified by the user, and sends the generated collection instruction to each vehicle 2 via the communication network 5 and the wireless base station 6. In addition, the processor 34 performs a position estimation process.

[0042] Figure 5 1 is a functional block diagram of the processor 34 related to the position estimation process. The processor 34 includes a vehicle detection unit 41, a vehicle position estimation unit 42, a feature detection unit 43, and a feature position estimation unit 44. The above-mentioned parts of the processor 34 are, for example, functional modules implemented by a computer program running on the processor 34. Alternatively, the above-mentioned parts of the processor 34 may also be dedicated computing circuits provided in the processor 34.

[0043] The vehicle detection unit 41 detects a first position of the vehicle 2 traveling on a predetermined road section at a first time, based on the fixed sensor signal received from the fixed sensor 3 .

[0044] In the case where the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 inputs the surveillance image generated by the surveillance camera into the recognizer, thereby detecting the vehicle 2 traveling on the specified road section. The vehicle detection unit 41 can use the so-called deep neural network (DNN) as such a recognizer, for example. As such a DNN, for example, a DNN with a convolutional neural network (CNN) type structure such as Single Shot MultiBox Detector (SSD) or Faster R-CNN (Fast Regional Convolutional Neural Network) is used, or a DNN with an attention mechanism such as Vision Transformer is used. In addition, the vehicle detection unit 41 can also use a recognizer based on other machine learning methods such as support vector machine (SVM) or adaBoost as such a recognizer. This recognizer uses a plurality of teaching images representing vehicles to perform pre-learning according to a prescribed learning method such as error back propagation to detect vehicles from images.

[0045] In addition, the recognizer may also be learned to detect the license plate installed on the vehicle. In this case, the vehicle detection unit 41 can recognize the registration number of the detected vehicle by inputting the number recognition unit that has been pre-learned in a manner of recognizing the registration number into the area where the license plate is represented on the surveillance image. Then, the vehicle detection unit 41 determines the identification number that matches the recognized registration number from the identification numbers of each vehicle 2 stored in the storage device 32 or the memory 33. In addition, the vehicle detection unit 41 can use a CNN-type DNN or a DNN with an attention mechanism as such a recognizer. Then, the vehicle detection unit 41 determines that the vehicle 2 with the determined identification number is detected. In addition, the vehicle detection unit 41 uses the generation time of the surveillance image showing the detected vehicle 2 as the first time. In addition, the vehicle detection unit 41 determines a point that is included in the area where the detected vehicle 2 is represented on the image, or a point closest to the area, among a plurality of points associated with the position coordinates of the real space, and uses the position coordinates of the real space corresponding to the determined point as the first position of the detected vehicle 2. In addition, the position of each pixel on the monitoring image corresponds to the direction from the monitoring camera to the object represented by the pixel. Therefore, the vehicle detection unit 41 can obtain the first position of the detected vehicle 2 based on the parameters of the monitoring camera, such as the direction from the monitoring camera corresponding to the position of the area where the vehicle 2 is represented on the monitoring image, the installation position of the monitoring camera, and the shooting direction.

[0046] In addition, the vehicle detection unit 41 may also refer to the driving information received from each vehicle 2 and determine the vehicle that is closest to the first position at the first moment among the vehicles 2 as the vehicle 2 located at the first position at the first moment. In this case, the vehicle detection unit 41 may omit the detection of the registration number. In addition, the identifier may be pre-learned to recognize the model of the detected vehicle. In this case, the vehicle detection unit 41 may determine the vehicle that is the same model as the model detected at the first moment among the vehicles 2 as the vehicle 2 located at the first position at the first moment. In this case, the vehicle detection unit 41 may also omit the detection of the registration number.

[0047] In addition, when the fixed sensor 3 is a vehicle detection sensor or a beacon device, the vehicle detection unit 41 uses the detection time included in the received vehicle detection signal as the first time, and uses the position detectable by the vehicle detection sensor as the first position. Then, the vehicle detection unit 41 refers to the driving information received from each vehicle 2, and determines the vehicle 2 that is closest to the first position at the first time as the vehicle 2 located at the first position at the first time.

[0048] The vehicle detection unit 41 notifies the vehicle position estimation unit 42 of the first time and the first position of the detected vehicle 2 .

[0049] The vehicle position estimating unit 42 estimates the position (second position) of the vehicle 2 at the time (second time) when an image expressing a predetermined feature included in the feature data and received from the vehicle 2 was generated, based on the first position of the detected vehicle 2 at the first time.

[0050] The vehicle position estimation unit 42 calculates a vector indicating the difference between the position of the vehicle 2 at the first moment (hereinafter, referred to as the first pre-correction position for convenience of explanation) included in the driving information received from the detected vehicle 2 and directed to the first position at the first moment detected by the vehicle detection unit 41 as a position correction vector. Then, the vehicle position estimation unit 42 uses the position correction vector to correct the position of the vehicle 2 at the time of image generation (second moment) included in the feature data (hereinafter, referred to as the second pre-correction position for convenience of explanation) as the second position (that is, the position obtained by adding the second pre-correction position to the position correction vector).

[0051] In addition, when the driving information does not include the first pre-correction position, the vehicle position estimating unit 42 obtains the first pre-correction position by internally dividing the position of the vehicle 2 at the time before and after the first time contained in the driving information by the difference between the time before and after and the first time. In addition, when the driving information includes distance measurement information, the first pre-correction position can be obtained by correcting the position of the vehicle 2 at the reference time using the distance measurement information from the reference time before or after the first time to the first time.

[0052] When the server 4 receives a plurality of feature data from the detected vehicle 2, the vehicle position estimating unit 42 can perform the above-mentioned processing on each feature data, thereby estimating the second position of the vehicle 2 at the second time when the image including the feature data was generated for each feature data. In addition, when the feature data includes a plurality of images and the time when the images were generated, the vehicle position estimating unit 42 can also perform the above-mentioned processing on each image, thereby estimating the second position of the vehicle 2 at the second time when the image was generated for each image.

[0053] The vehicle position estimating unit 42 notifies the characteristic position estimating unit 44 of the estimated second position.

[0054] The feature detection unit 43 detects the specified features that become the detection object based on the image contained in the feature data received from the detected vehicle 2. To this end, the feature detection unit 43 inputs the image into a recognizer for feature detection, thereby identifying the object area represented by the specified features represented in the image and the type of the specified features. The feature detection unit 43 can use a DNN with a CNN type structure or a DNN with an attention mechanism as such a recognizer. In addition, the feature detection unit 43 can also use a recognizer based on other machine learning methods such as a support vector machine (SVM) or adaBoost as such a recognizer. This recognizer uses multiple teaching images that represent specified features to perform pre-learning according to a specified learning method such as the error back propagation method to detect specified features from the image.

[0055] The feature detection unit 43 notifies the feature position estimation unit 44 of the second time when the image in which the predetermined feature is detected is generated, information indicating the position and size of the object region showing the predetermined feature on the image, and the type of the predetermined feature.

[0056] The feature position estimating unit 44 estimates the position of the feature based on the second position of the vehicle 2 when the image expressing the detected feature is generated.

[0057] As described above, for the monitoring camera, the position of each pixel on the image corresponds to the position from the camera 11 to the object represented by the pixel. Therefore, when the detected feature is a feature set on the road surface such as a road sign or a three-dimensional structure placed on the road surface, the feature position estimation unit 44 can estimate the position of the feature in the camera coordinate system with the camera 11 as the reference. At this time, the feature position estimation unit 44 can estimate the position of the feature based on the position of the camera 11 corresponding to the reference point in the object area, the traveling direction of the vehicle 2 at the second time when the image is generated, and the camera 11 parameters such as the shooting direction of the camera 11 and the installation height in the vehicle 2. In addition, the feature position estimation unit 44 can estimate the position of the detected feature in the camera coordinate system by transforming the position of the camera 11 into a position in the coordinate system of the real space as the second position, for example, by performing an affine transformation on the position of the feature. In addition, the reference point in the object area can be set as the center of gravity of the object area. In addition, when the feature is a three-dimensional structure placed on the road surface, the lower end position estimated as the object area means the position where the feature touches the road surface. Therefore, in this case, the reference point can be set to an arbitrary position on the lower end of the object area.

[0058] In addition, when the same feature is detected from a plurality of images generated at different times, the feature position estimating unit 44 may estimate the position of the feature according to a so-called structure from motion method. In this case, the feature position estimating unit 44 estimates the position of the feature by triangulation based on the second position and the travel direction of the vehicle 2 when each image was generated, the position of the object region showing the feature in each image, and the parameters of the camera 11.

[0059] Furthermore, the feature position estimating unit 44 may associate the same feature respectively represented in a plurality of images generated at different times between the images according to a tracking method called KLT (Kanade-Lucas-Tomasi) tracking.

[0060] The feature position estimating unit 44 stores information indicating the type and estimated position of the detected predetermined feature in the storage device 32. Alternatively, the feature position estimating unit 44 may output the information indicating the type and estimated position of the detected feature to other devices via the communication interface 31.

[0061] Figure 6 1 is a diagram for explaining the outline of the position of the estimated feature involved in this embodiment. Figure 6 As shown, in this example, at the first moment t1 when the vehicle 2 is detected by the fixed sensor 3, the first position P1 of the vehicle 2 is detected based on the fixed sensor signal. On the other hand, at the first moment t1, the position of the vehicle 2 (the first pre-correction position) Pg1 is detected by the GPS receiver 12 in the vehicle 2. Therefore, the position correction vector C = |P1-Pg1| is calculated. By using the position correction vector C, the position of the vehicle 2 (the second pre-correction position) Pg2 located by the GPS receiver 12 at the second moment t2 when the vehicle-mounted image IMG is generated by the camera 11 mounted on the vehicle 2 is corrected, thereby estimating the second position P2 of the vehicle 2 at the moment t2. Then, based on the second position P2, the position Ps of the feature S detected from the vehicle-mounted image IMG in the real space is estimated.

[0062] Figure 7 This is an operation flowchart of the position estimation process in the server 4. The processor 34 of the server 4 may execute the position estimation process according to the operation flowchart shown below.

[0063] The vehicle detection unit 41 of the processor 34 detects a first position of the vehicle 2 traveling on a predetermined road section at a first time based on the fixed sensor signal received from the fixed sensor 3 (step S101 ).

[0064] Furthermore, the vehicle position estimating unit 42 of the processor 34 estimates the second position of the vehicle 2 at the second time when the in-vehicle image showing the predetermined feature of the detection target is generated based on the first position of the detected vehicle 2 at the first time (step S102 ).

[0065] Furthermore, the feature detection unit 43 of the processor 34 detects a predetermined feature based on the vehicle-mounted image generated at the second time point and included in the feature data received from the detected vehicle 2 (step S103 ).

[0066] Then, the feature position estimating unit 44 of the processor 34 estimates the position of the feature based on the second position of the vehicle 2 when the vehicle-mounted image showing the detected feature was generated (step S104 ). Then, the processor 34 ends the position estimation process.

[0067] The processor 34 can generate or update a map based on the type and position of the detected feature. For example, the processor 34 adds information indicating the position and type of the feature for each detected feature in the map to be generated. In addition, in the map to be updated, if there is no feature of the same type as the detected feature within a predetermined distance from the position of the detected feature, the processor 34 adds information indicating the position and type of the detected feature to the map to be updated. In addition, in the map to be updated, if the position of the detected feature contains information related to a feature of a type different from the type of the detected feature, the processor 34 replaces the type of the feature at the position with the type of the detected feature. In addition, if a feature represented at a predetermined location on the map to be updated is not detected from feature data collected from each vehicle 2 passing through the predetermined location within a certain period of time, the processor 34 deletes the information of the feature at the predetermined location from the map.

[0068] The processor 34 can distribute the generated or updated map via the communication network 5 and the wireless base station 6 to the vehicle that uses the map in autonomous driving control or driving assistance.

[0069] As described above, the position estimating device uses the position of the vehicle detected by the fixed sensor that can accurately detect the position of the vehicle to estimate the position of the vehicle when the vehicle-mounted image is generated. Therefore, the position estimating device can accurately estimate the position of the vehicle when the vehicle-mounted image is generated. As a result, the position estimating device can accurately estimate the position of the feature detected from the vehicle-mounted image.

[0070] According to the modified example, the vehicle position estimating unit 42 can estimate the position of the vehicle 2 at the time when the vehicle image was generated, only for the vehicle image in which the feature detecting unit 43 detected a predetermined feature among a series of vehicle images received from the vehicle 2. This reduces the amount of calculation.

[0071] According to another modification, the vehicle position estimating unit 42 may also use the distance measurement information included in the driving information to obtain the moving direction and movement amount of the vehicle 2 from the first time when the vehicle 2 is detected by the fixed sensor 3 to the second time when the vehicle-mounted image showing the prescribed characteristics is generated. Then, the vehicle position estimating unit 42 may estimate the second position of the vehicle at the second time by correcting the first position using the moving direction and movement amount. Thus, even if the GPS receiver 12 mounted on the vehicle 2 cannot locate the position of the vehicle 2 at the second time, the vehicle position estimating unit 42 can estimate the second position of the vehicle 2.

[0072] According to another modified example, when the fixed sensor 3 is a monitoring camera, the vehicle detection unit 41 can detect the position of the vehicle 2 from a plurality of monitoring images generated at different times. Then, the vehicle position estimation unit 42 uses the position of the vehicle 2 detected from the monitoring image generated at the time closest to the second time among the plurality of monitoring images as the first position, thereby estimating the second position of the vehicle 2 at the second time. In this way, the monitoring image, which is a fixed sensor signal generated at the time closest to the time when the vehicle-mounted image showing the prescribed features is generated, is used for estimating the position of the vehicle 2, and thus the vehicle position estimation unit 42 can further improve the estimation accuracy of the second position of the vehicle 2.

[0073] Furthermore, when a plurality of fixed sensors 3 are provided, the vehicle detection unit 41 may perform the above-mentioned processing for each fixed sensor 3 to detect the vehicle 2. In this case, when the same vehicle 2 is detected from the fixed sensor signals generated by two or more fixed sensors 3, the vehicle position estimation unit 42 may estimate the second position of the vehicle 2 by using the position of the vehicle 2 that is close to the second pre-correction position included in the driving information among the positions of the detected vehicles 2 as the first position. In this way, since the position of the vehicle 2 detected by the fixed sensor 3 that is close to the second pre-correction position of the vehicle 2 when the vehicle-mounted image showing the prescribed characteristics is generated is used for the estimation of the second position, the vehicle position estimation unit 42 can further improve the estimation accuracy of the second position.

[0074] According to another variant, the processor 23 of the data acquisition device 14 mounted on the vehicle 2 may also perform the processing of the feature detection unit 43 in the above embodiment. In this case, the processor 23 only needs to include information indicating the generation time of the vehicle-mounted image in which the specified feature is detected, the position of the vehicle 2 located by the GPS receiver 12 at the generation time, the travel direction of the vehicle 2, the parameters of the camera 11, and the position and range of the object area represented by the specified feature on the vehicle-mounted image in the feature data. In addition, the feature position estimation unit 44 can refer to the information included in the feature data to estimate the position of the specified feature detected from the vehicle-mounted image.

[0075] According to another modified example, when the fixed sensor 3 is a surveillance camera, the vehicle detection unit 41 can estimate the area that becomes the blind spot of the vehicle 2 based on the surveillance image. In addition, when the estimated position of the feature detected from the vehicle-mounted image is included in the area that becomes the blind spot, the processor 34 can determine that the feature is a feature that is detected by mistake, and does not use the information related to the feature for the generation or update of the map. For example, when not only the vehicle 2 but also other vehicles (hereinafter referred to as surrounding vehicles) are detected from the surveillance image, the vehicle detection unit 41 can determine the area that is blocked by the surrounding vehicles when observed from the vehicle 2 as the area that becomes the blind spot based on the positional relationship between the vehicle 2 and the surrounding vehicles.

[0076] In addition, when a beacon device is used as the fixed sensor 3, the data acquisition device 14 of the vehicle 2 can detect the position of the vehicle 2. In this case, the memory 22 of the data acquisition device 14 pre-stores the position where the beacon signal from the beacon device can be received. In addition, the processor 23 of the data acquisition device 14 can set the time when the beacon signal sent from the beacon device is detected by the vehicle-mounted device mounted on the vehicle 2 as the first time, and detect the position where the beacon signal can be received as the first position. In addition, the processor 23 can include the first time and the first position, or the position correction vector in the driving information sent to the server 4. In addition, in this case, the processor 23 of the data acquisition device 14 can estimate the position of the detected feature by executing the processing of each part of the processor 34 of the server 4 in the above-mentioned embodiment. In addition, the processor 34 can include the position of the detected feature in the feature data. In this case, the data acquisition device 14 becomes another example of a position estimation device.

[0077] The computer program that enables a computer to implement the functions of the various parts of the processor of the position estimation device involved in each of the above-mentioned embodiments or modifications may also be provided in the form of being stored in a computer-readable storage medium. In addition, the computer-readable storage medium may be, for example, a magnetic storage medium, an optical storage medium, or a semiconductor memory.

[0078] As described above, those skilled in the art can make various changes according to the embodiments within the scope of the present invention.

Claims

1. A position estimation device, comprising: a vehicle detection unit that detects a first position of a vehicle traveling on a predetermined road section at a first time based on a fixed sensor installed on or near the predetermined road section; a vehicle position estimating unit that estimates a second position of the vehicle at a second time when an onboard image showing a predetermined feature is generated by an onboard camera mounted on the vehicle, based on the first position of the vehicle at the first time; and The feature position estimating unit estimates the position of the predetermined feature in the real space based on the second position.

2. The position estimation device according to claim 1, wherein: The fixed sensor is a monitoring camera, and the monitoring camera generates a monitoring image showing the predetermined road section at a predetermined period. The vehicle detection unit detects the position of the vehicle from each of the plurality of monitoring images generated at different times. The vehicle position estimating unit estimates the second position by using the position of the vehicle detected from a monitoring image generated at a time closest to the second time among the plurality of monitoring images as the first position.

3. The position estimation device according to claim 1 or 2, wherein: The vehicle position estimating unit estimates the second position by correcting a second pre-correction position of the vehicle measured by the satellite positioning device at the second time according to a difference between a first pre-correction position of the vehicle measured by the satellite positioning device mounted on the vehicle at the first time and the first position.

4. The position estimation device according to claim 1 or 2, wherein: further comprising a feature detection unit for detecting the predetermined feature from the vehicle-mounted image generated at the second time, The feature position estimating unit estimates the position of the predetermined feature in real space based on the position of the predetermined feature on the in-vehicle image and the second position.

5. A method for estimating a position, comprising: detecting a first position of a vehicle traveling on a predetermined road section at a first time based on a fixed sensor provided on or near the predetermined road section; estimating, based on the first position of the vehicle at the first time, a second position of the vehicle at a second time when an onboard image showing a predetermined feature is generated by an onboard camera mounted on the vehicle; and The position of the predetermined feature in the real space is estimated based on the second position.

6. A computer program product for position estimation, configured to cause a computer to execute the following: detecting a first position of a vehicle traveling on a predetermined road section at a first time based on a fixed sensor provided on or near the predetermined road section; estimating, based on the first position of the vehicle at the first time, a second position of the vehicle at a second time when an onboard image showing a predetermined feature is generated by an onboard camera mounted on the vehicle; and The position of the predetermined feature in the real space is estimated based on the second position.