Self-position estimation device

Abstract

To suppress the degradation of self-position estimation accuracy in a self-position estimation device.SOLUTION: A self-position estimation device 30 is adapted to a movable object 200 having a camera 10 and a communication device 20. The camera takes an image of surroundings of the movable object. The self-position estimation device is configured to estimate a self-position of the movable object and includes a feature point acquisition unit 311 which acquires an image feature point, a scale calculation unit 312 which calculates an actual scale of the image, and a self-position estimation unit 313 which estimates a self-position of the movable object based on the image feature point and the actual scale. The scale calculation unit acquires object information and an object image feature point, and calculates the actual scale based on the acquired object information and a feature value of the acquired object image feature point.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a self-location estimation device. [Background technology]

[0002] As a self-location estimation device for a vehicle, one that uses an image captured by a monocular camera mounted on the vehicle is known. Since the image captured by the monocular camera has an indefinite real scale, there is a risk of the accuracy of self-location estimation being reduced. The self-location estimation device equipped in a vehicle described in Patent Document 1 uses the positioning information of the vehicle itself received by a GNSS (Global Navigation Satellite System) receiver mounted on the vehicle itself, the positioning information of other vehicles received by a GNSS receiver mounted on the other vehicles and acquired by vehicle-to-vehicle communication, and characteristic points of landmarks included in a pre-stored local map to suppress the reduction in the accuracy of self-location estimation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-100942 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the self-location estimation device equipped in the vehicle described in Patent Document 1, there is a risk that the accuracy of self-location estimation may decrease in environments where the GNSS receiver cannot receive sufficient positioning information, such as inside tunnels and urban areas. In addition, it is necessary to create and store a database of landmark feature points in advance, which puts a strain on the storage capacity of the storage device. This problem is not limited to vehicles, but is also the same for self-location estimation devices mounted on any moving object such as an aircraft or a robot. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] According to one embodiment of the present disclosure, there is provided a self-location estimation device (30) that is provided in a moving body (200) having a camera (10) that captures an image of the surroundings and a communication device (20) and that estimates a self-location of the moving body. This control device includes a feature point acquisition unit (311) that acquires image feature points that are feature points present in an image captured by the camera, a scale calculation unit (312) that calculates a real scale of the image, and a self-location estimation unit (313) that estimates the self-location by using the image feature points and the real scale, and the scale calculation unit executes the following operations: acquiring object information that is acquired by the communication device from an external device (400) located in the vicinity of the moving body and is information about an object present in the image, acquiring object image feature points that are feature points about the object corresponding to the object information among the image feature points, and calculating the real scale by using a feature amount of the object image feature points and the object information.

[0007] According to this embodiment of the self-location estimation device, the real scale is calculated using the feature amount of the object image feature points and the object information, and the self-location is estimated using the image feature points and the real scale. Therefore, even in an environment where the GNSS receiver cannot receive sufficient positioning information, the self-location can be estimated using the camera mounted on the moving object, and the deterioration of the accuracy of the self-location estimation can be suppressed. In addition, since the object information acquired from the external device is used, the storage capacity of the own storage device can be suppressed from being compressed, compared to a form in which a database of landmark feature points is created in advance and stored in the own storage device. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram showing a schematic configuration of a control device according to the first embodiment. [Diagram 2] 4 is a flowchart showing a procedure of processing in the control device of the first embodiment. [Diagram 3]5 is a flowchart showing a procedure of a real scale calculation process according to the first embodiment. [Figure 4] FIG. 11 is a block diagram showing a schematic configuration of a control device according to a second embodiment. [Diagram 5] 10 is a flowchart showing the procedure of a real scale calculation process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment: A-1.Device configuration: As shown in FIG. 1, the vehicle 200 includes a camera 10, a communication device 20, a control device 30, and an actuator 40. The vehicle 200 is a vehicle capable of performing automatic driving, and is configured to be switchable between automatic driving and manual driving. "Automatic driving" refers to driving in which drive control, brake control, and steering control are automatically performed on behalf of the occupant. "Manual driving" refers to driving in which the occupant performs operations for drive control (pressing the accelerator pedal), brake control (pressing the brake pedal), and steering control (rotating the steering wheel). The vehicle 200 is configured as any vehicle, such as an electric vehicle (BEV: Battery Electric Vehicle), a hybrid vehicle (HEV: Hybrid Electric Vehicle), a fuel cell vehicle (FCV: Fuel Cell Vehicle), a gasoline vehicle, or a diesel vehicle. In addition, the vehicle 200 may be equipped with various sensors that can be used to detect the position and attitude of the vehicle 200, such as a Global Navigation Satellite System (GNSS) receiver, an Inertial Measurement Unit (IMU), and a Light Detection and Ranging (LiDAR).

[0010] The camera 10 captures an image by capturing an image of the periphery of the vehicle 200. In this embodiment, the camera 10 is configured as a monocular camera that captures an image in front of the vehicle 200. Note that the camera 10 is not limited to capturing an image in front of the vehicle 200, and may capture an image in the side or rear.

[0011] The communication device 20 communicates with an in-vehicle device 400 mounted on another vehicle 500 located in the vicinity of the vehicle 200, and acquires object information from the in-vehicle device 400. "Object information" refers to information related to the vehicle 500, which is an object included in an image captured by the camera 10. In this embodiment, the vehicle 500 corresponds to "another moving body" in this disclosure, and the in-vehicle device 400 corresponds to "external device" in this disclosure.

[0012] In this embodiment, the in-vehicle device 400 includes a CPU 410, a communication device 420, and a memory 430. The in-vehicle device 400 transmits object information stored in advance in the memory 430 from the communication device 420 to the control device 30 under the control of the CPU 410. For example, when the distance between the vehicle 500 and the vehicle 200 becomes less than a predetermined distance, the in-vehicle device 400 transmits the object information to the communication device 20. In this embodiment, the in-vehicle device 400 prestores feature point information FI as object information in the memory 430. The "feature point information FI" means information indicating the positions of a plurality of feature points (hereinafter also referred to as "object feature points") corresponding to at least a part of the vehicle 500, the feature amount of each object feature point, and the type of feature amount algorithm for calculating the feature amount of each object feature point. For example, the in-vehicle device 400 pre-stores, as feature point information FI, the coordinates of a plurality of feature points indicating the tail lamp portion of the vehicle 500 relative to the center of gravity of the vehicle 500, the feature amounts of each of the plurality of feature points, and the type of feature algorithm used to calculate the feature amounts.

[0013] In this embodiment, the in-vehicle device 400 prestores a plurality of feature amounts calculated by a plurality of types of feature amount algorithms for each object feature point. The types of feature amount algorithms for object feature points stored in the in-vehicle device 400 will be described later. The in-vehicle device 400 determines, between the scale calculation unit 312 described later, which type of feature amount algorithm is used to calculate a feature amount for each object feature point and transmit it as feature point information FI. Furthermore, the in-vehicle device 400 transmits, as feature point information FI, the feature amount calculated by the determined type of feature amount algorithm for each object feature point among the plurality of feature amounts. In this embodiment, the in-vehicle device 400 receives an available feature amount algorithm transmitted by the scale calculation unit 312 described later, and transmits feature point information FI including a feature amount calculated by any one type of feature amount algorithm available to the scale calculation unit 312 among the above-mentioned plurality of feature amounts.

[0014] The actuator 40 includes a drive device for accelerating the vehicle 200, a steering device for changing the traveling direction of the vehicle 200, and a braking device for decelerating the vehicle 200. The actuator 40 may further include an actuator for swinging the wipers of the vehicle 200, an actuator for opening and closing the power windows of the vehicle 200, and the like.

[0015] The control device 30 estimates the self-position of the vehicle 200 by using the object information acquired by the communication device 20 and the image acquired by the camera 10. The control device 30 corresponds to the "self-position estimation device" in this disclosure. The control device 30 also creates a driving route by using the estimated self-position, and controls the actuator 40 to make the vehicle 200 travel along the driving route.

[0016] In this embodiment, the control device 30 is configured as an ECU (Electronic Control Unit) having a CPU 31, a ROM 32, and a RAM 33. The CPU 31 loads a program prestored in the ROM 32 into the RAM 33 and executes it, thereby functioning as a feature point acquisition unit 311, a scale calculation unit 312, a self-position estimation unit 313, a route planning unit 314, and a vehicle control unit 315.

[0017] The feature point acquisition unit 311 acquires feature points (hereinafter also referred to as "image feature points") present in an image captured by the camera 10, and calculates a feature amount for each image feature point. The image feature points include not only feature points corresponding to the vehicle 500, but also feature points corresponding to buildings, roads, and the like that are background parts of the image. The scale calculation unit 312 calculates the real scale of the image captured by the camera 10. The "real scale" means a measure of length that appears on the image with respect to length in the real world. The feature point acquisition unit 311 and the scale calculation unit 312 each use a predetermined type of feature amount algorithm to execute calculation of the feature amount of the image feature point or calculation of the real scale. In this embodiment, the feature point acquisition unit 311 and the scale calculation unit 312 can each use three types of feature amount algorithms, namely, SIFT (Scale Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), and SURF (Speed ​​Up Robust Features), as feature amount algorithms.

[0018] The self-position estimation unit 313 estimates the self-position of the vehicle 200 by using the feature amount of the image feature point and the actual scale. The route planning unit 314 uses the estimated self-position to create a planned route that is a planned travel route of the vehicle 200. The vehicle control unit 315 controls the actuator 40 so that the vehicle 200 travels along the created planned route. Specific processing by each functional unit will be described later.

[0019] A-2. Processing in the control device 30: The control device 30 repeatedly executes the process shown in Fig. 2 while the control device 30 is in an operating state. In step S10, the feature point acquisition unit 311 and the scale calculation unit 312 each acquire an image captured by the camera 10. Note that, although the image is acquired directly from the camera 10 in Fig. 1, the image may be temporarily stored in the ROM 32 and then acquired from the ROM 32.

[0020] 2, the feature point acquisition unit 311 acquires image feature points from the acquired image and calculates a feature amount for each image feature point. In this embodiment, the feature point acquisition unit 311 acquires image feature points and calculates feature amounts by using any one type of feature amount algorithm among available feature amount algorithms.

[0021] In step S30, scale calculation unit 312 executes the actual scale calculation process shown in Fig. 3. In step S310, scale calculation unit 312 transmits to in-vehicle device 400 the types of feature amount algorithms that it can use.

[0022] In step S320, the scale calculation unit 312 acquires feature point information FI acquired from the in-vehicle device 400 by the communication device 20. As described above, the in-vehicle device 400 transmits feature point information FI including feature amounts calculated by any one type of feature amount algorithm available to the scale calculation unit 312, and the scale calculation unit 312 acquires the feature point information FI. Note that, although the feature point information FI is acquired directly from the communication device 20 in FIG. 1, it may be acquired from the ROM 32 after being temporarily stored in the ROM 32 from the communication device 20.

[0023] In step S330 shown in FIG. 3, the scale calculation unit 312 acquires from the image, among the image feature points, feature points (hereinafter also referred to as "object image feature points") corresponding to the vehicle 500 which is an object corresponding to the feature point information FI, and calculates a feature amount for each object image feature point. In this embodiment, the scale calculation unit 312 acquires image feature points from the image. Furthermore, the scale calculation unit 312 executes matching between the acquired image feature points and the object feature points indicated by the feature point information FI, and acquires, among the image feature points, image feature points similar to the positional relationship of the object feature points and the feature amount of each object feature point as object image feature points. At this time, the scale calculation unit 312 calculates the image feature points and the feature amount of the object image feature points by using the same type of algorithm as the feature amount algorithm indicated by the feature point information FI.

[0024] In this embodiment, when multiple vehicles are present in the image, the scale calculation unit 312 preferentially selects a vehicle from which to acquire object feature points. For example, the scale calculation unit 312 selects a vehicle that is closer to the vehicle 200. The scale calculation unit 312 may also use multiple images that are consecutive in a time series that have already been acquired to select a vehicle in which the movement amount of corresponding object feature points between the images is large, in other words, a vehicle that has a high relative speed with respect to the vehicle 200 or a vehicle that is traveling in a direction crossing in front of the vehicle 200. On the other hand, the scale calculation unit 312 may also use multiple images that are consecutive in a time series that have already been acquired to select a vehicle in which the movement amount of corresponding object feature points between the images is small, in other words, a vehicle that is traveling in the same direction as the vehicle 200. Note that since the actual scale is indefinite at the time of this step, the above-mentioned "distance to the vehicle 200" and "relative speed with respect to the vehicle 200" may be calculated in units of pixels. By selecting vehicles from which to obtain object feature points as a priority as described above, the number of object feature points used to calculate the actual scale, which will be described later, can be reduced, and an increase in the computational load for calculating the actual scale can be suppressed.

[0025] In step S340, scale calculation unit 312 calculates the real scale using the feature amounts of the object image feature points and the feature amounts of the object feature points. More specifically, scale calculation unit 312 calculates the real scale by calculating the ratio between the distances between the object feature points indicated by the feature point information FI and the distances between the image object feature points corresponding to the object feature points. Scale calculation unit 312 determines which object feature point each object image feature point corresponds to by determining the positional relationship between each object image feature point and each object feature point and whether or not their feature amounts match.

[0026] In step S40 shown in FIG. 2, the self-position estimation unit 313 estimates the self-position of the vehicle 200 using the acquired image feature points and the calculated real scale. More specifically, the self-position estimation unit 313 calculates the three-dimensional position of each image feature point in pixel units from a plurality of time-series consecutive images using image analysis techniques such as SfM (Structure from Motion) and CNN (Convolutional Neural Network). The scale calculation unit 312 converts the calculated three-dimensional position in pixel units into a three-dimensional position in meters using the real scale. This allows the scale calculation unit 312 to estimate the relative position, vehicle speed, and traveling direction of the vehicle 200 with respect to the vehicles 500 present around the vehicle 200, signs and traffic lights installed on the road, buildings on the roadside, and the like.

[0027] In step S50, the route planning unit 314 creates a planned route using the estimated self-position of the vehicle 200. More specifically, the route planning unit 314 generates a short-term route plan, such as adjusting the distance between the vehicle 200 and the vehicle 500 and the traveling position of the vehicle 200 in the traveling lane, according to the surrounding conditions of the vehicle 200, such as traffic lights and the distance to surrounding vehicles. In addition, if the vehicle 200 is equipped with a GNSS receiver and the absolute position of the vehicle 200 can be detected at a certain point in the past by the positioning information, the route planning unit 314 can also create a long-term route plan from the current absolute position of the vehicle 200, which is calculated according to the relative movement amount from the point where the positioning information was interrupted, to a destination set in advance. Note that the route plan is not limited to the traveling route of the vehicle 200, and may include information specifying vehicle control, such as the vehicle speed and traveling direction of the vehicle 200 at each point on the traveling route.

[0028] In step S60, the vehicle control unit 315 controls the actuator 40 so as to make the vehicle 200 travel according to the created route plan. The control device 30 repeatedly executes the above-described processing.

[0029] According to the control device 30 of the first embodiment described above, the real scale is calculated using the feature amount of the object image feature point and the object information, and the self-position is estimated using the image feature point and the real scale. Therefore, even in an environment where the GNSS receiver cannot receive sufficient positioning information, the self-position can be estimated using the camera 10 mounted on the moving body without using other sensors, and the deterioration of the accuracy of the self-position estimation can be suppressed. In addition, since the object information acquired from the in-vehicle device 400 is used, it is possible to suppress the pressure on the storage capacity of the ROM 32 compared to a form in which a database of landmark feature points is created in advance and stored in the ROM 32.

[0030] In addition, feature point information FI indicating the positions of multiple object feature points, the feature amounts of each object feature point, and the type of feature algorithm used to calculate the feature amounts of each object feature point is obtained from the in-vehicle device 400, so the actual scale can be easily calculated using the feature amounts of the object image feature points and the feature amounts of the object feature points indicated by the feature point information FI.

[0031] B. Second embodiment: The control device 30A of the second embodiment differs from the control device 30 of the first embodiment in that it further includes a rendering processing unit 316 as shown in Fig. 4, and in that in the real scale calculation process, steps S312 and S322 shown in Fig. 5 are executed instead of steps S310 and S320 shown in Fig. 3. The device configuration of the control device 30A of the second embodiment and other steps in the process by the control device 30 are the same as those of the control device 30 of the first embodiment, so the same configurations and steps are denoted by the same reference numerals and detailed descriptions thereof will be omitted.

[0032] 4, in this embodiment, the in-vehicle device 400 stores, as object information, a three-dimensional model TDM indicating a three-dimensional shape of the vehicle 500 in advance in the memory 430. The three-dimensional model TDM includes information indicating the size of the vehicle 500 in the real world.

[0033] The rendering processing unit 316 acquires the three-dimensional model TDM from the in-vehicle device 400 via the communication device 20, and renders the three-dimensional model TDM to acquire a two-dimensional image. In this embodiment, the rendering processing unit 316 renders the three-dimensional model TDM to create four two-dimensional images of the vehicle 500 viewed from four directions, namely, front, rear, left, and right. The rendering processing unit 316 acquires feature points (hereinafter, "model feature points") present in the converted two-dimensional image, and calculates a feature amount for each model feature point. The rendering processing unit 316, like the feature point acquisition unit 311 and the scale calculation unit 312 described above, calculates the feature amount of the model feature point by using a predetermined type of feature amount algorithm. In this embodiment, the rendering processing unit 316 can use three types of feature amount algorithms, SIFT, ORB, and SURF, as the feature amount algorithm.

[0034] As shown in FIG. 5, in step S312, the rendering processing unit 316 acquires the three-dimensional model TDM acquired from the in-vehicle device 400 by the communication device 20, and renders the three-dimensional model TDM to acquire a two-dimensional image.

[0035] In step S322, the rendering processor 316 acquires model feature points from the created two-dimensional image and calculates feature amounts for each model feature point. In this embodiment, the rendering processor 316 calculates the feature amounts of the model feature points by using any one of the three available feature amount algorithms described above.

[0036] In step S330, the scale calculation unit 312 acquires object image feature points and calculates feature amounts for each object feature point. In this embodiment, the scale calculation unit 312 acquires image feature points from an image. Furthermore, the scale calculation unit 312 executes matching between the acquired multiple image feature points and multiple model feature points, and acquires, as object image feature points, image feature points that are similar to the positional relationships of the multiple model feature points or feature amounts of each model feature point among the multiple image feature points. At this time, the scale calculation unit 312 calculates the feature amounts of the image feature points and the object image feature points using the same type of feature algorithm as the feature algorithm used to calculate the feature amounts of the model feature points.

[0037] In step S340, the scale calculation unit 312 calculates the real scale by using the feature amounts of the model feature points and the feature amounts of the image object image feature points. More specifically, the scale calculation unit 312 calculates the real scale by calculating the ratio between the distance between a plurality of model feature points and the distance between a plurality of image object feature points that respectively correspond to the plurality of model feature points.

[0038] According to the control device 30A of the second embodiment described above, the three-dimensional model TDM of the vehicle 500 is acquired as object information, and the real scale is calculated using the feature amounts of the model feature points and the feature amounts of the object image feature points acquired by rendering the three-dimensional model TDM. Therefore, the feature amounts of the model feature points and the feature amounts of the object image feature points can be calculated using any feature amount algorithm, and compared to a form in which feature amounts calculated using a feature amount algorithm of a predetermined type are referenced, it is possible to suppress the inability to calculate the real scale due to a mismatch in the feature amount algorithm. Also, compared to a form in which feature amounts of feature points at predetermined locations on the vehicle 500 are acquired, the real scale can be calculated using the feature amounts of feature points at any location on the vehicle 500.

[0039] C. Third embodiment: The control device 30 of the third embodiment differs from the control device 30 of the first embodiment in that it further acquires the position of the vehicle 500 from the on-board device 400 as object information, and in that it performs bundle adjustment using the position of the vehicle 500 in step S40 of the real scale calculation process. The device configuration of the control device 30 of the third embodiment and other steps in the processing by the control device 30 are the same as those of the control device 30 of the first embodiment, so the same configurations and steps are denoted by the same reference numerals and detailed descriptions thereof will be omitted.

[0040] In this embodiment, the vehicle 500 includes various sensors (not shown) that can be used to detect the position and attitude of the vehicle 500, such as a GNSS receiver, an IMU, and a LiDAR, and is configured to be able to detect the position of the vehicle 500. The in-vehicle device 400 transmits the position of the vehicle 500 to the control device 30 as object information in addition to the above-mentioned feature point information FI.

[0041] In step S40 shown in FIG. 2, the self-position estimation unit 313 performs bundle adjustment using the position of the vehicle 500 acquired as object information, and corrects the reprojection error onto the image feature point. More specifically, in the bundle adjustment, the self-position estimation unit 313 estimates the three-dimensional coordinates of the subject corresponding to each image feature point based on the correspondence between each image feature point between a plurality of images successive in time series. Then, the self-position estimation unit 313 reprojects the three-dimensional coordinates of the subject corresponding to each image feature point onto the image plane based on the self-position of the vehicle 200 when each image was acquired, in other words, the position and attitude of the camera 10. The self-position estimation unit 313 corrects the self-position of the vehicle 200 when each image was acquired so that the reprojection error calculated as the distance between the reprojection point and the image feature point is minimized.

[0042] Usually, in the bundle adjustment, the self-location estimation unit 313 calculates the reprojection error by referring to the image feature points corresponding to the background part under the assumption that the background part of the image showing the fixed object such as a road or a building does not move in the real world. However, the vehicle 500 moves between a plurality of images that are successive in time series, so the above assumption does not hold. Therefore, when the position of the vehicle 500 is not used, the coordinates in the real world of the part corresponding to the image feature point of the vehicle 500 are not known, so the image feature points corresponding to the vehicle 500 cannot be referred to, and the self-location estimation unit 313 calculates the reprojection error by referring only to the image feature points corresponding to the background part. In contrast, when the position of the vehicle 500 is used, the coordinates in the real world of the part corresponding to the image feature point of the vehicle 500 can be specified, so the self-location estimation unit 313 can calculate the reprojection error by referring to the image feature points corresponding to the vehicle 500 in addition to the image feature points corresponding to the background part. In this way, when the position of the vehicle 500 is used as the object information, the reprojection error can be calculated by referring to more image feature points than when the position of the vehicle 500 is not used, and the decrease in the accuracy of the self-location estimation can be further suppressed.

[0043] The control device 30 of the third embodiment described above provides the same effects as those of the first embodiment. In addition, since the object information including the position of the vehicle 500 is acquired and a process of correcting a reprojection error onto an image feature point by bundle adjustment using the position of the vehicle 500 is further executed, a decrease in the accuracy of self-location estimation can be further suppressed.

[0044] D. Other embodiments: (D1) In the above embodiment, the control device 30 is mounted on the vehicle 200, but the present disclosure is not limited thereto. The control device 30 may be mounted on any moving object, such as an aircraft, a robot, or a ship. Similarly, the in-vehicle device 400 may be mounted on any moving object, such as an aircraft, a robot, or a ship.

[0045] (D2) In the above embodiment, the camera 10 is configured as a monocular camera, but the present disclosure is not limited thereto. The camera 10 may be configured as a stereo camera. In such a configuration, the self-position estimation unit 313 may calculate the three-dimensional position of each image feature point using stereo matching. This configuration also provides the same effects as the above embodiment. In addition, since the real scale that can be calculated by stereo matching can be compared with the real scale calculated by the above-mentioned processing by the scale calculation unit 312, the real scale can be calculated with higher accuracy, and the decrease in the self-position estimation accuracy can be further suppressed.

[0046] (D3) In the above embodiment, the control device 30 is configured as an ECU having a CPU 31, but the present disclosure is not limited to this. The control device 30 may be configured as a SoC (System on Chip) having a GPU (Graphics Processing Unit), an NPU (Neural network Processing Unit), a DSP (Digital Signal Processor), etc. instead of the CPU 31. Even with this form, the same effects as the above embodiment can be achieved.

[0047] (D4) In the above embodiment, the feature point acquisition unit 311, the scale calculation unit 312, the self-position estimation unit 313, the route planning unit 314, and the vehicle control unit 315 are realized in the same ECU, but the present disclosure is not limited to this. For example, an ECU that realizes the feature point acquisition unit 311, the scale calculation unit 312, and the self-position estimation unit 313, which are functional units that realize self-position estimation, and an ECU that realizes the route planning unit 314 and the vehicle control unit 315, which are functional units that realize vehicle control using the estimated self-position, may be provided independently of each other. In this embodiment, the ECU that realizes the feature point acquisition unit 311, the scale calculation unit 312, and the self-position estimation unit 313 corresponds to the "self-position estimation device" in the present disclosure.

[0048] (D5) In the above embodiment, the feature point acquisition unit 311 and the scale calculation unit 312 can use three types of feature amount algorithms, SIFT, ORB, and SURF, respectively, as feature amount algorithms, but the present disclosure is not limited to this. The feature point acquisition unit 311 and the scale calculation unit 312 may use different types of feature amount algorithms. In addition, the types of feature amount algorithms are not limited to the above three types, and any known feature amount algorithm such as HoG (Histograms of Oriented Gradients) or CNN (Convolutional Neural Network) may be used. In addition, the number of feature amount algorithms that the feature point acquisition unit 311 and the scale calculation unit 312 can use may be four or more, or may be two. Even with this embodiment, the same effects as those of the above embodiment can be achieved.

[0049] Furthermore, the scale calculation unit 312 may be configured to be able to use only one type of feature amount algorithm. In such a configuration, the scale calculation unit 312 may calculate the actual scale when the feature amount algorithm indicated by the received feature point information FI is of a usable type. Even with such a configuration, the actual scale can be calculated when the feature amount algorithm indicated by the received feature point information FI is of a usable type, and a decrease in the accuracy of self-location estimation can be suppressed.

[0050] (D6) In the above embodiment, scale calculation unit 312 acquires object information from on-board device 400 mounted on vehicle 500, but the present disclosure is not limited to this. Scale calculation unit 312 may acquire, as object information, either one of feature point information FI and three-dimensional model TDM of a feature, such as a sign or a traffic light, that is installed in advance around vehicle 200, and the position of the feature.

[0051] (D7) In the above embodiment, the control device 30 repeatedly executes the process shown in FIG. 2 while the control device 30 is in an operating state, but the present disclosure is not limited thereto. The control device 30 may start or stop the process shown in FIG. 2 according to the driving environment of the vehicle 200. For example, when the vehicle 200 is configured to receive positioning information by a GNSS receiver and detect the position of the vehicle itself, the control device 30 may start the process shown in FIG. 2 in an environment where the positioning information cannot be stably received, such as an urban area or inside a tunnel, and end the process when the positioning information can be stably received. According to this embodiment, the process shown in FIG. 2 is not executed in a driving environment where the vehicle itself can be detected by the positioning information, so that an increase in power consumption in the control device 30 can be suppressed.

[0052] (D8) In the above embodiment, as shown in FIG. 2, the control device 30 executes step S30 after step S20, but the present disclosure is not limited to this. The control device 30 may execute step S30 in parallel with step S20. According to this embodiment, multiple processes can be executed in parallel, so that the processes in the control device 30 can be executed efficiently.

[0053] (D9) In the above embodiment, when multiple vehicles are present in the image, the scale calculation unit 312 preferentially selects a vehicle for which object feature points are to be acquired, but the present disclosure is not limited to this. The scale calculation unit 312 may acquire object feature points for multiple vehicles present in the image and calculate the real scale. According to this embodiment, compared to an embodiment in which object feature points are acquired for a single vehicle, the real scale can be calculated using more object feature points, so that the real scale can be calculated with higher accuracy and the decrease in the accuracy of self-location estimation can be further suppressed.

[0054] (D10) In the above embodiment, the scale calculation unit 312 transmits the available feature amount algorithms to the in-vehicle device 400, and the in-vehicle device 400 transmits the feature amount calculated by any one type of feature amount algorithm available to the scale calculation unit 312 among the multiple feature amounts calculated by the multiple types of feature amount algorithms for each object feature point, but the present disclosure is not limited to this. The in-vehicle device 400 may transmit the feature amounts calculated by two or more types of feature amount algorithms available to the scale calculation unit 312 among the multiple feature amounts calculated by the multiple types of feature amount algorithms for each object feature point. According to this embodiment, it is possible to suppress a decrease in the degree of freedom in selecting the type of feature amount algorithm used by the scale calculation unit 312.

[0055] Furthermore, the in-vehicle device 400 may transmit a plurality of feature amounts calculated by two or more types of feature amount algorithms for each object feature point, regardless of the type of feature amount algorithm available to the scale calculation unit 312. When there are a plurality of feature amount algorithms available among the feature amount algorithms for calculating the plurality of feature amounts received as the feature point information FI, the scale calculation unit 312 may determine the type of feature amount algorithm to be used according to a predetermined priority order, and calculate the real scale by using the feature amount of the object image feature point calculated by using the determined type of feature amount algorithm and the feature amount of the object feature point. The priority order is, for example, determined to preferentially use SIFT as the feature amount algorithm when the accuracy of self-location estimation is important, and determined to preferentially use ORB as the feature amount algorithm when the reduction of the calculation load is important. Note that the priority order may be determined in any order regardless of the above. According to this embodiment, when a plurality of feature amounts calculated by two or more types of feature amount algorithms for each object feature point are transmitted, it is possible to easily determine which type of feature amount algorithm to use.

[0056] (D11) In the above embodiment, the feature point acquisition unit 311 and the scale calculation unit 312 each acquire an image, acquire image feature points and object image feature points using a feature amount algorithm, and calculate feature amounts, but the present disclosure is not limited to this. Only the feature point acquisition unit 311 may acquire an image, acquire image feature points and object image feature points, and calculate the respective feature amounts. The scale calculation unit 312 may calculate the real scale using the object image feature points acquired by the feature point acquisition unit 311. In other words, the scale calculation unit 312 may acquire the object feature points from the feature point acquisition unit 311 without using a feature amount algorithm. In such a form, the feature point acquisition unit 311 communicates with the in-vehicle device 400 to determine the type of feature amount algorithm that can be used, similar to the scale calculation unit 312 in the above embodiment. According to this embodiment, only the feature point acquisition unit 311 acquires image features and object image features using a feature algorithm, and the scale calculation unit 312 acquires object features from the feature point acquisition unit 311 without using a feature algorithm, so that an increase in the computational load on the control device 30 can be suppressed.

[0057] (D12) In the above embodiment, the rendering processing unit 316 renders the three-dimensional model TDM to create four two-dimensional images of the vehicle 500 viewed from four directions, i.e., front, rear, left, right, and right, but the present disclosure is not limited to this. The rendering processing unit 316 may create eight two-dimensional images of the vehicle 500 viewed from eight directions, i.e., from front, rear, left, right, and diagonal directions. According to this embodiment, it is possible to prevent the difference between the posture of the vehicle 500 in the two-dimensional image created by rendering and the posture of the vehicle 500 in the image acquired by the camera 10 from becoming large, and it is possible to prevent the rotation process and the like for matching the model feature points and the object image feature points from becoming complicated.

[0058] Furthermore, the rendering processing unit 316 may create only one two-dimensional image of the vehicle 500 viewed from one predetermined direction. The posture of the vehicle 500 in the two-dimensional image is preferably a posture that is likely to appear in an image acquired by the camera 10. For example, it can be said that the posture in a two-dimensional image of the three-dimensional model TDM viewed from the rear is more likely to appear in an image acquired by the camera 10 than the posture in a two-dimensional image viewed from an oblique direction. According to this embodiment, the number of two-dimensional images created in the rendering process can be reduced, and therefore an increase in the computational load of the rendering process can be suppressed.

[0059] (D13) In the above embodiment, the control device 30 acquires the feature point information FI and the position of the vehicle 500 as object information to perform self-location estimation, but the present disclosure is not limited thereto. The control device 30 may acquire only the position of the vehicle 500 as object information to perform self-location estimation. For example, the scale calculation unit 312 may calculate the real scale by calculating the ratio between the amount of movement of the vehicle 500 in pixel units in multiple images that are consecutive in time series and the amount of movement of the actual vehicle 500, and the self-location estimation unit 313 may perform self-location estimation using the real scale. The "amount of movement of the vehicle 500 in pixel units" is obtained, for example, by calculating the displacement of the positional relationship between the image feature point corresponding to the vehicle 500 and the image feature point corresponding to the background part in each image. Note that in this embodiment, the correspondence between the vehicle present in the image and the vehicle that has transmitted the position information can be identified, for example, by comparing the absolute position of the vehicle 500 calculated by image analysis with the position of the vehicle 500 acquired as object information. The calculation of the absolute position of the vehicle 500 by image analysis can be realized by using the relative position between the vehicles 200 and 500 calculated by image analysis and the absolute position of the vehicle 200. Furthermore, as described above, when the absolute position of the vehicle 200 was detected at a certain point in the past by the positioning information, the absolute position of the vehicle 200 can be calculated according to the relative movement amount calculated by image analysis from the point where the positioning information was interrupted. According to this embodiment, since the self-position estimation can be performed using only the position of the vehicle 500 as the object information, the amount of information acquired from the in-vehicle device 400 as the object information can be reduced.

[0060] The control device 30 and the method described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control device 30 and the method described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control device 30 and the method described in the present disclosure may be implemented by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. Also, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer.

[0061] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in each embodiment corresponding to the technical features in the form described in the Summary of the Invention column can be appropriately replaced or combined in order to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. (Form 1) A self-position estimation device (30) is provided on a moving body (200) having a camera (10) for capturing an image of the surroundings and a communication device (20), and estimates the self-position of the moving body, a feature point acquisition unit (311) for acquiring image feature points that are feature points present in an image captured by the camera; A scale calculation unit (312) for calculating an actual scale of the image; a self-position estimation unit (313) that estimates the self-position by using the image feature points and the real scale; Equipped with The scale calculation unit Obtaining object information obtained by the communication device from an external device (400) located in the vicinity of the moving object, the object information being information about an object present in the image; acquiring an object image feature point, which is a feature point for the object corresponding to the object information, from among the image feature points; Calculating the real scale by using the feature amounts of the object image feature points and the object information. Self-location estimation device. (Form 2) A self-location estimation device according to aspect 1, The external device is provided on another moving body (500) located in the vicinity of the moving body. Self-location estimation device. (Form 3) The self-location estimation device according to aspect 1 or 2, The object information includes a position of the object; The self-location estimation unit further performs a process of correcting a reprojection error onto the image feature points by bundle adjustment using the position of the object. Self-location estimation device. (Form 4) A self-location estimation device according to aspect 1, The external device is provided on a feature installed around the moving body. Self-location estimation device. (Form 5) A self-location estimation device according to aspect 1 or aspect 4, The object information includes a position of the object; The self-location estimation unit further performs a process of correcting a reprojection error onto the image feature points by bundle adjustment using the position of the object. Self-location estimation device. (Form 6) A self-location estimation device according to any one of aspects 1 to 5, the object information includes feature point information (FI) indicating positions of a plurality of object feature points which are a plurality of feature points corresponding to at least a portion of the object, feature amounts of each of the object feature points, and a type of feature amount algorithm for calculating the feature amounts of each of the object feature points; The scale calculation unit acquiring, from among the image feature points, feature points corresponding to the object feature points as the object image feature points; calculating the real scale by using the feature amounts of the object image feature points and the feature amounts of the object feature points; Self-location estimation device. (Form 7) A self-location estimation device according to aspect 6, the external device stores in advance a plurality of feature amounts calculated for each of the object feature points by a plurality of types of the feature amount algorithms, The external device is determining, for each of the object feature points, which type of feature amount algorithm among the plurality of types of feature amount algorithms is used to calculate the feature amount to be transmitted as the feature point information, between at least one of the feature point acquisition unit and the scale calculation unit; transmitting, to the communication device, as the feature point information, a feature calculated by the feature algorithm of the determined type among the plurality of feature amounts for each of the object feature points; Execute Self-location estimation device. (Form 8) A self-location estimation device according to aspect 6, the external device pre-stores a plurality of feature amounts calculated for each of the object feature points by a plurality of types of the feature amount algorithms, and transmits, as the feature point information, a plurality of feature amounts calculated for each of the object feature points by two or more types of the feature amount algorithms out of the plurality of feature amounts to the communication device. Self-location estimation device. (Form 9) A self-location estimation device according to aspect 6, the scale calculation unit is configured to be able to use a plurality of types of feature amount algorithms, and calculates the actual scale by using a feature amount of the object image feature point calculated by using a feature amount algorithm of the same type as the feature amount algorithm of the feature amount of the object feature point received as the feature point information, and the feature amount of the object feature point. Self-location estimation device. (Form 10) A self-location estimation device according to aspect 6, the scale calculation unit is configured to be able to use a plurality of types of the feature amount algorithms; the external device pre-stores a plurality of feature amounts calculated for each of the object feature points by a plurality of types of the feature amount algorithms, and transmits to the communication device, as the feature point information, a plurality of feature amounts calculated for each of the object feature points by two or more types of the feature amount algorithms out of the plurality of feature amounts; The scale calculation unit determining a type of the feature amount algorithm to be used in accordance with a predetermined priority order when there are a plurality of available feature amount algorithms among the feature amount algorithms for calculating each of the plurality of feature amounts received as the feature point information; calculating the real scale by using the feature amounts of the object image feature points calculated by using the feature amount algorithm of the determined type and the feature amounts of the object feature points; Self-location estimation device. (Form 11) A self-location estimation device according to any one of aspects 1 to 5, The object information includes a three-dimensional model (TDM) of the object; The self-location estimation device further includes a rendering processing unit (316) that renders the three-dimensional model and acquires model feature points that are feature points of the three-dimensional model, The scale calculation unit acquiring, from among the image feature points, feature points corresponding to the model feature points as the object image feature points; calculating the actual scale by using the feature amounts of the object image feature points and the feature amounts of the model feature points; Self-location estimation device. [Explanation of symbols]

[0062] 10: camera, 20: communication device, 30, 30A: control device, 200: vehicle, 311: feature point acquisition unit, 312: scale calculation unit, 313: self-position estimation unit, 400: in-vehicle device, 500: vehicle

Claims

1. A self-position estimation device (30, 30A) is provided on a moving body (200) having a camera (10) for capturing an image of the surroundings and a communication device (20), and estimates the self-position of the moving body, A feature point acquisition unit (311) that acquires image feature points that are feature points present in an image captured by the camera; A scale calculation unit (312) for calculating an actual scale of the image; a self-position estimation unit (313) that estimates the self-position by using the image feature points and the real scale; Equipped with The scale calculation unit Obtaining object information obtained by the communication device from an external device (400) located in the vicinity of the moving object, the object information being information about an object present in the image; acquiring an object image feature point, which is a feature point for the object corresponding to the object information, from among the image feature points; Calculating the real scale by using the feature amounts of the object image feature points and the object information. Self-location estimation device.

2. The self-location estimation device according to claim 1 , The external device is provided in another moving body (500) located in the vicinity of the moving body. Self-location estimation device.

3. The self-location estimation device according to claim 2, The object information includes a position of the object; The self-location estimation unit further performs a process of correcting a reprojection error onto the image feature points by bundle adjustment using the position of the object. Self-location estimation device.

4. The self-location estimation device according to claim 1 , The external device is provided on a feature installed around the moving body. Self-location estimation device.

5. The self-location estimation device according to claim 4, The object information includes a position of the object; The self-location estimation unit further performs a process of correcting a reprojection error onto the image feature points by bundle adjustment using the position of the object. Self-location estimation device.

6. The self-location estimation device according to any one of claims 1 to 5, the object information includes feature point information (FI) indicating positions of a plurality of object feature points which are a plurality of feature points corresponding to at least a portion of the object, feature amounts of each of the object feature points, and a type of feature amount algorithm for calculating the feature amounts of each of the object feature points; The scale calculation unit Among the image feature points, a feature point corresponding to the object feature point is acquired as the object image feature point; calculating the real scale by using the feature amounts of the object image feature points and the feature amounts of the object feature points; Self-location estimation device.

7. The self-location estimation device according to claim 6, the external device stores in advance a plurality of feature amounts calculated for each of the object feature points by a plurality of types of the feature amount algorithms, The external device is determining, for each of the object feature points, which type of feature amount algorithm among the plurality of types of feature amount algorithms is used to calculate the feature amount to be transmitted as the feature point information, between at least one of the feature point acquisition unit and the scale calculation unit; transmitting, to the communication device, as the feature point information, a feature calculated by the feature algorithm of the determined type among the plurality of feature amounts for each of the object feature points; Execute Self-location estimation device.

8. The self-location estimation device according to claim 6, the external device pre-stores a plurality of feature amounts calculated for each of the object feature points by a plurality of types of the feature amount algorithms, and transmits, as the feature point information, a plurality of feature amounts calculated for each of the object feature points by two or more types of the feature amount algorithms out of the plurality of feature amounts to the communication device. Self-location estimation device.

9. The self-location estimation device according to claim 6, the scale calculation unit is configured to be able to use a plurality of types of feature amount algorithms, and calculates the actual scale by using a feature amount of the object image feature point calculated by using a feature amount algorithm of the same type as the feature amount algorithm of the feature amount of the object feature point received as the feature point information, and the feature amount of the object feature point. Self-location estimation device.

10. The self-location estimation device according to claim 6, the scale calculation unit is configured to be able to use a plurality of types of the feature amount algorithms; the external device pre-stores a plurality of feature amounts calculated for each of the object feature points by a plurality of types of the feature amount algorithms, and transmits to the communication device, as the feature point information, a plurality of feature amounts calculated for each of the object feature points by two or more types of the feature amount algorithms out of the plurality of feature amounts; The scale calculation unit determining a type of the feature amount algorithm to be used in accordance with a predetermined priority order when there are a plurality of available feature amount algorithms among the feature amount algorithms for calculating each of the plurality of feature amounts received as the feature point information; calculating the real scale by using the feature amounts of the object image feature points calculated by using the feature amount algorithm of the determined type and the feature amounts of the object feature points; Self-location estimation device.

11. The self-location estimation device according to any one of claims 1 to 5, the object information includes a three-dimensional model (TDM) of the object; The self-location estimation device further includes a rendering processing unit (316) that renders the three-dimensional model and acquires model feature points that are feature points of the three-dimensional model, The scale calculation unit Among the image feature points, feature points corresponding to the model feature points are acquired as the object image feature points; calculating the actual scale by using the feature amounts of the object image feature points and the feature amounts of the model feature points; Self-location estimation device.

Images