Self position estimation device

Abstract

A self-position estimation device (30), which is provided to a moving body (200) having a communication device (20) and a camera (10) that captures an image of the surroundings, and estimates the self-position of the moving body, is provided with: a feature point acquisition unit (311) that acquires image feature points; a scale calculation unit (312) that calculates the actual scale of the image; and a self-position estimation unit (313) that estimates a self-position using the image feature points and the actual scale, the scale calculation unit performing: acquiring object information; obtaining feature points of the object image; an actual scale is calculated using feature quantities of feature points of an object image and object information.

Description

Technical Field

[0001] The present disclosure relates to a self-position estimation device. Background Art

[0002] As a vehicle's own position estimation device, there is a known device that uses an image taken by a single-lens reflex camera mounted on the vehicle. Since the actual scale of the image taken by the single-lens reflex camera is uncertain, the accuracy of the self-position estimation may decrease. In the self-position estimation device of the vehicle described in Patent Document 1, the positioning information of the own vehicle received by the GNSS (Global Navigation Satellite System) receiver mounted on the own vehicle, the positioning information of other vehicles received by the GNSS receiver mounted on other vehicles and obtained through vehicle-to-vehicle communication, and the feature points of the landmarks included in the pre-stored local map are used to suppress the decrease in the accuracy of the self-position estimation.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-100942 Summary of the invention

[0006] However, in the vehicle's own position estimation device described in Patent Document 1, the accuracy of the self-position estimation may decrease in environments such as tunnels and towns where the GNSS receiver cannot fully receive positioning information. In addition, it is necessary to prepare and store a database of landmark feature points in advance, which puts pressure on the storage capacity of the storage device. Such problems are not limited to vehicles, but also exist in the self-position estimation device of any mobile body such as an aircraft or a robot.

[0007] The present disclosure can be implemented as the following technical solutions.

[0008] According to a technical solution of the present disclosure, a self-position estimation device is provided, which is arranged on a mobile body and estimates the self-position of the mobile body, wherein the mobile body has a camera and a communication device for photographing the surroundings. The control device comprises: a feature point acquisition unit, which acquires feature points existing in the image photographed by the camera, namely, image feature points; a scale calculation unit, which calculates the actual scale of the image; and a self-position estimation unit, which estimates the self-position using the image feature points and the actual scale; the scale calculation unit executes: acquiring object information, which is information related to the object existing in the image, acquired by the communication device from an external device located in the vicinity of the mobile body; acquiring object image feature points, which are feature points of the object corresponding to the object information among the image feature points; and calculating the actual scale using the feature amount of the object image feature points and the object information.

[0009] The self-position estimation device according to the technical solution uses the feature quantity of the feature points of the object image and the object information to calculate the actual scale, and uses the image feature points and the actual scale to estimate the self-position. Therefore, in an environment where the GNSS receiver cannot fully receive the positioning information, the self-position can be estimated using the camera mounted on the mobile body, which can suppress the decrease in the accuracy of the self-position estimation. In addition, since the object information obtained from the external device is used, it is possible to suppress the storage capacity of the self-storage device from being compressed compared to the method of pre-making a database of the feature points of the landmark and storing it in the self-storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing a schematic configuration of a control device according to the first embodiment.

[0011] Figure 2 This is a flowchart showing the processing procedure of the control device according to the first embodiment.

[0012] Figure 3 This is a flowchart showing the procedure of the actual scale calculation process according to the first embodiment.

[0013] Figure 4 This is a block diagram showing a schematic configuration of a control device according to the second embodiment.

[0014] Figure 5 This is a flowchart showing the procedure of the actual scale calculation process according to the second embodiment. DETAILED DESCRIPTION

[0015] A. First Implementation Method:

[0016] A-1. Device structure:

[0017] like Figure 1 As shown, the vehicle 200 includes a camera 10, a communication device 20, a control device 30, and an actuator 40. The vehicle 200 is a vehicle capable of performing automatic driving, and is configured to be switchable between automatic driving and manual driving. "Automatic driving" refers to driving in which drive control, brake control, and steering control are automatically performed instead of the driver. "Manual driving" refers to driving in which the driver performs operations for drive control (depressing the accelerator pedal), operations for brake control (depressing the brake pedal), and operations for steering control (rotation of the steering wheel). The vehicle 200 is configured as any vehicle such as an electric vehicle (BEV: Battery Electric Vehicle), a hybrid electric vehicle (HEV: Hybrid Electric Vehicle), a fuel cell vehicle (FCV: Fuel Cell Vehicle), a gasoline vehicle, or a diesel vehicle. In addition, the vehicle 200 may also include various sensors that can be used to detect the position and posture of the vehicle 200, such as a GNSS (Global Navigation Satellite System) receiver, an IMU (Inertial Measurement Unit), and a LiDAR (Light Detection And Ranging).

[0018] The camera 10 captures images of the surroundings of the vehicle 200. In the present embodiment, the camera 10 is configured as a single-lens reflex camera that captures images of the front of the vehicle 200. The camera 10 is not limited to capturing images of the front of the vehicle 200, but may also capture images of the side or rear thereof.

[0019] The communication device 20 communicates with the vehicle-mounted device 400 mounted on another vehicle 500 located around the vehicle 200, and obtains object information from the vehicle-mounted device 400. "Object information" refers to information related to the vehicle 500, which is an object included in the image captured by the camera 10. In this embodiment, the vehicle 500 corresponds to the "other moving body" in the present disclosure, and the vehicle-mounted device 400 corresponds to the "external device" in the present disclosure.

[0020] In the present embodiment, the vehicle-mounted device 400 includes a CPU 410, a communication device 420, and a memory 430. The vehicle-mounted device 400 transmits the object information pre-stored in the memory 430 from the communication device 420 to the control device 30 according to the control performed by the CPU 410. For example, when the distance between the vehicle 500 and the vehicle 200 is less than a preset distance, the vehicle-mounted device 400 transmits the object information to the communication device 20. In the present embodiment, the vehicle-mounted device 400 pre-stores the feature point information FI in the memory 430 as the object information. The "feature point information FI" refers to information indicating the positions of a plurality of feature points (hereinafter also referred to as "object feature points") corresponding to at least a portion of the vehicle 500, the feature amount of each object feature point, and the type of feature amount algorithm for calculating the feature amount of each object feature point. For example, as feature point information FI, the vehicle-mounted device 400 pre-stores coordinates of multiple feature points representing the taillight part of the vehicle 500 based on the center of gravity of the vehicle 500, the feature values ​​of each of the multiple feature points, and the type of feature value algorithm used to calculate the feature values.

[0021] In the present embodiment, the vehicle-mounted device 400 pre-stores a plurality of feature quantities calculated by a plurality of feature quantity algorithms for each object feature point. The types of feature quantity algorithms for the object feature points stored in the vehicle-mounted device 400 will be described later. The vehicle-mounted device 400 determines, with the scale calculation unit 312 described later, which feature quantity is calculated by which of the plurality of feature quantity algorithms to send as feature point information FI for each object feature point. Furthermore, as feature point information FI, the vehicle-mounted device 400 sends, to the communication device 20, a feature quantity calculated by the determined type of feature quantity algorithm from among the plurality of feature quantities for each object feature point. In the present embodiment, the vehicle-mounted device 400 receives a usable feature quantity algorithm sent by the scale calculation unit 312 described later, and sends feature point information FI including a feature quantity calculated by one of the plurality of feature quantities that can be used by the scale calculation unit 312.

[0022] The actuator 40 includes a driving device for accelerating the vehicle 200, a steering device for changing the traveling direction of the vehicle 200, and a braking device for decelerating the vehicle 200. In addition, the actuator 40 may also include an actuator for swinging the wipers of the vehicle 200, an actuator for opening and closing the electric windows of the vehicle 200, and the like.

[0023] The control device 30 estimates the own position of the vehicle 200 using the object information obtained by the communication device 20 and the image obtained by the camera 10. The control device 30 is equivalent to the "own position estimation device" in the present disclosure. In addition, the control device 30 uses the estimated own position to create a driving path and controls the actuator 40 so that the vehicle 200 travels along the driving path.

[0024] In the present embodiment, the control device 30 is configured as an ECU (Electronic Control Unit) having a CPU 31, a ROM 32, and a RAM 33. The CPU 31 functions as a feature point acquisition unit 311, a scale calculation unit 312, a self-position estimation unit 313, a path planning unit 314, and a vehicle control unit 315 by expanding a program pre-stored in the ROM 32 into the RAM 33 and executing the program.

[0025] The feature point acquisition unit 311 acquires feature points (hereinafter also referred to as "image feature points") existing in the image captured by the camera 10, and calculates feature values ​​for each image feature point. The image feature points include not only feature points corresponding to the vehicle 500, but also feature points corresponding to buildings, roads, etc. that are the background part of the image. The scale calculation unit 312 calculates the actual scale of the image captured by the camera 10. "Actual scale" refers to the scale of the length represented on the image relative to the length in the real world. The feature point acquisition unit 311 and the scale calculation unit 312 use a feature value algorithm of a predetermined type to perform calculation of the feature value of the image feature point or calculation of the actual scale. In this embodiment, as feature value algorithms, the feature point acquisition unit 311 and the scale calculation unit 312 can use three feature value algorithms, namely SIFT (Scale Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), and SURF (Speeded Up Robust Features).

[0026] The own position estimation unit 313 estimates the own position of the vehicle 200 using the feature value of the image feature point and the actual scale. The path planning unit 314 uses the estimated own position to create a planned path as the planned travel path of the vehicle 200. The vehicle control unit 315 controls the actuator 40 so that the vehicle 200 travels along the created planned path. The specific processing performed by each functional unit will be described later.

[0027] A-2. Processing in the control device 30:

[0028] The control device 30 is for Figure 2 The processing shown is repeatedly executed while the control device 30 is in the operating state. In step S10, the feature point acquisition unit 311 and the scale calculation unit 312 respectively acquire an image captured by the camera 10. Figure 1 In the embodiment, the image is directly acquired from the camera 10 , but the image may be acquired from the ROM 32 after being temporarily stored in the ROM 32 .

[0029] exist Figure 2 In step S20 shown, the feature point acquisition unit 311 acquires image feature points from the acquired image and calculates feature values ​​for each image feature point. In this embodiment, the feature point acquisition unit 311 uses any one of the available feature value algorithms to acquire image feature points and calculate feature values.

[0030] In step S30, the scale calculation unit 312 executes Figure 3 In step S310 , the scale calculation unit 312 transmits the type of feature quantity algorithm that it can use to the vehicle-mounted device 400 .

[0031] In step S320, the scale calculation unit 312 obtains the feature point information FI obtained by the communication device 20 from the vehicle-mounted device 400. As described above, the vehicle-mounted device 400 sends the feature point information FI including the feature quantity calculated by a certain feature quantity algorithm that can be used by the scale calculation unit 312, and the scale calculation unit 312 obtains the feature point information FI. Figure 1 In the embodiment, the feature point information FI is directly obtained from the communication device 20 , but the feature point information FI may be temporarily stored in the ROM 32 from the communication device 20 and then obtained from the ROM 32 .

[0032] exist Figure 3 In step S330 shown, the scale calculation unit 312 obtains feature points (hereinafter also referred to as "object image feature points") corresponding to the vehicle 500 as the object corresponding to the feature point information FI from the image, and calculates the feature amount for each object image feature point. In the present embodiment, the scale calculation unit 312 obtains image feature points from the image. Furthermore, the scale calculation unit 312 performs matching of the obtained plurality of image feature points with the plurality of object feature points represented by the feature point information FI, and obtains, as object image feature points, image feature points that are similar in positional relationship to the plurality of object feature points and in feature amounts of each object feature point from the plurality of image feature points. At this time, the scale calculation unit 312 calculates the feature amounts of the image feature points and the object image feature points using the same type of algorithm as the feature amount algorithm represented by the feature point information FI.

[0033] In the present embodiment, when there are multiple vehicles in the image, the scale calculation unit 312 preferentially selects the vehicle for which the object feature point is obtained. The scale calculation unit 312 selects, for example, a vehicle that is closer to the vehicle 200. In addition, the scale calculation unit 312 may use the multiple images that have been obtained and are continuous in time series to select a vehicle with a larger amount of movement of the object feature points that correspond to each other between the images, in other words, a vehicle with a larger relative speed to the vehicle 200 or a vehicle that is traveling in the direction that crosses the front of the vehicle 200. On the other hand, the scale calculation unit 312 may use the multiple images that have been obtained and are continuous in time series to select a vehicle with a smaller amount of movement of the object feature points that correspond to each other between the images, in other words, a vehicle that is traveling in the same direction as the vehicle 200. In addition, since the actual scale is uncertain at the time of this step, the above-mentioned "distance to the vehicle 200" and "relative speed to the vehicle 200" may also be calculated in pixel units. By preferentially selecting the vehicle from which the object feature points are acquired as described above, the number of object feature points used in the calculation of the actual scale described later can be reduced, and an increase in the calculation load for the calculation of the actual scale can be reduced.

[0034] In step S340, the scale calculation unit 312 calculates the actual scale using the feature values ​​of the object image feature points and the feature values ​​of the object feature points. More specifically, the scale calculation unit 312 calculates the actual scale by calculating the ratio of the distances between the plurality of object feature points represented by the feature point information FI and the distances between the plurality of image object feature points corresponding to the plurality of object feature points. The scale calculation unit 312 determines which object feature point each object image feature point corresponds to by determining the positional relationship between each object image feature point and each object feature point and whether each feature value is consistent.

[0035] exist Figure 2 In step S40 shown, the own position estimation unit 313 estimates the own position of the vehicle 200 using the acquired image feature points and the calculated actual scale. More specifically, the own position estimation unit 313 uses image analysis techniques such as SfM (Structure from Motion) and CNN (Convolutional Neural Network) to calculate the three-dimensional position of each image feature point in pixel units based on multiple images that are continuous in time series. The scale calculation unit 312 uses the actual scale to transform the calculated three-dimensional position in pixel units into a three-dimensional position in meter units. Thus, the scale calculation unit 312 can estimate the relative position, vehicle speed, and travel direction of the vehicle 200, the signs and traffic lights set on the road, and the buildings on the shoulder of the road, etc., around the vehicle 200.

[0036] In step S50, the path planning unit 314 uses the estimated position of the vehicle 200 to create a planned path. More specifically, the path planning unit 314 generates a short-term path plan such as the distance between the vehicle 200 and the vehicle 500, the adjustment of the driving position of the vehicle 200 in the driving lane, etc. based on the surrounding conditions of the vehicle 200 such as the signal or the distance to the surrounding vehicles. In addition, when the vehicle 200 is equipped with a GNSS receiver and can detect the absolute position of the vehicle 200 at a certain point in the past through positioning information, the path planning unit 314 can also create a long-term path plan from the current absolute position of the vehicle 200 calculated based on the relative movement amount from the point where the positioning information is interrupted to a predetermined destination. In addition, the path plan is not limited to the driving path of the vehicle 200, and can also include information on the vehicle control of the vehicle 200 at each point on the driving path, such as the speed and direction of travel of the vehicle 200.

[0037] In step S60, the vehicle control unit 315 controls the actuator 40 so that the vehicle 200 travels according to the created route plan. The control device 30 repeatedly executes the above-described processing.

[0038] According to the control device 30 of the first embodiment described above, the actual scale is calculated using the feature value of the feature point of the object image and the object information, and the self-position is estimated using the image feature point and the actual scale. Therefore, even in an environment where the GNSS receiver cannot fully receive the positioning information, the self-position can be estimated using the camera 10 mounted on the mobile body without using other sensors, which can suppress the decrease in the accuracy of the self-position estimation. In addition, since the object information obtained from the vehicle-mounted device 400 is used, it is possible to suppress the storage capacity of ROM32 from being compressed compared to the method of pre-making a database of feature points of landmarks and storing them in ROM32.

[0039] In addition, since feature point information FI representing the positions of multiple object feature points, the feature values ​​of each object feature point, and the type of feature value algorithm for calculating the feature values ​​of each object feature point is obtained from the vehicle-mounted device 400, the actual scale can be easily calculated using the feature values ​​of the object image feature points and the feature values ​​of the object feature points represented by the feature point information FI.

[0040] B. Second Implementation Method:

[0041] like Figure 4 As shown, the control device 30A of the second embodiment further includes a rendering processing unit 316, and performs a real-scale calculation process. Figure 5 Step S312 and step S322 shown in FIG. Figure 3 The steps S310 and S320 shown are different from the control device 30 of the first embodiment in this respect. The device structure of the control device 30A of the second embodiment and other procedures in the processing performed by the control device 30 are the same as those of the control device 30 of the first embodiment, so the same reference numerals are given to the same structure and the same procedure, and their detailed description is omitted.

[0042] like Figure 4 As shown, in this embodiment, the vehicle-mounted device 400 stores in advance a three-dimensional model TDM representing the three-dimensional shape of the vehicle 500 as object information in the memory 430. The three-dimensional model TDM includes information representing the dimensions of the vehicle 500 in the real world.

[0043] The rendering processing unit 316 obtains the three-dimensional model TDM from the vehicle-mounted device 400 through the communication device 20, and renders the three-dimensional model TDM to obtain a two-dimensional image. In the present embodiment, the rendering processing unit 316 renders the three-dimensional model TDM to produce four two-dimensional images of the vehicle 500 observed from four directions: front, back, left, and right. The rendering processing unit 316 obtains the feature points (hereinafter referred to as "model feature points") existing in the transformed two-dimensional image, and calculates the feature value for each model feature point. The rendering processing unit 316 calculates the feature value of the model feature point using a feature value algorithm of a predetermined type, similar to the feature point acquisition unit 311 and the scale calculation unit 312 described above. In the present embodiment, as the feature value algorithm, the rendering processing unit 316 can use three feature value algorithms: SIFT, ORB, and SURF.

[0044] like Figure 5 As shown, in step S312, the rendering processing unit 316 obtains the three-dimensional model TDM obtained by the communication device 20 from the vehicle-mounted device 400, and renders the three-dimensional model TDM to obtain a two-dimensional image.

[0045] In step S322, the rendering processing unit 316 obtains model feature points based on the created two-dimensional image and calculates feature values ​​for each model feature point. In this embodiment, the rendering processing unit 316 calculates feature values ​​of the model feature points using any of the three feature value algorithms that can be used above.

[0046] In step S330, the scale calculation unit 312 obtains the object image feature points and calculates the feature quantity for each object feature point. In the present embodiment, the scale calculation unit 312 obtains the image feature points from the image. Furthermore, the scale calculation unit 312 performs matching between the obtained multiple image feature points and the multiple model feature points, and obtains the image feature points that are similar to the positional relationship with the multiple model feature points and the feature quantity of each model feature point as the object image feature points. At this time, the scale calculation unit 312 uses the same type of feature quantity algorithm as the feature quantity algorithm used to calculate the feature quantity of the model feature points to calculate the feature quantity of the image feature points and the object image feature points.

[0047] In step S340, the scale calculation unit 312 calculates the actual scale using the feature values ​​of the model feature points and the feature values ​​of the image object feature points. More specifically, the scale calculation unit 312 calculates the actual scale by calculating the ratio of the distance between the plurality of model feature points to the distance between the plurality of image object feature points respectively corresponding to the plurality of model feature points.

[0048] According to the control device 30A of the second embodiment described above, the three-dimensional model TDM of the vehicle 500 is obtained as object information, and the actual scale is calculated using the feature values ​​of the model feature points and the feature values ​​of the object image feature points obtained by rendering the three-dimensional model TDM. Therefore, it is possible to use an arbitrary feature value algorithm to calculate the feature values ​​of the model feature points and the feature values ​​of the object image feature points, and compared with a method of referring to feature values ​​calculated using a feature value algorithm of a predetermined type, it is possible to suppress the situation where the actual scale cannot be calculated due to inconsistency of the feature value algorithm. In addition, compared with a method of obtaining feature values ​​of feature points of a predetermined part of the vehicle 500, it is possible to use feature values ​​of feature points of an arbitrary part of the vehicle 500 to calculate the actual scale.

[0049] C. 3rd Implementation Method:

[0050] The control device 30 of the third embodiment is different from the control device 30 of the first embodiment in that the position of the vehicle 500 is also obtained from the vehicle-mounted device 400 as object information, and in step S40 of the actual scale calculation process, the position of the vehicle 500 is used to perform bundle adjustment. The device structure of the control device 30 of the third embodiment and other procedures in the processing performed by the control device 30 are the same as those of the control device 30 of the first embodiment, so the same reference numerals are given to the same structure and the same procedure, and their detailed description is omitted.

[0051] In the present embodiment, the vehicle 500 is provided with various sensors (not shown) such as a GNSS receiver, an IMU, and a LiDAR that can be used to detect the position and posture of the vehicle 500, and is configured to detect the position of the vehicle 500. In addition to sending the above-mentioned feature point information FI to the control device 30, the vehicle-mounted device 400 also sends the position of the vehicle 500 as object information.

[0052] exist Figure 2 In step S40 shown, the self-position estimation unit 313 uses the position of the vehicle 500 obtained as object information to perform bundling adjustment and correct the reprojection error of the image feature points. More specifically, in the bundling adjustment, the self-position estimation unit 313 estimates the three-dimensional coordinates of the subject corresponding to each image feature point based on the correspondence between the image feature points of a plurality of images that are continuous in time series. Furthermore, the self-position estimation unit 313 reprojects the three-dimensional coordinates of the subject corresponding to each image feature point onto the image plane based on the self-position of the vehicle 200 when each image is obtained, in other words, based on the position and posture of the camera 10. The self-position estimation unit 313 corrects the self-position of the vehicle 200 when each image is obtained so that the reprojection error calculated as the distance between the reprojection point and the image feature point is minimized.

[0053] Generally, in the bundling adjustment, the self-position estimation unit 313 calculates the reprojection error by referring to the image feature points corresponding to the background part, under the premise that the background part of the image in which the road, building, etc. as a fixed object is reflected does not move in the real world. However, since the vehicle 500 moves between a plurality of images that are continuous in a time series, the above premise does not hold. Therefore, when the position of the vehicle 500 is not used, since the coordinates in the real world of the part corresponding to the image feature points of the vehicle 500 are unknown, the image feature points corresponding to the vehicle 500 cannot be referred to, and the self-position estimation unit 313 only refers to the image feature points corresponding to the background part to calculate the reprojection error. In contrast, when the position of the vehicle 500 is used, since the coordinates in the real world of the part corresponding to the image feature points of the vehicle 500 can be determined, the self-position estimation unit 313 can refer to the image feature points corresponding to the background part and the image feature points corresponding to the vehicle 500 to calculate the reprojection error. In this way, when the position of the vehicle 500 is used as object information, the reprojection error can be calculated with reference to more image feature points, compared to the case where the position of the vehicle 500 is not used, and the decrease in the accuracy of the own position estimation can be further suppressed.

[0054] The control device 30 according to the third embodiment described above has the same effect as the first embodiment described above. In addition, since the following processing is also performed: object information including the position of the vehicle 500 is obtained, and the reprojection error to the image feature point is corrected by bundling adjustment using the position of the vehicle 500, the decrease in the accuracy of the self-position estimation can be further suppressed.

[0055] D. Other implementation methods:

[0056] (D1) In the above embodiment, the control device 30 is mounted on the vehicle 200, but the present disclosure is not limited to this. The control device 30 can be mounted on any mobile object such as an aircraft, a robot, a ship, etc. In addition, the vehicle-mounted device 400 can also be mounted on any mobile object such as an aircraft, a robot, a ship, etc.

[0057] (D2) In the above embodiment, the camera 10 is configured as a monocular camera, but the present disclosure is not limited to this. The camera 10 can also be configured as a stereo camera. In this form, the self-position estimation unit 313 can also use stereo matching to calculate the three-dimensional position of each image feature point. Through this form, the same effect as the above embodiment is achieved. In addition, since the actual scale calculated by stereo matching can be compared with the actual scale calculated by the above processing performed by the scale calculation unit 312, the actual scale can be calculated with better accuracy, and the decrease in the accuracy of the self-position estimation can be further suppressed.

[0058] (D3) In the above embodiment, the control device 30 is configured as an ECU having a CPU 31, but the present disclosure is not limited thereto. The control device 30 may also be configured as a SoC (System on Chip) having a GPU (Graphics Processing Unit), an NPU (Neural network Processing Unit), a DSP (Digital Signal Processor), etc., instead of the CPU 31. This form also achieves the same effect as the above embodiment.

[0059] (D4) In the above-mentioned embodiment, the feature point acquisition unit 311, the scale calculation unit 312, the self-position estimation unit 313, the path planning unit 314 and the vehicle control unit 315 are implemented in the same ECU, but the present disclosure is not limited to this. For example, the ECU that implements the feature point acquisition unit 311, the scale calculation unit 312 and the self-position estimation unit 313 as the functional unit for realizing the self-position estimation, and the ECU that implements the path planning unit 314 and the vehicle control unit 315 as the functional unit for realizing the vehicle control using the estimated self-position may be provided independently of each other. In this form, the ECU that implements the feature point acquisition unit 311, the scale calculation unit 312 and the self-position estimation unit 313 is equivalent to the "self-position estimation device" in the present disclosure.

[0060] (D5) In the above embodiment, as feature quantity algorithms, the feature point acquisition unit 311 and the scale calculation unit 312 can use three feature quantity algorithms, namely SIFT, ORB and SURF, respectively, but the present disclosure is not limited to this. The feature point acquisition unit 311 and the scale calculation unit 312 can also use feature quantity algorithms of different types. In addition, the types of feature quantity algorithms are not limited to the above three types, and any known feature quantity algorithms such as HoG (Histograms of Oriented Gradients) or CNN (Convolutional Neural Network) can also be used. In addition, the number of feature quantity algorithms that can be used by the feature point acquisition unit 311 and the scale calculation unit 312 can also be more than 4, or it can be two. Through this form, the same effect as the above embodiment is achieved.

[0061] In addition, the scale calculation unit 312 may be configured to use only one feature quantity algorithm. In this form, the scale calculation unit 312 may calculate the actual scale when the feature quantity algorithm indicated by the received feature point information FI is a usable type. With this form, the actual scale can be calculated when the feature quantity algorithm indicated by the received feature point information FI is a usable type, and the decrease in the accuracy of the self-position estimation can be suppressed.

[0062] (D6) In the above embodiment, the scale calculation unit 312 obtains object information from the vehicle-mounted device 400 mounted on the vehicle 500, but the present disclosure is not limited to this. The scale calculation unit 312 may also obtain the feature point information FI of the ground object and the three-dimensional model TDM and the position of the ground object as object information from an external device provided on a ground object such as a sign or a traffic light pre-set around the vehicle 200.

[0063] (D7) In the above embodiment, while the control device 30 is in the operating state, the control device 30 repeatedly executes Figure 2 The control device 30 may also start or stop the process according to the driving environment of the vehicle 200. Figure 2 For example, when the vehicle 200 is configured to detect the position of the vehicle by receiving positioning information through a GNSS receiver, the control device 30 may start the process in an environment such as a town or a tunnel where the positioning information cannot be stably received. Figure 2 The processing shown in the figure ends when the positioning information can be stably received. According to this aspect, since the vehicle is not executed in a driving environment where the vehicle can detect its own position through the positioning information, Figure 2 The processing shown in the figure can suppress an increase in power consumption of the control device 30.

[0064] (D8) In the above embodiment, if Figure 2 As shown, the control device 30 executes step S30 after step S20, but the present disclosure is not limited thereto. The control device 30 may also execute step S30 in parallel with step S20. According to this aspect, since a plurality of processes can be executed in parallel, the processes in the control device 30 can be executed efficiently.

[0065] (D9) In the above embodiment, when there are multiple vehicles in the image, the scale calculation unit 312 preferentially selects the vehicle for which the object feature points are obtained, but the present disclosure is not limited to this. The scale calculation unit 312 may also obtain the object feature points of multiple vehicles existing in the image and calculate the actual scale. According to this form, compared with the form of obtaining the object feature points of one vehicle as the object, more object feature points can be used to calculate the actual scale, so the actual scale can be calculated with better accuracy, and the decrease in the accuracy of the self-position estimation can be further suppressed.

[0066] (D10) In the above-mentioned embodiment, the scale calculation unit 312 sends the feature quantity algorithm that can be used to the vehicle-mounted device 400, and the vehicle-mounted device 400 sends the feature quantity calculated by a certain feature quantity algorithm that can be used by the scale calculation unit 312, among the multiple feature quantities calculated by the multiple feature quantity algorithms for each object feature point, but the present disclosure is not limited to this. The vehicle-mounted device 400 may also send the feature quantity calculated by two or more feature quantity algorithms that can be used by the scale calculation unit 312, among the multiple feature quantities calculated by the multiple feature quantity algorithms for each object feature point. According to this form, the degree of freedom of selection of the type of feature quantity algorithm used by the scale calculation unit 312 can be suppressed from decreasing.

[0067] In addition, the vehicle-mounted device 400 may send multiple feature quantities calculated by two or more feature quantity algorithms for each object feature point regardless of the type of feature quantity algorithm that the scale calculation unit 312 can use. In the case where there are multiple feature quantity algorithms that can be used among the feature quantity algorithms for calculating each of the multiple feature quantities received as the feature point information FI, the scale calculation unit 312 may determine the type of feature quantity algorithm to be used according to a preset priority order, and use the feature quantity of the object image feature point calculated using the determined type of feature quantity algorithm and the feature quantity of the object feature point to calculate the actual scale. Regarding the priority order, for example, when the accuracy of the self-position estimation is emphasized, it is set to prioritize SIFT as the feature quantity algorithm, and when the reduction of the calculation load is emphasized, it is set to prioritize ORB as the feature quantity algorithm. In addition, the priority order may be set in any order regardless of the above. According to this form, when multiple feature quantities calculated by two or more feature quantity algorithms for each object feature point are sent, it is easy to determine which feature quantity algorithm to use.

[0068] (D11) In the above-mentioned embodiment, the feature point acquisition unit 311 and the scale calculation unit 312 respectively acquire images, use a feature quantity algorithm, acquire image feature points and object image feature points, and thereby calculate feature quantities, but the present disclosure is not limited to this. Alternatively, only the feature point acquisition unit 311 may acquire an image, acquire image feature points and object image feature points to calculate each feature quantity. The scale calculation unit 312 may also use the object image feature points acquired by the feature point acquisition unit 311 to calculate the actual scale. That is, the scale calculation unit 312 may acquire object feature points from the feature point acquisition unit 311 without using a feature quantity algorithm. In this form, the feature point acquisition unit 311, like the scale calculation unit 312 in the above-mentioned embodiment, communicates with the vehicle-mounted device 400 to determine the type of feature quantity algorithm that can be used. According to this form, since only the feature point acquisition unit 311 uses the feature quantity algorithm to obtain image feature points and object image feature points, and the scale calculation unit 312 obtains object feature points from the feature point acquisition unit 311 without using the feature quantity algorithm, the increase in computational load in the control device 30 can be suppressed.

[0069] (D12) In the above embodiment, the rendering processing unit 316 renders the three-dimensional model TDM to produce four two-dimensional images obtained by observing the vehicle 500 from four directions of front, back, left, and right, but the present disclosure is not limited to this. The rendering processing unit 316 may also produce eight two-dimensional images obtained by observing the vehicle 500 from eight directions of front, back, left, right, and oblique directions. According to this form, it is possible to suppress the difference between the posture of the vehicle 500 in the two-dimensional image produced by rendering and the posture of the vehicle 500 in the image obtained by the camera 10 from becoming larger, and it is possible to suppress the rotation processing used for matching the model feature points with the object image feature points from becoming complicated.

[0070] In addition, the rendering processing unit 316 may only generate one two-dimensional image obtained by observing the vehicle 500 from one preset direction. The posture of the vehicle 500 in the two-dimensional image is preferably a posture that is more likely to appear in the image obtained by the camera 10. For example, it can be said that the posture in the two-dimensional image obtained by observing the three-dimensional model TDM from the rear is more likely to appear in the image obtained by the camera 10 than the posture in the two-dimensional image obtained by observing from an oblique direction. According to this form, since the number of two-dimensional images generated in the rendering process can be suppressed, the increase in the calculation load of the rendering process can be suppressed.

[0071] (D13) In the above-mentioned embodiment, the control device 30 obtains the feature point information FI and the position of the vehicle 500 as object information and performs the self-position estimation, but the present disclosure is not limited to this. The control device 30 may also obtain only the position of the vehicle 500 as object information and perform the self-position estimation. For example, the scale calculation unit 312 may also calculate the actual scale by calculating the ratio of the movement amount of the vehicle 500 in pixel units in a plurality of images that are continuous in time series to the actual movement amount of the vehicle 500, and the self-position estimation unit 313 performs the self-position estimation using the actual scale. The "movement amount of the vehicle 500 in pixel units" is obtained, for example, by calculating the displacement of the positional relationship between the image feature points corresponding to the vehicle 500 and the image feature points corresponding to the background part in each image. In addition, in this form, the correspondence between the vehicle existing in the image and the vehicle that has sent the position information can be determined by, for example, comparing the absolute position of the vehicle 500 calculated by image analysis with the position of the vehicle 500 obtained as the object information. The calculation of the absolute position of the vehicle 500 based on the image analysis can be realized by using the relative position of the vehicle 200 and the vehicle 500 calculated by the image analysis, and the absolute position of the vehicle 200. In addition, when the absolute position of the vehicle 200 is detected at a certain point in the past by the positioning information as described above, the absolute position of the vehicle 200 can be calculated based on the relative movement amount calculated by the image analysis from the point where the positioning information is interrupted. According to this form, since the self-position estimation can be performed using only the position of the vehicle 500 as the object information, the amount of information obtained from the vehicle-mounted device 400 as the object information can be reduced.

[0072] The control device 30 and the method thereof described in the present disclosure may also be implemented by a dedicated computer provided by a processor and a memory programmed to execute one or more functions embodied by a computer program. Alternatively, the control device 30 and the method thereof described in the present disclosure may also be implemented by a dedicated computer provided by a processor composed of one or more dedicated hardware logic circuits. Alternatively, the control device 30 and the method thereof described in the present disclosure may also be implemented by one or more dedicated computers as described below, which are composed of a combination of a processor and a memory programmed to execute one or more functions and a processor composed of one or more dedicated hardware logic circuits. In addition, the computer program may also be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer.

[0073] The present disclosure is not limited to the above-mentioned embodiments, but can be implemented by various structures within the scope of its main purpose. For example, the technical features in each embodiment corresponding to the technical features in the technical solution recorded in the invention content column can be appropriately replaced or combined in order to solve part or all of the above-mentioned problems or to achieve part or all of the above-mentioned effects. In addition, as long as the technical features are not described as necessary in this specification, they can be appropriately removed.

Claims

1. A self-position estimation device, provided on a mobile body having a camera for photographing surroundings and a communication device, for estimating the self-position of the mobile body, The self-position estimation device comprises: A feature point acquisition unit that acquires feature points existing in the image captured by the camera, namely, image feature points; A scale calculation unit, which calculates the actual scale of the image; and The self-position estimation unit estimates the self-position using the image feature points and the actual scale. The scale calculation unit above performs: acquiring object information, the object information being acquired by the communication device from an external device located around the mobile body, the object information being information related to the object existing in the image; Obtaining an object image feature point, where the object image feature point is a feature point of the object corresponding to the object information among the image feature points; The actual scale is calculated using the feature values ​​of the feature points of the object image and the object information.

2. The self-position estimation device according to claim 1, The external device is installed on another moving body located around the moving body.

3. The self-position estimation device according to claim 2, The object information includes the location of the object. The self-position estimation unit further performs the following processing: using the position of the object, a reprojection error onto the image feature point is corrected by bundling adjustment.

4. The self-position estimation device according to claim 1, The external device is installed on a ground object provided around the moving body.

5. The self-position estimation device according to claim 4, The object information includes the location of the object. The self-position estimation unit further performs the following processing: using the position of the object, a reprojection error onto the image feature point is corrected by bundling adjustment.

6. The self-position estimation device according to any one of claims 1 to 5, The object information includes feature point information, the feature point information indicating the positions of a plurality of feature points corresponding to at least a portion of the object, i.e., a plurality of object feature points, a feature quantity of each of the object feature points, and a type of a feature quantity algorithm for calculating the feature quantity of each of the object feature points; The scale calculation unit obtains a feature point corresponding to the object feature point from among the image feature points as the object image feature point. The scale calculation unit calculates the actual scale using the feature amount of the object image feature point and the feature amount of the object feature point.

7. The self-position estimation device according to claim 6, The external device stores in advance a plurality of feature quantities calculated by a plurality of feature quantity algorithms for each of the feature points of the object. The above external device executes: Determining, with respect to each of the object feature points, which feature quantity calculated by the feature quantity algorithm among the plurality of feature quantity algorithms is to be sent as the feature point information, between at least one of the feature point acquisition unit and the scale calculation unit; For each of the object feature points, a feature amount calculated by the feature amount algorithm of the determined type among the plurality of feature amounts is transmitted to the communication device as the feature point information.

8. The self-position estimation device according to claim 6, The external device pre-stores a plurality of feature quantities calculated respectively by a plurality of the feature quantity algorithms for each of the feature points of the object, and sends a plurality of feature quantities calculated respectively by two or more of the feature quantity algorithms among the plurality of feature quantities for each of the feature points of the object as the feature point information to the communication device.

9. The self-position estimation device according to claim 6, The scale calculation unit is configured to be able to use a plurality of feature quantity algorithms, and calculates the actual scale by utilizing the feature quantity of the feature points of the object image calculated using a feature quantity algorithm of the same type as the feature quantity of the object feature points received as the feature point information, and the feature quantity of the object feature points.

10. The self-position estimation device according to claim 6, The scale calculation unit is configured to be able to use a plurality of the feature quantity algorithms. The external device pre-stores a plurality of feature quantities calculated by a plurality of feature quantity algorithms for each of the feature points of the object, and sends a plurality of feature quantities calculated by two or more of the feature quantity algorithms from the plurality of feature quantities to the communication device as the feature point information for each of the feature points of the object. When there are a plurality of usable feature quantity algorithms among the feature quantity algorithms for respectively calculating the plurality of feature quantities received as the feature point information, the scale calculation unit determines the type of the feature quantity algorithm to be used according to a preset priority order, The scale calculation unit calculates the actual scale by using the feature amount of the object image feature point calculated using the feature amount algorithm of the determined type and the feature amount of the object feature point.

11. The self-position estimation device according to any one of claims 1 to 5, The object information includes a three-dimensional model of the object. The self-position estimation device further includes a rendering processing unit, which renders the three-dimensional model to obtain feature points of the three-dimensional model, namely, model feature points. The scale calculation unit obtains a feature point corresponding to the model feature point from among the image feature points as the object image feature point. The scale calculation unit calculates the actual scale using the feature values ​​of the object image feature points and the feature values ​​of the model feature points.

Images