Self-localization device, self-localization method, and self-localization program

Abstract

To provide a self-position estimation device, a self-position estimation method, and a self-position estimation program that can reduce a processing time.SOLUTION: A self-position estimation device 1 includes: an image data acquisition unit 11 that acquires image data output from a multi-camera 20 composed of a plurality of cameras mounted on a moving object MC at each set interval; a feature point extraction unit 12 that extracts feature points from the image data; a memory unit 13 that stores position information on each feature point in each piece of image data; a flow creation unit 14 that creates flows, which are sets of corresponding feature points in different image data in each time series; a selection unit 15 that selects flows satisfying constraint conditions as correct flows using the predetermined constraint conditions including epipolar constraints calculated based on a predicted motion of the moving object; and a self-position calculation unit 16 that calculates a position and an orientation of the moving object using information on the correct flows selected by the selection unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to a self-localization device, a self-localization method, and a self-localization program. [Background technology]

[0002] Conventionally, methods for estimating one's own position using SLAM (Simultaneous Localization And Mapping) are known, as shown in Patent Document 1, for example. SLAM is a technology that uses a mobile device equipped with cameras and sensors to simultaneously estimate its own position and create an environmental map, which is a three-dimensional map, without relying on satellite systems such as GPS. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-62807 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the above-mentioned conventional technology has the problem that the amount of processing required for self-localization to create a 3D map is large and the processing time is long. This disclosure was created in view of the above-mentioned points, and its purpose is to provide a self-localization device, a self-localization method, and a self-localization program that can shorten the processing time. [Means for solving the problem]

[0005] This disclosure can be implemented in the following forms:

[0006] According to one embodiment of this disclosure, a self-position estimation device is provided. This self-position estimation device is a self-position estimation device for estimating the self-position of a moving object (MC), and includes an image data acquisition unit (11) that acquires image data output from a multi-camera (20) consisting of a plurality of cameras (21, 22, 23, 24) mounted on the moving object at set intervals, a feature point extraction unit (12) that extracts feature points from the image data acquired by the image data acquisition unit, a storage unit (13) that stores the position information of each feature point in each of the image data, and the current feature point which is a feature point in the latest image data and the feature point in past image data. The system includes: a flow creation unit (14) that associates a point, which is a pre-feature point, with a flow, which is a set of corresponding feature points in the image data that differs for each time series; a selection unit (15) that selects from the flows created by the flow creation unit a flow that satisfies predetermined constraints, including an epipolar constraint calculated based on the predicted motion of the moving body, as the correct flow; and a self-position calculation unit (16) that calculates the position and orientation of the moving body using the information of the correct flow selected by the selection unit. The constraints include a reference range for flow length, which is the length of the flow in the continuous image data in a time series, as an indicator for selecting the flow. The selection unit selects from the created flows those whose flow length falls within the reference range. The reference range for flow length is set individually for each of the multiple cameras. In other forms of this disclosure, a self-localization device is provided. This self-localization device estimates the self-localization of a moving object (MC), and includes an image data acquisition unit (11) that acquires image data output from a multi-camera (20) consisting of a plurality of cameras (21, 22, 23, 24) mounted on the moving object at set intervals, a feature point extraction unit (12) that extracts feature points from the image data acquired by the image data acquisition unit, a storage unit (13) that stores the position information of each feature point in each of the image data, and the current feature point which is a feature point in the latest image data and the feature point in past image data. The system includes: a flow creation unit (14) that associates a point with a pre-feature point and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series; a selection unit (15) that selects from the flows created by the flow creation unit a flow that satisfies predetermined constraints, including an epipolar constraint calculated based on the predicted motion of the moving body, and determines that the flow is the correct flow; and a self-position calculation unit (16) that calculates the position and orientation of the moving body using the information of the correct flow selected by the selection unit. The constraints include a reference range of flow length, which is the length of the flow in the image data that is continuous in the time series, as an indicator for selecting the flow. The selection unit selects from the created flows the ones whose flow length is within the reference range. The reference range of flow length is set according to the motion state of the moving body. In other forms of this disclosure, a self-localization device is provided. This self-localization device estimates the self-localization of a moving object (MC), and includes an image data acquisition unit (11) that acquires image data output from a multi-camera (20) consisting of a plurality of cameras (21, 22, 23, 24) mounted on the moving object at set intervals, a feature point extraction unit (12) that extracts feature points from the image data acquired by the image data acquisition unit, a storage unit (13) that stores the position information of each feature point in each of the image data, and the current feature point which is a feature point in the latest image data and the feature point in past image data. The system includes: a flow creation unit (14) that associates a point with a pre-feature point and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series; a selection unit (15) that selects from the flows created by the flow creation unit the flows that satisfy predetermined constraints, including an epipolar constraint calculated based on the predicted motion of the moving body, as correct flows; and a self-position calculation unit (16) that calculates the position and orientation of the moving body using the information of the correct flows selected by the selection unit. If the number of correct flows selected by the selection unit is less than a predetermined standard, the selection unit performs the flow selection again using constraints that have been changed from the previously used constraints until the number of correct flows is equal to or greater than the standard.

[0007] This type of self-localization device allows for the estimation of a vehicle's position without creating a 3D map, using image data output from multiple cameras acquired by the image data acquisition unit. The selection unit then uses predetermined constraints, including epipolar constraints, to select flows that satisfy these constraints as the correct flows. By using these constraints, the appropriate search range can be limited when identifying appropriate current feature points within the latest image data, enabling prediction of current feature points and reducing search processing time. As a result, the processing time for self-localization in the self-localization device can be shortened. [Brief explanation of the drawing]

[0008] [Figure 1]It is a block diagram showing a schematic system configuration of the self - position estimation apparatus according to the first embodiment. [Figure 2] It is a plan view showing a vehicle on which the self - position estimation apparatus is mounted. [Figure 3] It is a flowchart showing a procedure for obtaining a feature point list of the current frame. [Figure 4] It is a diagram for explaining epipolar constraint. [Figure 5] It is a diagram showing the motion information of the vehicle dynamically and the position changes of each camera. [Figure 6] It is a histogram showing the statistical distribution of flow length. [Figure 7] It is a histogram showing the statistical distribution of flow length. [Figure 8] It is a histogram showing the statistical distribution of flow length. [Figure 9] It is a histogram showing the statistical distribution of flow length. [Figure 10] It is a diagram showing the motion information of the vehicle dynamically and the position changes of each camera. [Figure 11] It is a histogram showing the statistical distribution of flow length. [Figure 12] It is a histogram showing the statistical distribution of flow length. [Figure 13] It is a histogram showing the statistical distribution of flow length. [Figure 14] It is a histogram showing the statistical distribution of flow length. [Figure 15] It shows an example of an arbitrary current frame, and is a diagram illustrating feature points and flows within the frame. [Figure 16] It shows an example of an arbitrary current frame, and is a diagram illustrating feature points and flows within the frame.

Mode for Carrying Out the Invention

[0009] Hereinafter, the embodiments will be described based on FIGS. 1 to 16. A. First Embodiment: A1. Configuration of the self-localization device 1: The configuration of the self-position estimation device 1 of the first embodiment will be described with reference to Figures 1 and 2. The self-position estimation device 1 of the first embodiment is mounted on an autonomous driving vehicle MC (hereinafter simply referred to as "vehicle MC") as a moving object, and estimates the self-position of the vehicle MC. The vehicle MC can move by autonomous driving based on the self-position estimated by the self-position estimation device 1. Note that "self-position" includes the position and orientation of the moving object, and means position and attitude.

[0010] As shown in Figure 1, the vehicle MC includes a self-position estimation device 1, a vehicle control unit 10, and a camera 20. The vehicle control unit 10 is a functional unit that controls the vehicle MC, and the camera 20 is electrically connected to the vehicle control unit 10. Image data captured by the camera 20 (hereinafter also referred to as "frames") is notified to the vehicle control unit 10. In addition, GNSS (Global Navigation Satellite System) sensors and other sensors (not shown) are electrically connected to the vehicle control unit 10.

[0011] As shown in Figure 2, the camera 20 is configured as a compound-eye multi-camera with a total of four cameras: a front camera 21, a rear camera 22, a right camera 23, and a left camera 24. The front camera 21 acquires peripheral images in front of the vehicle MC. The rear camera 22 acquires peripheral images behind the vehicle MC.

[0012] The right camera 23 acquires peripheral images to the right when viewed in the direction of travel of the vehicle MC. The left camera 24 acquires peripheral images to the left when viewed in the direction of travel of the vehicle MC. Hereafter, unless otherwise specified, cameras 21, 22, 23, and 24 will simply be referred to as "camera 20". Camera 20 is a wide-angle camera, typically a fisheye lens, and the combined field of view of the four cameras covers a 360-degree field of view.

[0013] Refer to Figure 1 again. The self-position estimation device 1 of this embodiment is electrically connected to the vehicle control unit 10 of the vehicle MC. Specifically, since the self-position estimation device 1 of this embodiment is connected to the OBD2 (On Board Diagnostics 2) of the vehicle MC, the self-position estimation device 1 can receive various information from the vehicle control unit 10 of the vehicle MC.

[0014] The connection between the self-position estimation device 1 and the vehicle MC is not limited to this; for example, wireless communication may be used. Alternatively, the self-position estimation device 1 and the vehicle MC may not be connected, and the self-position estimation device 1 may be equipped with a separate camera 20. In this embodiment, power is supplied to the self-position estimation device 1 from the OBD2, but this is not limited; the self-position estimation device 1 may be equipped with a separate battery, or power may be supplied to the self-position estimation device 1 from the vehicle MC's cigarette lighter socket.

[0015] The self-localization device 1 comprises the following functional modules: an image data acquisition unit 11, a feature point extraction unit 12, a storage unit 13, a flow creation unit 14, a sorting unit 15, and a self-localization calculation unit 16. The self-localization device 1 is configured as a well-known computer, comprising a CPU and memory such as ROM or RAM. Each functional module 11 to 16 is realized by the CPU executing a program stored in memory.

[0016] The image data acquisition unit 11 acquires image data of the surrounding environment of the vehicle MC. Specifically, the image data acquisition unit 11 performs the process of acquiring image data output from the camera 20 from the vehicle MC at set intervals.

[0017] The feature point extraction unit 12 performs the process of extracting feature points from image data acquired by the image acquisition unit 11. Feature points include, for example, edges, corners, and color distributions. The feature point extraction method may be an indirect method such as ORB, SIFT, or SURF, or a direct method such as DSO (Direct Sparse Odometry).

[0018] The memory unit 13 performs the process of storing the 3D position coordinates as positional information for each feature point within each image data. In addition, the memory unit 13 stores various data used in a series of processes for self-position estimation of the vehicle MC, as well as calculation results.

[0019] The flow creation unit 14 executes the process of creating a flow, which is a set of corresponding feature points in different image data for each time period in which the image was captured. When creating the flow, the flow creation unit 14 associates the previous feature points, which are feature points in past image data stored in the storage unit 13, with the current feature points, which are feature points in the latest image data. Specifically, the association between the previous feature points in the image data one step prior to the latest and the current feature points is performed for each frame.

[0020] The selection unit 15 performs a process of selecting from the flows created by the flow creation unit 14 that satisfy predetermined constraints as correct flows. The constraints include epipolar constraints calculated based on the predicted motion of the vehicle MC. Details of the constraints will be explained in detail later in the section on the self-position estimation method, along with the control flowchart.

[0021] The self-position calculation unit 16 performs a process of calculating the position and orientation of the vehicle MC using the flow information selected by the selection unit 15. The calculated position and orientation of the vehicle MC, i.e., the position and attitude, become the estimated current position of the vehicle MC. Here, SLAM processing is used for self-position calculation. SLAM is an acronym for Simultaneous Localization And Mapping, and is a method that performs self-position estimation and environmental map creation simultaneously. In this embodiment, however, an environmental map as a 3D map is not created, and the position of the vehicle MC is estimated using only images from the camera 20.

[0022] A2. Self-location estimation method: Next, the self-position estimation method using the self-position estimation device 1 described above will be explained. Figure 3 is a flowchart showing the procedure for obtaining a list of feature points of the current frame to be used for calculating the current (i.e., latest) self-position. "Current frame" is the latest frame, and so on, "previous frame" is the frame immediately preceding the current frame, and "frame before that" is the frame immediately preceding the previous frame. Both "previous frame" and "frame before that" correspond to past image data.

[0023] The feature point list contains the information from the above flow. The process shown in Figure 3 is repeatedly executed by the self-localization device 1 at set intervals. The set interval here is the same as the set interval for shooting by the camera 20.

[0024] As shown in Figure 3, in S101, the previously estimated values ​​of the change in angle and movement of the vehicle MC are read. The "previously estimated values" are the change in angle and movement of the vehicle MC between the frame before last and the previous frame, and are calculated in advance and stored in the memory unit 13. Next, in S102, using the previously estimated values ​​read in S101, a basic matrix is ​​created between the previous frame and the current frame, assuming constant velocity motion. The basic matrix is ​​the matrix used when expressing the epipolar constraint mathematically.

[0025] Next, in S103, the epipolar line is created. The epipolar line is created using a known method with the basic matrix created in S102, the 3D coordinates of the feature points in the current frame and the focal length of the camera 20, and the 3D coordinates of the feature points in the previous frame and the focal length of the camera 20.

[0026] Then, in S104, the upper threshold due to epipolar constraint and the reference range of flow length are determined. This S104 process is performed by the sorting unit 15. Note that the S104 process is not limited to the sorting unit 15; other functional units may also perform it. "Flow length" refers to the length of a single flow connecting a previous feature point and the current feature point corresponding to the previous feature point. "Flow length" is the length of the flow between consecutive image data in a time series; in other words, it refers to the length of the flow between one frame. The upper threshold due to epipolar constraint and the reference range of flow length determined in S104 are collectively called "constraint conditions." The constraint condition that satisfies the upper threshold due to epipolar constraint is called the "first condition," and the constraint condition that satisfies the reference range of flow length is called the "second condition."

[0027] After the constraints are determined in S104, a determination is made in S105 based on the constraints. Specifically, it is determined whether the first and second conditions described above are met. More precisely, if the coordinates of the current feature point are within the upper limit threshold of a predetermined distance from the epipolar line, and the flow length connecting the current feature point and the previous feature point is within the reference range, it is determined that the constraints are met. If either the first or second condition is not met, it is determined that the constraints are not met. This determination is performed for all feature points in the frame. Further details regarding the determination and evaluation of constraints in S104 and S105 will be described later.

[0028] Next, in S106, feature points that satisfy the constraints are added to the "feature point list". The "feature point list" is a list that stores information on feature points and flows used to calculate self-localization after the completion of this control routine. Next, in S107, it is determined whether the number of flows added to the feature point list is equal to or greater than a certain threshold. The threshold number of flows is predetermined by prior testing as the number required for calculating appropriate self-localization. Then, in S107, if the number of flows is equal to or greater than the threshold (S107: Yes), the process proceeds to S109, and the feature point list is output. The feature point list output here is a decimated version from which information on feature points and flows that do not satisfy the constraints has been removed. After processing in S109, this processing routine ends. After that, self-localization is performed using the information in the output feature point list, for example, by the two-point method.

[0029] On the other hand, if the number of flows added to the feature point list in S107 is not equal to or greater than the threshold (S107: No), the process proceeds to S108, and the constraint is relaxed. This is because if the number of flows is too small and below the threshold, the current self-localization calculation cannot be performed, or the accuracy is significantly reduced. Therefore, the constraint is relaxed to increase the number of flows in the feature point list.

[0030] "Relaxing the constraints" specifically means increasing the upper limit threshold for the predetermined distance from the epipolar line, or increasing the reference range for the flow length. After the constraints are relaxed, the process returns to the determination process in S105, and the processes in S108, S105, and S106 are repeated until the number of flows exceeds the reference value in S107.

[0031] A3. Details of the constraints: Next, we will explain the details of determining and judging the constraints in S104 and S105 above. First, we will explain the upper threshold due to the epipolar constraint using Figure 4. Figure 4 is a diagram for explaining the epipolar constraint, and it shows the geometry of a three-dimensional space captured from cameras at different positions. In Figure 4, point e and point e R is the epipole, which is the position where the other camera is imaged from one camera. Also, the epipolar line E is the line that is a candidate for "where the object is imaged when looking from camera O L at X L and the object is imaged" when considering "where the object is imaged when looking from camera O R ".

[0032] As shown in FIG. 4, the object at X P is imaged at X L when viewed from camera O in the previous frame, and is imaged at X L when viewed from camera O in the current frame. When only knowing that "the object is imaged at X R when looking from camera O R ", the position where the object is imaged at camera O L (where X L exists) is limited to "somewhere on the epipolar line E". This is called the epipolar constraint.

[0033] In the case of the conventional method for creating a 3D map, since the approximate position in the depth direction of Xp can be known from two frames, the search range when searching for feature points within the current frame can be two-dimensionally determined within a relatively small range. However, in this embodiment, since there is no 3D map, although it is known that it is on the extension of the X line, which is the straight line connecting O L and X L , the position in the depth direction on the X line is unknown. That is, in FIG. 4, it may be at X2, X1, or X0. Also, although it is known that it is near X R , even when referring to the vicinity of the epipolar line E, the search range for feature points is wide and not determined.

[0034] ​​​​Therefore, in this embodiment, when searching for the current feature point within the current frame, the search range S is defined as the range within a predetermined distance from the epipolar line E. In other words, the search range S is the region within the current frame that is sandwiched between two parallel lines L1 and L2, which are set at predetermined distances on both sides from the epipolar line E. The "distance from the epipolar line E" is the one-dimensional distance between the epipolar line E and the lines L1 and L2 within the current frame, which is a two-dimensional plane in Figure 4. In other words, in S104, the upper limit threshold due to the epipolar constraint, i.e., the search range S, is determined based on the created epipolar line and the pre-set distance.

[0035] Then, in S105, the selection unit 15 selects from the created flows those in which the position coordinates of the current feature point within the current frame are within the search range S as correct flows. The upper threshold due to epipolar constraint, i.e., a predetermined distance from the epipolar line E, is used as an indicator for flow selection. The upper threshold for distance is set in advance using prior test data, taking into account the search processing time and the positional accuracy of the feature points, so as not to make the search range S too wide.

[0036] Next, the determination of the reference range for flow length will be explained with reference to Figures 5 to 14. Figure 5 dynamically shows the motion information of the vehicle MC and the positional changes of each camera 21, 22, 23, and 24, showing the form when turning left in an arbitrary environment. Figures 6 to 9 are histograms showing the frequency distribution of flow length. Figure 6 is a histogram based on data acquired by the front camera 21, and Figure 7 is a histogram based on data acquired by the rear camera 22. Figure 8 is a histogram based on data acquired by the right camera 23, and Figure 9 is a histogram based on data acquired by the left camera 24. In Figures 6 to 9, the horizontal axis represents flow length and the vertical axis represents the occurrence value. The flow length scale is a unitless numerical value for convenience in data processing to see the magnitude of the flow length. The same applies to Figures 11 to 14, which will be described later.

[0037] As shown in Figures 5 to 9, when the vehicle MC turns left, the histograms of flow lengths acquired by each camera 21, 22, 23, and 24 are different. As shown in Figure 6, in the histogram from the front camera 21, the number of occurrences for a flow length of 30 is exceptionally high at approximately 90, followed by the number of occurrences for a flow length of 40 at approximately 60, while the others are small. As shown in Figure 7, in the histogram from the rear camera 22, the number of occurrences for a flow length of 30 is exceptionally high at 60, while the number of occurrences for flow lengths of 20, 40, and 50 is around 20, which is less than half the number of occurrences for a flow length of 30.

[0038] Furthermore, as shown in Figure 8, the histogram from the right camera 23 shows that the occurrence values ​​for flow lengths of 50 and 60 are exceptionally large. As shown in Figure 9, the histogram from the left camera 24 shows that the occurrence values ​​increase as the flow length increases from 10 to 40, with the largest occurrence value being 60 at a flow length of 40. On the other hand, the occurrence values ​​for flow lengths of 50 and 60 are small, at 10 or less.

[0039] As described above, it was found that the statistical distribution of flow length differs for each camera 21-24. Furthermore, it was found that estimation is more accurate when using flows with larger occurrence values, i.e., longer flows. Therefore, in this embodiment, these properties are utilized to appropriately set the reference range of flow length for each camera 21-24. The reference range of flow length is also set according to driving conditions such as right turns and left turns. Note that driving conditions such as right turns and left turns correspond to examples of "movement states of the moving object".

[0040] In this embodiment, for example, for the front camera 21 (see Figure 6), the reference range for the flow length is set to "25 to 45" near the mode of 30 and 40. For the rear camera 22 (see Figure 7), the reference range for the flow length is set to "25 to 35" near the mode of 30. For the right camera 23 (see Figure 8), the reference range for the flow length is set to "45 to 60" near the mode of 50 and 60.

[0041] In other words, the reference range for the flow length during a left turn in the front camera 21 and rear camera 22 is set based on the mode of the flow length. "Based on the mode" may be a form in which the mode is used as the median and an arbitrary range based on empirical rules is taken above or below it, as in the example above, or it may be a form in which values ​​predetermined by testing are taken only above or below the mode. In short, it means any form in which the reference range is set using the "mode" as an indicator. The same applies to "based on the mean" below.

[0042] For the left camera 24 (see Figure 9), the reference range for flow length is set to "25-45," which is close to the average value of 40. In other words, the reference range for flow length in the left camera 24 is set based on the average value of the flow length. Whether to use the mode or the mean as the indicator for the reference range can be appropriately selected for each camera 21, 22, 23, and 24 depending on the shape of the histogram. Furthermore, the acceptable range from the mode or mean is predetermined based on prior test data, etc. Therefore, the specific numerical values ​​of the above reference range can be changed as appropriate.

[0043] The following describes an example of a right turn. Figure 10 dynamically shows the motion information of the vehicle MC and the positional changes of each camera 21, 22, 23, and 24, illustrating the form of a right turn in an arbitrary environment. Figures 11 to 14 are histograms showing the statistical distribution of flow length. Figure 11 is a histogram based on data acquired by the front camera 21, and Figure 12 is a histogram based on data acquired by the rear camera 22. Figure 13 is a histogram based on data acquired by the right camera 23, and Figure 14 is a histogram based on data acquired by the left camera 24.

[0044] As can be seen by comparing Figures 6 to 14 as appropriate, the appearance of objects captured by each camera 21 to 24 differs when turning right and when turning left, and therefore, the histograms of flow length are naturally different for each. As shown in Figure 11, in the histogram from the front camera 21 when turning right, values ​​of 50 or less are dispersed. When the histogram has this form, it is preferable to take the mean rather than the mode. In this case, for example, for the front camera 21, the reference range of flow length is set to "15 to 25", which is close to the mean of 20.

[0045] On the other hand, as shown in Figure 12, in the histogram from the rear camera 22, the number of occurrences of a flow length of 20 is approximately 260, which is extremely large, while the others are small. When the histogram has this shape, it is preferable to set the reference range for the flow length to "15 to 25", which is near the mode of 20. Similarly, as shown in Figures 13 and 14, in the histograms from the right camera 23 and the left camera 24, it is preferable to determine the reference range for the flow length to be, for example, near the mean.

[0046] As described above, histograms from each camera 21-24 are obtained for the current frame, and a reference range for flow length is determined as needed during self-position estimation. In S105, the selection unit 15 selects from the created flows those whose flow length between the current frame and the previous frame falls within the reference range as correct flows. In other words, the reference range obtained from the histogram, which is a statistical representation of flow length, is used as an indicator for flow selection. The above flow distribution changes according to the vehicle MC's movement speed and turning direction. Generally, the above reference range increases as the vehicle MC's movement speed increases. Also, the value tends to be larger for cameras positioned opposite to the vehicle MC's turning direction.

[0047] Figures 15 and 16 show examples of arbitrary current frames, illustrating feature points and flows within the frame. Figure 15 shows a frame obtained by the front camera 21 during a left turn. Figure 16 shows a frame obtained by the right camera 23 during a left turn. In each figure, multiple white circles indicate current feature points C, black lines drawn from feature points indicate erroneous flows Fe, and gray lines drawn from feature points indicate positive flows Fc. An "erroneous flow" is a flow that is determined not to satisfy the constraint conditions by the judgment in S105. A "positive flow" is a flow that is determined to satisfy the constraint conditions by the judgment in S105.

[0048] Thus, there is a difference between the positive flow Fc and the false flow Fe. In the first embodiment, instead of using all of the flow drawn using the extracted current feature points for self-localization, the self-localization of the vehicle MC is calculated using the information of the positive flow Fc shown in Figures 15 and 16.

[0049] [effect] (1) According to the self-position estimation device 1 of the first embodiment described above, the self-position of the vehicle MC can be estimated from images captured by the camera 20 as a multi-camera, without the use of sensors or the like, and without the creation of a 3D map. Therefore, the processing time in the self-position estimation device 1 can be shortened. In addition, since a 3D map is not created, the memory resources in the self-position estimation device 1 can be reduced.

[0050] (2) Furthermore, in the self-localization device 1 of the first embodiment described above, a 3D map is not created, so information in the depth direction of the image cannot be reflected in the current frame. However, in the self-localization device 1 of the first embodiment described above, the range in which feature points are expected to exist is set as a predetermined distance from the epipolar line, and a 2D prediction range based on the epipolar line can be set as the search range S, so that the range in which feature points exist can be predicted, and the search processing time can be shortened.

[0051] (3) According to the self-position estimation device 1 of the first embodiment described above, a predetermined upper threshold value, which is calculated based on the predicted motion of the vehicle MC and is set as a predetermined distance from the epipolar line, is included as an indicator for flow selection. The selection unit 15 then selects from the created flows those in which the position coordinates of the current feature points are within the range of the predetermined upper threshold value, which is set as a predetermined distance from the epipolar line, as correct flows. This improves the accuracy of self-position estimation of the vehicle MC.

[0052] (4) Furthermore, according to the self-position estimation device 1 of the first embodiment described above, a reference range obtained from a histogram as a statistical measure of flow length is included as an indicator for flow selection. Then, the selection unit 15 selects from the created flows those whose flow length falls within the reference range as correct flows. This improves the accuracy of self-position estimation of the vehicle MC.

[0053] Furthermore, in the first embodiment described above, only flows that satisfy both the first and second conditions are selected as correct flows and used for self-position estimation, thereby further improving the accuracy of the vehicle MC's self-position estimation.

[0054] (5) According to the self-position estimation device 1 of the first embodiment described above, the second condition is determined individually for each camera 21 to 24 based on the shape of the histogram obtained from the front, rear, left, and right cameras 21 to 24. That is, an appropriate reference range, such as the vicinity of the mode or the vicinity of the mean, is determined from the shape of the histogram. Therefore, flow can be extracted more accurately.

[0055] (6) According to the self-position estimation device 1 of the first embodiment described above, in S102, the basic matrix is ​​created on the premise of constant velocity motion, so the calculation of the basic matrix can be easily performed.

[0056] (7) According to the self-position estimation device 1 of the first embodiment described above, after determining the constraint conditions (S105), if the number of flows that clear the determination conditions is less than the standard (S107), the previous constraint conditions are relaxed and the determination is made again (S108). This makes it possible to avoid the number of flows used for self-position estimation being excessively small.

[0057] B. Other embodiments: (B1) In the first embodiment described above, the self-position estimation device 1 is mounted on a vehicle MC, but it may also be mounted on other objects such as ships, drones, autonomous mobile robots, or moving objects such as people.

[0058] (B2) In the self-position estimation device 1 of the first embodiment described above, the camera 20 is composed of four cameras, one in the front, one in the back, one in the left, and one in the right, but any two or more compound cameras will suffice.

[0059] (B3) In the self-position estimation device 1 of the first embodiment described above, in S102, a basic matrix was created between the previous frame and the current frame, assuming constant velocity motion. However, in addition to using the previously calculated value, steering information from the vehicle MC's CAN data may also be used to predict the operation.

[0060] (B4) In the self-position estimation device 1 of the first embodiment described above, the mode or mean value in the histogram was used to determine the reference range of the flow length, but the median value may also be used. For example, in statistics, if there are cases where the flow length deviates significantly, the median value may be applied instead of the mean value, or other appropriate measures may be taken.

[0061] (B5) The self-localization device 1 and its methods described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the self-localization device 1 and its methods described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the self-localization device 1 and its methods described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program (self-localization program) may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium.

[0062] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in each embodiment corresponding to the technical features in the embodiments described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0063] 1...Self-position estimation device, 10...Vehicle control unit, 11...Image data acquisition unit, 12...Feature point extraction unit, 13...Storage unit, 14...Flow creation unit, 15...Selection unit, 16...Self-position calculation unit, 20...Camera, 21...Front camera, 22...Rear camera, 23...Right camera, 24...Left camera, MC...Vehicle

Claims

1. A self-position estimation device for estimating the self-position of a moving object (MC), An image data acquisition unit (11) acquires image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the aforementioned mobile body at set intervals, A feature point extraction unit (12) extracts feature points from the image data acquired by the image data acquisition unit, A storage unit (13) that stores the positional information of each feature point in each of the aforementioned image data, A flow creation unit (14) associates current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series. A selection unit (15) selects from the flows created by the flow creation unit a flow that satisfies predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, as the correct flow. The system includes a self-position calculation unit (16) that calculates the position and orientation of the moving body using the correct flow information selected by the sorting unit, The aforementioned constraints include a reference range for flow length, which is the length of the flow in the continuous image data in the time series, as an indicator for selecting the flow. The sorting unit selects from the created flows those whose flow length falls within the reference range. The reference range of the flow length is set individually for each of the multiple cameras in the self-position estimation device.

2. A self-position estimation device for estimating the self-position of a moving object (MC), An image data acquisition unit (11) acquires image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the aforementioned mobile body at set intervals, A feature point extraction unit (12) extracts feature points from the image data acquired by the image data acquisition unit, A storage unit (13) that stores the positional information of each feature point in each of the aforementioned image data, A flow creation unit (14) associates current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series. A selection unit (15) selects from the flows created by the flow creation unit a flow that satisfies predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, as the correct flow. The system includes a self-position calculation unit (16) that calculates the position and orientation of the moving body using the correct flow information selected by the sorting unit, The aforementioned constraints include a reference range for flow length, which is the length of the flow in the continuous image data in the time series, as an indicator for selecting the flow. The sorting unit selects from the created flows those whose flow length falls within the reference range. The reference range of the flow length is set according to the movement state of the moving body in the self-position estimation device.

3. A self-position estimation device for estimating the self-position of a moving object (MC), An image data acquisition unit (11) acquires image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the aforementioned mobile body at set intervals, A feature point extraction unit (12) extracts feature points from the image data acquired by the image data acquisition unit, A storage unit (13) that stores the positional information of each feature point in each of the aforementioned image data, A flow creation unit (14) associates current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series. A selection unit (15) selects from the flows created by the flow creation unit a flow that satisfies predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, as the correct flow. The system includes a self-position calculation unit (16) that calculates the position and orientation of the moving body using the correct flow information selected by the sorting unit, The sorting unit, if the number of selected correct flows is less than a predetermined standard, performs the sorting of flows again using the constraints changed from the previously used constraints until the number of correct flows is equal to or greater than the standard, in a self-localization device.

4. The aforementioned constraints include a predetermined upper threshold value, which is calculated as the distance from the epipolar line based on the predicted motion of the moving object, as an indicator for selecting the flow. The self-localization device according to any one of claims 1 to 3, wherein the selection unit selects from the created flow the one in which the position coordinates of the current feature point are within the range of the epipolar line to the upper threshold in the latest image data.

5. The self-position estimation device according to claim 1 or claim 2, wherein the reference range of the flow length is set using statistics of the flow length.

6. The self-position estimation device according to claim 5, wherein the reference range of the flow length is set based on the mode in the frequency distribution of the flow length.

7. The self-position estimation device according to claim 5, wherein the reference range of the flow length is set based on the mean value in the frequency distribution of the flow length.

8. The self-position estimation device according to claim 5, wherein the reference range of the flow length is set based on the median value in the frequency distribution of the flow length.

9. A self-position estimation method for estimating the self-position of a moving object (MC) using a self-position estimation device, The self-localization device comprises an image data acquisition unit (11), a feature point extraction unit (12), a storage unit (13), a flow creation unit (14), a sorting unit (15), and a self-localization calculation unit (16). The image data acquisition unit acquires image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the mobile body at set intervals. The feature point extraction unit performs the step of extracting feature points from the image data acquired by the image data acquisition unit, The memory unit performs the step of storing the positional information of each feature point in each of the image data, The process involves the flow creation unit associating current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, to create a flow, which is a set of corresponding feature points in different image data for each time series. The selection unit selects from the flows created by the flow creation unit the flows that satisfy predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, and identifies them as correct flows. The self-position calculation unit calculates the position and orientation of the moving body using the correct flow information selected by the selection unit, The aforementioned constraints include a reference range for flow length, which is the length of the flow in the continuous image data in the time series, as an indicator for selecting the flow. The selection process includes selecting from the created flows those whose flow length falls within the reference range. A self-position estimation method in which the reference range of the flow length is set individually for each of the multiple cameras.

10. A self-position estimation method for estimating the self-position of a moving object (MC) using a self-position estimation device, The self-localization device comprises an image data acquisition unit (11), a feature point extraction unit (12), a storage unit (13), a flow creation unit (14), a sorting unit (15), and a self-localization calculation unit (16). The image data acquisition unit acquires image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the mobile body at set intervals. The feature point extraction unit performs the step of extracting feature points from the image data acquired by the image data acquisition unit, The memory unit performs the step of storing the positional information of each feature point in each of the image data, The process involves the flow creation unit associating current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, to create a flow, which is a set of corresponding feature points in different image data for each time series. The selection unit selects from the flows created by the flow creation unit the flows that satisfy predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, and identifies them as correct flows. The self-position calculation unit calculates the position and orientation of the moving body using the correct flow information selected by the selection unit, The aforementioned constraints include a reference range for flow length, which is the length of the flow in the continuous image data in the time series, as an indicator for selecting the flow. The selection process includes selecting from the created flows those whose flow length falls within the reference range. A self-position estimation method in which the reference range of the flow length is set according to the movement state of the moving body.

11. A self-position estimation method for estimating the self-position of a moving object (MC) using a self-position estimation device, The self-localization device comprises an image data acquisition unit (11), a feature point extraction unit (12), a storage unit (13), a flow creation unit (14), a sorting unit (15), and a self-localization calculation unit (16). The image data acquisition unit acquires image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the mobile body at set intervals. The feature point extraction unit performs the step of extracting feature points from the image data acquired by the image data acquisition unit, The memory unit performs the step of storing the positional information of each feature point in each of the image data, The process involves the flow creation unit associating current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, to create a flow, which is a set of corresponding feature points in different image data for each time series. The selection unit selects from the flows created by the flow creation unit the flows that satisfy predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, and identifies them as correct flows. The self-position calculation unit calculates the position and orientation of the moving body using the correct flow information selected by the selection unit, A self-localization method comprising the selection step, which, if the number of selected correct flows is less than a predetermined standard, repeats the selection of flows using constraints modified from the previously used constraints until the number of correct flows equals or exceeds the standard.

12. A self-position estimation program for estimating the self-position of a moving object (MC), The function includes acquiring image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the aforementioned mobile body at set intervals, A function to extract feature points from the aforementioned image data, A function to store the positional information of each feature point in each of the aforementioned image data, A function that associates current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series. A function to select from the created flows that satisfy predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, as the correct flows, The computer is provided with a function to calculate the position and orientation of the moving object using the selected correct flow information. The aforementioned constraints include a reference range for flow length, which is the length of the flow in the continuous image data in the time series, as an indicator for selecting the flow. The aforementioned selection function includes a function for selecting from the created flows those whose flow length falls within the aforementioned reference range. The reference range of the flow length is set individually for each of the multiple cameras by a self-localization program.

13. A self-position estimation program for estimating the self-position of a moving object (MC), The function includes acquiring image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the aforementioned mobile body at set intervals, A function to extract feature points from the aforementioned image data, A function to store the positional information of each feature point in each of the aforementioned image data, A function that associates current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series. A function to select from the created flows that satisfy predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, as the correct flows, The computer is provided with a function to calculate the position and orientation of the moving object using the selected correct flow information. The aforementioned constraints include a reference range for flow length, which is the length of the flow in the continuous image data in the time series, as an indicator for selecting the flow. The aforementioned selection function includes a function for selecting from the created flows those whose flow length falls within the aforementioned reference range. The reference range of the flow length is set according to the movement state of the moving object, and is a self-position estimation program.

14. A self-position estimation program for estimating the self-position of a moving object (MC), The function includes acquiring image data output from a multi-camera (20) consisting of multiple cameras (21, 22, 23, 24) mounted on the aforementioned mobile body at set intervals, A function to extract feature points from the aforementioned image data, A function to store the positional information of each feature point in each of the aforementioned image data, A function that associates current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, and creates a flow, which is a set of corresponding feature points in the image data that differs for each time series. A function to select from the created flows that satisfy predetermined constraints, including epipolar constraints calculated based on the predicted motion of the moving body, as the correct flows, The computer is provided with a function to calculate the position and orientation of the moving object using the selected correct flow information. The self-localization program includes a function that, if the number of selected correct flows is less than a predetermined standard, repeats the selection of flows using modified constraints from those previously used until the number of correct flows equals or exceeds the standard.

Images