Information processing device, mobile device, information processing method, and computer program

Abstract

To provide an information processing device, mobile device, information processing method, and program capable of notifying of appropriateness of map information generated from sensor information.SOLUTION: An information processing device in a map information generation system comprises: a sensor information acquisition unit configured to acquire sensor information from one or more sensors of a mobile vehicle; a map generation unit configured to generate map information in surroundings of the sensors based on the sensor information; a map appropriateness estimation unit configured to estimate appropriateness of the map information based on at least either of the sensor information and the map information; and a map appropriateness notification unit for notifying of the appropriateness.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a mobile device, an information processing method, a computer program, etc. that generate map information. [Background technology]

[0002] Autonomous vehicles such as automated guided vehicles are used in factories and logistics warehouses. To estimate the position and orientation of such autonomous vehicles, cameras and LIDAR (Laser Imaging Detection and Ranging) sensors are sometimes used. A known method involves obtaining time-dependent position and orientation differences from the results of measuring the environment around the vehicle, and then calculating the position and orientation values ​​by comparing them with pre-generated map information.

[0003] In most cases, the map information used to estimate the position and orientation of this type of autonomous vehicle is manually generated in advance by the user using sensors such as cameras.One method proposed for generating highly accurate map information is to generate map information based on a closed path (Non-Patent Document 1).

[0004] Furthermore, the SUSAN (Smallest Univalue Segment Assimilating Nucleus) operator is known as a method for calculating feature points of an object in order to generate map information from image data (Non-Patent Document 2). Furthermore, a method has been proposed for guiding a moving object so as to reliably form a closed path (Patent Document 1). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] MA Raul, JMM Montiel and JD Tardos., “ORB-SLAM:A Versatile and Accurate Monocular SLAM System” Trans. Robotics vol. 31, 2015 [Non-patent document 2] SM Smith and JM Brady, ``SUSAN-a new approach to low level image processing,'' Int'l J Comput. Vision, vol.23,no.1, pp.45-78, 1997. [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-146952 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the method of Patent Document 1 does not provide a means for the user to know whether the map information is appropriate in terms of accuracy. SUMMARY OF THE INVENTION The present invention has been made in consideration of the above problems, and has as its object to provide an information processing device or the like that is capable of notifying the suitability of map information. [Means for solving the problem]

[0008] In order to solve the above problems, an information processing device according to one aspect of the present invention comprises: a sensor information acquisition means for acquiring sensor information from one or more sensors; a map information generating means for generating map information of the vicinity of the sensor based on the sensor information; a map suitability estimation means for estimating suitability of the map information based on at least one of the sensor information and the map information generated by the map information generation means; and a notification means for notifying the appropriateness.、 The map suitability estimation means estimates the suitability based on the sensor information and on whether or not a closed path is formed on the movement trajectory of the sensor. It is characterized by: [Effects of the Invention]

[0009] According to the present invention, it is possible to realize an information processing device or the like that is capable of notifying the suitability of map information. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a functional block diagram of a map information generating system using an information processing device according to a first embodiment of the present invention. [Figure 2] 10 is a flowchart showing a processing flow for estimating the suitability of map information in the map information generating system according to the first embodiment. [Figure 3] FIG. 4 is a diagram illustrating an example of a method for notifying the suitability of map information in the first embodiment. [Figure 4] 13 is a flowchart showing a processing flow for estimating the suitability of map information in the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present invention will be described by way of example with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified. [Example]

[0012] In the first embodiment, an example will be described in which the method of the present invention is applied to a map information generation system in which a user moves a mobile object, for example by remote control, and generates map information based on images taken by a camera mounted on the mobile object.

[0013] 1 is a functional block diagram of a map information generation system using an information processing device according to a first embodiment of the present invention. The map information generation system of this embodiment includes a mobile object 100 as a mobile device and an information processing device 102, and the mobile object 100 has the form of, for example, an AMR (autonomous mobile robot). The information processing device 102 may be mounted on an AMR (autonomous mobile robot) as a mobile body.

[0014] 1 are realized by causing a computer (not shown) included in an information processing device or the like to execute a computer program stored in a memory (not shown) as a storage medium. However, some or all of these may be realized by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs).

[0015] Furthermore, the individual functional blocks of the information processing device 102 shown in FIG. 1 do not have to be housed in the same housing, and the information processing device may be configured as separate devices connected to each other via signal paths. The information processing device 102 includes a built-in CPU as a computer, and the CPU controls the operation of each part of the entire device based on a computer program stored in a memory as a storage medium.

[0016] In this embodiment, a mobile object 100 equipped with a camera as a sensor collects map information to generate a 3D map, and estimates the suitability of the 3D map information. The map information here includes a 3D arrangement of feature points detected from an image captured by the camera, and the suitability of the 3D map information is estimated based on the detection reliability of the feature points.

[0017] This embodiment will be described with reference to FIGS. In FIG. 1, 100 is a moving body in the map information generating system, and 101 is an image sensor (camera sensor) such as a CMOS image sensor mounted on the moving body 100, which acquires two-dimensional array data of luminance as imaging data.

[0018] Reference numeral 102 denotes an information processing device in the map information generation system, and 103 denotes a sensor information acquisition unit that is part of the information processing device 102 and acquires sensor information from one or more sensors 101. In this embodiment, imaging data is acquired as the sensor information. Reference numeral 104 denotes a map generation unit that generates map information of the surrounding environment of the sensor 101 based on the sensor information acquired by the sensor information acquisition unit 103.

[0019] Reference numeral 105 denotes a map suitability estimation unit that estimates the suitability of map information based on the sensor information acquired by the sensor information acquisition unit 103 or the map information generated by the map generation unit 104. Reference numeral 106 denotes a map suitability notification unit that notifies the suitability estimated by the map suitability estimation unit 105. The information processing device 102 can be mounted on the mobile body 100. When the information processing device 102 is not mounted on the mobile body 100, the information processing device 102 controls the mobile body 100 via a communication network.

[0020] Next, Fig. 2 is a flowchart showing a processing flow for estimating the suitability of map information in the map information generation system of the first embodiment. Note that Fig. 2 shows an example in which processing is executed in parallel, but processing may also be executed in series. The operation of each step in Fig. 2 is performed by the computer inside the information processing device 102 executing a computer program stored in memory.

[0021] Step S200 is a step in which the sensor information acquisition unit 103 in FIG. 1 acquires sensor information from the sensor 101 and stores it in a storage unit (not shown). Step S201 is a step in which the map generating unit 104 in FIG. 1 generates map information based on the sensor information acquired in step S200.

[0022] The map information in this embodiment is composed of a group of three-dimensional position information of feature points of an object calculated from image data. The feature points are calculated using, for example, the SUSAN (Smallest Univalue Segment Assimilating Nucleus) operator (Non-Patent Document 2).

[0023] The method for calculating feature points is not limited to this, and any method that can calculate feature points in imaging data may be used. For example, feature points may be calculated based on three-dimensional feature amounts such as SHOT (Signature of Histograms of Orientations) feature amounts from multiple imaging data and feature points.

[0024] Step S202 is a step for saving the map information generated in step S201. Step S203 is a step in which the map suitability estimation unit 105 in FIG. 1 estimates the suitability of the map information based on the sensor information acquired by the sensor information acquisition unit 103 or the map information generated by the map generation unit 104. Step S204 is a step in which the map suitability notification unit 106 in FIG. 1 notifies the user of the suitability of the map information estimated in step S203.

[0025] The estimation of the suitability of the map information in step S203 is calculated by integrating the reliability of the feature points, which are the map information generated in step S201 in Fig. 2. The reliability R of each feature point is calculated by using the imaging data saved in step S200 to find the sum of the absolute values ​​of the change (difference) between the luminance at the target feature point position and the luminance around that feature point, and then taking the ratio of this sum to a predetermined threshold (R = sum of absolute values ​​of change ÷ threshold).

[0026] The suitability X of the map information calculated by integrating the reliability R of each feature point is the average value of the reliability R of all feature points. Therefore, the larger the value of the suitability X, the more appropriate the map information that has been generated. The suitability X described above may be the sum of the reliability R of all feature points, as long as the larger the value, the more appropriate the map information that has been generated. Furthermore, the feature points used to calculate the suitability X may be limited to a user-specified or system-specific region of interest (ROI).

[0027] The reliability R of each feature point described above uses the imaging data acquired and saved in step S200 in Fig. 2, but it is sufficient if it can be calculated from the luminance at the feature point position and the luminance around that feature point. Therefore, it is not necessary to save the imaging data in step S200, and the reliability R may be calculated after the imaging data is acquired.

[0028] Next, FIG. 3 is a diagram showing an example of a method for notifying the suitability of map information in the first embodiment, and shows an example in which the suitability of map information is notified using a GUI (Graphical User Interface) in step S204. Reference numeral 300 denotes an operation terminal operated by a user, and 301 denotes a display unit of the operation terminal 300. Reference numeral 302 denotes an imaging data display unit that is part of the display unit 301 and displays imaging data at the current time point.

[0029] Reference numeral 303 denotes a mark superimposed on the imaging data display section 302 to indicate the position of a feature point for which the reliability R estimated in step S203 is, for example, 1.0 or more. Reference numeral 304 denotes a mark superimposed on the imaging data display section 302 to indicate the position of a feature point for which the reliability R estimated in step S203 is less than 1.0.

[0030] In this embodiment, the marks 303 and 304 indicating the positions of the feature points are displayed in different colors depending on the reliability to notify the user. However, it is sufficient to display them in a form that allows the user to distinguish that the reliability of the feature points 303 and 304 is different, and for example, the marks may be displayed in different sizes or shapes. That is, the detection reliability may be notified by any one of the size, shape, and color of the feature point, or by a numerical value.

[0031] Therefore, it can be seen that the more marks 303 superimposed on the imaging data display section 302, the higher the appropriateness of the map information. 305 is a part of the display section 301, and is a map overhead section (a section that displays the entire feature point group from an arbitrary height, for example) that displays the map information generated in step S201 in Fig. 2.

[0032] Reference numeral 306 denotes a mark that indicates the overhead position corresponding to mark 303 and is superimposed on the map overhead section 305, and reference numeral 307 denotes a mark that indicates the overhead position corresponding to mark 304 and is superimposed on the map overhead section 305. In this way, the more marks that are superimposed on the map overhead section 305, as indicated by 306, the higher the appropriateness of the map information. Reference numeral 308 denotes a suitability display section that displays the suitability of the map information calculated in step S203 of FIG. 2 as a percentage.

[0033] Instead of displaying the suitability of the map information numerically, for example, a stimulus that can be perceived by the senses of hearing, touch, etc. may be generated to indicate the degree of suitability. Specifically, the degree of suitability may be displayed in a different size or color of predetermined characters, or in a different vibration or sound, depending on the degree of suitability. As described above, according to this embodiment, it is possible to estimate the suitability of map information using the detection reliability of feature points of an object, and to notify the user of the suitability of the map information being generated.

[0034] In the above embodiment, the suitability of the map information estimated in step S203 in Fig. 2 is estimated based on the detection reliability of the feature point calculated from the change (difference) in luminance at the feature point position and the luminance around the feature point. However, the detection reliability of the feature point may also be estimated based on the luminance of the imaging data.

[0035] That is, the reliability R of each feature point estimated in step S203 in Fig. 2 may be estimated by integrating the contrast of a single piece of imaging data. Specifically, the reliability R of each feature point is calculated using the minimum luminance IMin and maximum luminance IMax of each piece of imaging data, for example, based on the following Equations 1 and 2.

[0036] Contrast of a single image data = (IMax - IMin) ÷ (IMax + IMin) (Equation 1) R = AVG (contrast of a single image data) (Equation 2)

[0037] By estimating the detection reliability using such calculations, the closer the reliability R of a feature point is to 1.0 (stronger the contrast), the more reliable the feature point is, and it is possible to show the user that highly accurate map information has been generated. [Example]

[0038] In the first embodiment, the method of calculating and notifying the suitability of map information being generated based on the detection reliability of feature points of an object calculated from sensor information has been described. In the second embodiment, the distribution of feature points of an object calculated from sensor information is used to estimate the appropriateness of map information.

[0039] That is, similar to Example 1, a mobile body equipped with a camera as a sensor collects map information to generate a 3D map, and the suitability of the 3D map information is estimated. However, in Example 2, the suitability is calculated based on whether or not there is a bias in the distribution of feature points calculated from the sensor information. The functional block diagram and the processing flow for estimating the suitability of map information in the second embodiment may be the same as those shown in FIGS. 1 and 2 described in the first embodiment. The detailed processing of step S203, which is different from the first embodiment, will be described below.

[0040] In the second embodiment, the suitability X of the map information estimated in step S203 of Fig. 2 is estimated from the evenness De of the distribution of feature points present in the image data. The evenness De of the distribution of feature points is calculated based on whether the feature points are present evenly in the image data. The determination of whether or not there is a bias in the feature points is performed as follows: The screen is divided into a plurality of sections designated by the user or set in advance by the system.

[0041] Then, it is determined whether or not the number of feature points for each section is equal to or greater than a predetermined threshold. Specifically, the total number of sections with feature points equal to or greater than a predetermined threshold, Dm, is calculated using Equation 3, and the appropriateness X is calculated using Equation 4. Dm = COUNT (divisions with a number of feature points equal to or greater than a predetermined threshold) (Equation 3) X=Dm / (total number of sections)...(Formula 4)

[0042] The map information appropriateness X calculated by the above formula 4 is an estimate of the map information appropriateness based on the distribution of the object's feature points, and is the ratio of the feature point distribution to the total number of sections. Therefore, the closer the appropriateness X is to 1.0, the more appropriate the map information has been generated. Although the above-described partitions are two-dimensional based on the number of dimensions for calculating feature points, it is sufficient if the area can be divided, and therefore the partitions to be divided may be three-dimensional.

[0043] In this case, the degree of uniformity of feature point distribution De is calculated by determining whether there are a number of feature points equal to or greater than a predetermined threshold for each divided 3D section. Whether there is a bias in the feature points can be determined by using the above formulas 3 and 4, just as in the case of 2D sections. As described above, in the second embodiment, the suitability of map information can be estimated using the distribution of feature points of an object, thereby making it possible to notify the suitability of map information being generated.

[0044] In the second embodiment, the suitability of the map information estimated in step S203 is estimated based on whether or not there is a bias in the distribution of the feature points of the object. However, the suitability of the map information may also be estimated based on the number of pieces of imaging data containing the feature points used to generate the map information.

[0045] That is, the appropriateness X of the map information estimated in step S203 is calculated as follows. For example, for image data taken over a specified period specified by the user or preset by the system, the number of image data pieces having a number of feature points equal to or greater than a specified threshold is calculated, and the ratio of these numbers is calculated based on Equation 5 to estimate the image quality. X = number of image data with feature points equal to or greater than a predetermined threshold ÷ total number of image data (Equation 5)

[0046] That is, the number of pieces of image data having a number of feature points equal to or greater than a predetermined threshold value during a predetermined period is calculated, and this number is divided by the total number of pieces of image data during the predetermined period, thereby calculating the appropriateness X of the map information during the predetermined period. The closer the suitability X of the map information calculated in this way is to 1.0, the more appropriate the map information has been generated.

[0047] In the above example, the period for calculating the suitability X of the map information is the section from the start to the end of the map information generation system, but it may be any period between two different points in time. Therefore, the period may be divided into multiple periods, or the period may be dynamically set based on the subject distance, the length of time, the time of day, etc. Furthermore, since a larger value of the suitability X of the map information indicates that more appropriate map information has been generated, the number of pieces of image data per unit time (e.g., 1 second) that have feature points equal to or greater than a predetermined threshold may be displayed, for example, for each unit time.

[0048] Furthermore, although the example has been described in which the suitability X of map information is displayed as the ratio or number of pieces of image data having a number of feature points equal to or greater than a predetermined threshold relative to the total number of pieces of image data for a predetermined period, it is sufficient if the suitability can be calculated based on the feature points in the image data for a predetermined period. Therefore, instead of the ratio or number, the total number or average number of feature points equal to or greater than a predetermined reliability in the image data for a predetermined period may be displayed. In this case, too, it can be seen that the larger the value of the suitability X, the more appropriate map information has been generated. [Example]

[0049] In the first and second embodiments, a method has been described in which the appropriateness of map information being generated is calculated and notified based on the amount and distribution of feature points of an object calculated from sensor information. In the third embodiment, an example will be described in which the suitability of map information is estimated using environmental information about the surrounding environment of a sensor, which is map information currently being generated.

[0050] The same principle applies to a mobile vehicle equipped with a camera as a sensor, which collects map information, generates a 3D map, and estimates the appropriateness of that 3D map information. In the third embodiment, the suitability is calculated based on whether or not a change in the light source environment that reduces the accuracy of the position and orientation measurement of the moving object exists in the environment in which the moving object moves. The process flow for estimating the suitability of the configuration diagram and map information in the third embodiment may be the same as those shown in FIGS. 1 and 2 in the first and second embodiments, respectively.

[0051] The detailed processing of step S203, which is different from the first and second embodiments, will be described below. In the third embodiment, the suitability X of the map information estimated in step S203 in FIG. 2 is estimated based on whether or not there is an illumination variation (fluctuation in illuminance) as environmental information of the surrounding environment of the sensor 101 in FIG. The presence or absence of illumination fluctuation (fluctuation in illuminance), which is environmental information, is determined by whether or not the luminance of the captured image data changes over time.

[0052] Specifically, it is calculated based on the brightness variance V calculated from the average brightness group Y{Y0, Y1, ..., Yn} of all pixels of each image data in the entire image data group {D0, D1, ..., Dn}. If this brightness variance V is equal to or greater than a predetermined threshold, it is determined that illumination variation exists, and if it is less than the predetermined threshold, it is determined that illumination variation does not exist.

[0053] In the third embodiment, the suitability X of the map information estimated in step S203 in Fig. 2 is a binary value, where X = 1 when it is estimated that there is no illumination variation as environmental information, and X = 0 when it is estimated that there is illumination variation. Therefore, when the suitability X is 1, it indicates that appropriate map information has been generated.

[0054] Although the aforementioned suitability X is a binary value, it may also be possible to determine the extent of illumination variation. Therefore, the brightness variance V estimated in step S203 may be used as the suitability X (X = V) of the map information. In this case, the value of the suitability X indicates the strength of the brightness variance, and therefore, a smaller value of the suitability X suggests that more suitable map information has been generated.

[0055] Additionally, in the third embodiment, the determination of whether or not illumination variation exists is based on the luminance variance of the imaging data acquired from the sensor 101 in Fig. 1, but it is sufficient if it is possible to determine whether or not illumination variation exists. Therefore, it does not matter whether the moving object 100 in Fig. 1 is moving or stationary. Furthermore, since it is sufficient to be able to calculate the luminance variance from the average luminance of each imaging data, the luminance variance may be calculated from the average luminance of a specific region of the imaging data.

[0056] Furthermore, if there is illumination variation, there is a large difference between the average brightness values ​​of the image data acquired from the sensor 101 in Fig. 1 at two points in time. Therefore, if the difference in average brightness between the two points in time is equal to or greater than a predetermined threshold, it is determined that there is illumination variation, and if it is less than the predetermined threshold, it is determined that there is no illumination variation. The predetermined threshold is a value set by the user or a value preset by the system. As described above, in the third embodiment, the suitability of map information can be estimated using the surrounding environment, thereby making it possible to notify the suitability of map information being generated.

[0057] In the third embodiment, the suitability of the map information estimated in step S203 in Fig. 2 is calculated based on whether or not there is a change in illumination as environmental information about the surrounding environment of the sensor 101 in Fig. 1. However, the suitability X of the map information may be estimated based on whether or not there is information about a moving object in the map information generated in step S201 in Fig. 2 as environmental information about the surrounding environment of the sensor 101 in Fig. 1.

[0058] At this time, whether or not a moving object exists in the image data of the sensor 101 in Fig. 1 is determined by pattern matching using SSD (Sum of Squared Difference) between the image data and moving object template data. Then, based on the result of this pattern matching, the suitability X of the map information estimated in step S203 in Fig. 2 is estimated. Moving objects include, for example, devices with a moving mechanism and objects capable of autonomous movement, such as carts, conveyor belts, elevators, and people. They also include trees that sway due to external forces such as wind.

[0059] In this case, if the similarity calculated by pattern matching using SSD for all image data is less than a predetermined threshold, it is determined that no moving object exists, and if it is greater than or equal to the predetermined threshold, it is determined that a moving object exists. The predetermined threshold is a value set by the user or a value preset by the system. The suitability X of the map information estimated in step S203 in Fig. 2 is a binary value, where X = 1 when it is estimated that no moving object exists as environmental information, and X = 0 when it is estimated that a moving object exists. Therefore, when the suitability X is 1, it indicates that appropriate map information has been generated.

[0060] In the above description, whether or not information about a moving object is present in the map information generated in step S201 of FIG. 2, and whether or not a moving object is present in the image capture data of the sensor 101 in FIG. 1, is estimated by pattern matching using SSD. However, any method capable of calculating similarity may be used. Sum of Absolute Difference (SAD) may be used instead of SSD. In addition, an image recognition method using a trained model such as a Convolutional Neural Network (CNN) may be used as long as it is a method capable of determining whether or not a moving object is present.

[0061] Furthermore, although the suitability X of the map information is estimated as a binary value depending on whether or not a moving object is present, a large value of the suitability X is sufficient to indicate that appropriate map information has been generated, and therefore the suitability X of the map information may be displayed as multiple values. That is, for example, based on the number of detected moving objects N, the reciprocal (1 / N) of the number of detected moving objects or the negative value (-N) of the number of detected moving objects may be displayed as the suitability X.

[0062] The imaging data used for pattern matching can be obtained from any sensor that can collect the surrounding environment of sensor 101 in Figure 1, so a bird's-eye view sensor separate from sensor 101 mounted on mobile object 100 can be provided and used.

[0063] The suitability X of the map information may be estimated based on whether or not a repeated pattern exists as environmental information about the surrounding environment of the sensor 101 in FIG. That is, when the suitability X of the map information is estimated in step S203 of FIG. 2, it may be determined whether or not a repeated pattern exists in the image data by integrating the results of frequency analysis of each image data.

[0064] Specifically, an amplitude spectrum is calculated by performing a discrete Fourier transform on each piece of image data, and a determination is made as to whether or not there are frequency components with amplitudes equal to or greater than a predetermined threshold within a predetermined frequency range excluding the DC component (0 Hz). If there are frequency components with amplitudes equal to or greater than the predetermined threshold, it is determined that a repeating pattern exists, and if there are no frequency components equal to or greater than the predetermined threshold, it is determined that a repeating pattern does not exist.

[0065] When estimating the suitability X of the map information by integrating the results of frequency analysis of each piece of image data, the suitability X can be calculated as the reciprocal of the number of pieces of image data containing a repeating pattern. As a result, the more pieces of image data containing a repeating pattern there are, the higher the suitability X will be, indicating that appropriate map information has been generated. In this way, the environmental information may include any one of the following: illumination variation, the presence or absence of a moving object, and the presence or absence of a repeating pattern. [Example]

[0066] In the first to third embodiments, a method for notifying the suitability of map information being generated by estimating the feature points of an object and the surrounding environment from sensor information and map information being generated has been described. In the fourth embodiment, the suitability is estimated based on sensor information, based on whether or not a closed path is formed on the movement trajectory of the moving body or the sensor mounted on the moving body. The processing flow for estimating the suitability of the configuration diagram and map information in the fourth embodiment may be the same as that shown in Fig. 1 and Fig. 2 used in the first to third embodiments. The detailed processing of step S203, which is different from the first to third embodiments, will be described below.

[0067] In the fourth embodiment, the suitability X of the map information estimated in step S203 in Fig. 2 is determined based on whether the trajectory of the moving object 100 is a closed path, based on the position and orientation information of the sensor 101 acquired by the sensor information acquisition unit 103 in Fig. 1. Whether it is a closed path is determined based on whether the difference amount of the position and orientation information at the start and end times of the movement is less than a predetermined threshold. If it is less than the predetermined threshold, it is determined to be a closed path, and if it is equal to or greater than the predetermined threshold, it is determined to not be a closed path.

[0068] The suitability X of the map information estimated in step S203 of FIG. 2 is a binary value. If it is determined in step S203 that the route is a closed route, this indicates that the map information has been generated appropriately. The closed path section mentioned above is a section at the start and end times, but it is sufficient to estimate the closed path status throughout the entire section. Therefore, the section may be divided and integrated for judgment. Specifically, the difference in position and orientation information of the sensor between two points in time is used.

[0069] The position and orientation information of the sensor at each time point is Pi, and the position and orientation information of the sensor at the current time is Ps. The position and orientation information Pt of the sensor that is closest to Ps is searched for from Ps and Pi at a time point in the past that is equal to or greater than a predetermined threshold time. Then, the differential distance D (D = ABS(Pt - Ps)) of the position and orientation information of the sensor is calculated. The search for the position and orientation information Pt of the sensor is performed by calculating the difference between Ps and each Pi, and the Pi with the smallest absolute value of the difference is selected. The predetermined threshold is a value specified by the user or set in advance by the system. Furthermore, although time is used as the basis for the threshold for selecting Pi, the amount of movement may also be used as the basis in relation to the moving speed of the moving object.

[0070] When dividing a section, the appropriateness X of the map information estimated in step S203 in Fig. 2 becomes multi-valued, and X = D. In this case, the closer the appropriateness X is to 0.0, the more appropriate the map information that has been generated.

[0071] As explained above, it is possible to determine whether a closed path is formed using the similarity calculated based on the position and orientation information of the sensor at two points in time, and to estimate the appropriateness of the map information accordingly, thereby notifying the appropriateness of the map information being generated.

[0072] In the fourth embodiment, when generating map information in step S201 of FIG. 2, a closed path may be detected, and the accuracy of the map information may be improved (Non-Patent Document 1) at that time, and the appropriateness X of the map information may be estimated based on the amount of correction of each piece of position and orientation information in the improvement of accuracy.

[0073] In this case, the suitability X of the map information estimated in step S203 in Fig. 2 is calculated based on the amount of correction to be made at each point in time for improving accuracy. Specifically, the correction method for improving accuracy at each point in time uses the method in Non-Patent Document 1, but the correction amount Ci at each point in time is set to the difference between before and after correction, and the correction average value Cm is set to the average value of Ci.

[0074] The suitability X of the map information estimated in step S203 in Fig. 2 is multi-valued, and is set to X = Cm. Therefore, in this case, the closer the suitability X is to 0.0, the more suitable the map information that has been generated. In this case, for example, the appropriateness X may be calculated as the sum of the correction amounts (X=SUM(Ci)). In addition, the appropriateness X may be calculated based on the total distance of the route to be corrected, rather than the correction amount at each time point.

[0075] In the fourth embodiment, the suitability of the map information may be determined based on the degree (proportion) of closed route sections to the entire route by determining closed route sections based on the similarity of brightness between the captured image data. In this case, the suitability X of the map information estimated in step S203 in FIG. 2 is determined based on whether the trajectory of the moving object 100 in FIG. 1 is a closed path.

[0076] The determination of whether a path is closed is estimated based on whether there is a significant difference between the current image data and the previous image data. However, the previous image data must be greater than a threshold specified by the user or preset by the system. The significant difference is calculated using a two-sample T-test on the brightness of the image data.

[0077] Specifically, in the case of a T-test of two paired samples with a significance level of 1%, the reference value P is 2.576, and if the absolute value of the calculated T-value is less than the reference value P, it is estimated that there is no significant difference between the image data taken at the two points in time. Therefore, it is assumed that the trajectory of the moving object 100 is a closed path, and if it is equal to or greater than the reference value P, there is a significant difference between the image data taken at the two points in time, and therefore the trajectory of the moving object 100 is not a closed path. This reference value P is specified from a value set by the user or a significance level according to a normal distribution.

[0078] The suitability X of the map information estimated in step S203 in Fig. 2 is a binary value, where X = 1 when the trajectory of the moving object 100 in Fig. 1 is estimated to be a closed path, and X = 0 when it is estimated not to be a closed path. Therefore, when the suitability X is 1, it indicates that appropriate map information has been generated.

[0079] The calculation of the significant difference described above also includes a form in which a method such as SSD or SAD for calculating the similarity of imaging data in Example 3 is used. In that case, if the similarity is less than a predetermined threshold, it is determined that there is no significant difference between the imaging data at two time points, and if the similarity is equal to or greater than the predetermined threshold, it is determined that there is a significant difference between the imaging data at two time points.

[0080] This embodiment also includes a form in which affine transformation is used based on the difference in the angle of view of each piece of imaging data or the orientation of the moving object 100 in Fig. 1 before determining whether or not there is a significant difference between the two pieces of imaging data. Note that any method other than affine transformation may be used as long as it is a method for correcting the geometric difference between the two pieces of imaging data. Furthermore, not only two-dimensional transformation but also three-dimensional transformation may be used. [Example]

[0081] In the first to fourth embodiments, it has been shown that it is possible to notify the suitability of the map information being generated based on the sensor information, the surrounding environment of the map information being generated, and whether a closed route is formed. In the fifth embodiment, two types of sensors are provided: a camera sensor for generating map information and a movement amount sensor for measuring the movement amount of a moving object. The suitability of the map information is estimated based on the difference between the movement amount of the camera sensor (first movement amount) estimated based on the map information being generated and the movement amount of the moving object in real space (second movement amount) estimated from the sensor information acquired by the movement amount sensor.

[0082] Fig. 4 is a flowchart showing a processing flow for estimating the suitability of map information in Example 5. Note that the operation of each step in Fig. 4 is performed by the internal computer of the information processing device 102 executing a computer program stored in the memory.

[0083] A mobile vehicle equipped with a camera as a sensor collects map information to generate a 3D map, and estimates the adequacy of the 3D map information. The map information here is the 3D arrangement of feature points detected by the image sensor. The sensors include the camera sensor for generating map information, as well as a movement amount sensor for measuring the movement amount of the mobile body by measuring the rotation amount of a movement motor for moving the mobile body.The appropriateness of the map information being generated is estimated based on the difference between the movement amount of the camera sensor estimated from the map information and the movement amount of the mobile body calculated from the rotation amount of the movement motor provided on the mobile body.

[0084] A detailed processing flow of step S203 in FIG. 2 in the fifth embodiment will be described with reference to the flowchart in FIG. In step S400, it is determined whether or not the camera continues to acquire image data. If the answer is Yes in step S400, the process proceeds to step S401. If the answer is No, the estimation of the appropriateness using the difference in the amount of movement is terminated.

[0085] In step S401, the movement amount of the camera sensor is estimated from the map information being generated. That is, the movement amount MV of the sensor 101 in the virtual space is estimated based on the map information generated by the map generation unit 104 using geometric transformation. The amount of movement of the moving object 100 and the sensor 101 in FIG. 1 is estimated while the sensor information acquisition unit 103 is acquiring sensor information.

[0086] Furthermore, in step S402, the amount of movement MR of the moving body 100 in real space is estimated based on the amount of rotation of the movement motor provided in the moving body 100. The movement amount MR of the moving body 100 in real space may be calculated by any method that can calculate the movement distance of the moving body 100 between predetermined points in time, for example, a method using a GPS (Global Positioning System) may be used. Furthermore, a method that calculates the movement amount from the movement of the moving body may also be used.

[0087] For example, a fixed camera with a bird's-eye view equipped with a rotation mechanism may be provided, and the movement amount MR may be calculated based on the amount of rotation of a motor when the fixed camera tracks the moving object. Alternatively, a 6-axis acceleration sensor or the like may be provided on the moving object, and the movement amount MR may be calculated based on the output of the 6-axis acceleration sensor. Next, in step S403, the difference between the movement amount MV and the movement amount MR is calculated based on the movement amount MV of the sensor estimated in step S401 and the movement amount MR of the moving body estimated in step S402.

[0088] Specifically, the difference D between the movement amounts MV and MR is calculated by subtracting the movement amounts MV and MR after converting them into millimeter units in real space. It is not necessary to convert them into millimeter units in real space before subtraction; instead, they can be converted into a common unit system in a common space before subtraction. The common space may be, for example, a path on a two-dimensional plane or a three-dimensional space. Furthermore, a space or unit system uniquely defined by the system may be used.

[0089] Next, in step S404, the map suitability estimation unit 105 in FIG. 1 estimates the suitability of the map information based on the difference D in the amount of movement calculated in step S403, and the process returns to step S400. The suitability X of the map information estimated in step S404 in FIG. 4 is multivalued, and X = ABS (difference D in the amount of movement). Therefore, the closer the suitability X is to 0.0, the higher the reliability, indicating that appropriate map information has been generated. Alternatively, if the difference D in the amount of movement is equal to or less than a predetermined value, it may be determined to be appropriate, and if it is greater than the predetermined value, it may be determined to be inappropriate, and this may be notified.

[0090] As described above, in the fifth embodiment, the suitability of map information is estimated based on the difference between the amount of movement of a moving object estimated from map information generated based on the output of a camera sensor and the amount of movement of a moving object estimated from the output of a movement amount sensor, thereby making it possible to notify the suitability of map information being generated.

[0091] As described above, in the fifth embodiment, the suitability of the map information estimated in step S203 of FIG. 2 is calculated based on the difference in the amount of movement at the same time between the sensor information acquired from two types of sensors. However, the suitability X of the map information may be estimated based on the difference between the amount of movement of the sensor estimated from the map information being created and the amount of movement of the moving object calculated from the image data acquired from the camera sensor.

[0092] That is, the movement amount MR of the moving object calculated in step S402 of Fig. 4 may be calculated from the optical flow of the image data, for example, using the Lucaskanade method. In this method, the motion vector of each pixel is calculated, so the average vector of all the motion vectors is used as the movement amount MR. That is, the second movement amount is estimated from the motion vector obtained from the sensor information.

[0093] The difference D in the amount of movement calculated in step S403 in Fig. 4 is the difference between two types of amount of movement (D = MV - MR) as in this embodiment, and the suitability X of the map information estimated in step S404 is multi-valued, with X = ABS (difference D in amount of movement). Therefore, the closer the suitability X is to 0.0, the more appropriate the map information that has been generated. As mentioned above, if the difference D in amount of movement is equal to or less than a predetermined value, it may be determined to be appropriate, and if it is greater than the predetermined value, it may be determined to be inappropriate, and this may be notified.

[0094] In the above, the amount of movement of the moving object is calculated from the optical flow using the Lucaskanade method, but any method may be used as long as it can calculate the amount of movement of each pixel from the captured image data. In the above first to fifth embodiments, the sensor 101 in FIG. 1 is mounted on the moving body 100, but it may be arranged in a device separate from the moving body 100 and configured to communicate with the information processing device .

[0095] Furthermore, the sensor 101 may be, for example, a sensor such as a LIDAR, and it is sufficient if the sensor allows the map generating unit 104 to generate map information. The suitability estimated by the map suitability estimation unit 105 in FIG. 1 may be calculated by distinguishing between the suitability of the entire area of ​​the map information generated by the map generation unit 104 and the suitability of each local area.

[0096] As described above, in the first embodiment, the suitability of map information is estimated using the detection reliability of feature points of an object, thereby making it possible to notify the suitability of map information being generated. In the second embodiment, the suitability of map information is estimated using the distribution of feature points of an object, thereby making it possible to notify the suitability of map information being generated. In the third embodiment, the suitability of map information is estimated using the surrounding environment, thereby making it possible to notify the suitability of map information being generated.

[0097] In the fourth embodiment, the suitability of map information being generated can be notified by estimating the suitability of map information using a similarity calculated based on the position and orientation information of the sensor at two points in time. In the fifth embodiment, the suitability of map information being generated can be notified by estimating the suitability of map information using a difference between the amount of movement of the sensor moving body estimated from the map information and the amount of movement of the moving body estimated from the position and orientation information of the sensor. Furthermore, the suitability of map information may be estimated by appropriately combining the estimation methods of the above-mentioned Examples 1 to 5. This allows the suitability to be estimated more precisely and in more detail.

[0098] 1 has a form of an AMR (autonomous mobile robot) and includes a driving device such as a travel motor or engine for moving (traveling) the AMR, a movement direction control device for changing the movement direction of the AMR, and a movement control unit for controlling the drive amount of the driving device and the movement direction of the movement direction control device.

[0099] The movement control unit has a built-in CPU as a computer and memory that stores computer programs, and communicates with other devices, thereby controlling, for example, the information processing device 102 and acquiring map information, position and orientation information, driving route information, etc. from the information processing device 102. The AMR as the moving body 100 is configured so that the movement direction, movement amount, and movement route of the AMR are controlled by a movement control unit based on map information generated by the information processing device 102 and the travel route searched.

[0100] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0101] 100 Mobile 101 Sensors 102 Information processing equipment 103 Sensor information acquisition unit 104 Map Generation Unit 105 Map suitability estimation unit 106 Map Adequacy Notification Unit

Claims

1. a sensor information acquisition means for acquiring sensor information from one or more sensors; a map information generating means for generating map information of the vicinity of the sensor based on the sensor information; a map suitability estimation means for estimating suitability of the map information based on at least one of the sensor information and the map information generated by the map information generation means; a notification means for notifying the appropriateness, The information processing apparatus is characterized in that the map suitability estimation means estimates the suitability based on the sensor information, based on whether or not a closed path is formed on the movement trajectory of the sensor.

2. 2. The information processing apparatus according to claim 1, wherein the map suitability estimation means calculates detection reliability of feature points of objects included in the map information, and estimates the suitability based on the detection reliability.

3. The information processing apparatus according to claim 2 , wherein the detection reliability of the feature point is calculated based on a difference between a luminance at the position of the feature point and a luminance around the feature point.

4. 4. The information processing apparatus according to claim 2, wherein the notification means notifies the detection reliability by one of the size, shape, and color of the feature point, or by a numerical value.

5. 5. The information processing apparatus according to claim 1, wherein the map suitability estimation means estimates the suitability based on a distribution of feature points of an object calculated from the sensor information.

6. 6. The information processing apparatus according to claim 1, wherein the map suitability estimation means estimates environmental information based on the map information, and estimates the suitability based on the environmental information.

7. 7. The information processing apparatus according to claim 6, wherein the environmental information includes one of illumination fluctuation, presence or absence of a moving object, and presence or absence of a repeating pattern.

8. the sensor information acquisition means acquires position and orientation information of the sensor; The information processing device according to any one of claims 1 to 7, characterized in that the map suitability estimation means calculates a similarity between the position and orientation information at two points in time acquired by the sensor information acquisition means, and estimates the suitability based on the similarity.

9. The information processing device according to any one of claims 1 to 8, characterized in that the map suitability estimation means estimates the suitability based on a difference between a first movement amount of the sensor estimated from the map information and a second movement amount of the sensor estimated from the sensor information.

10. 10. The information processing apparatus according to claim 9, wherein the second movement amount is estimated from a motion vector obtained from the sensor information.

11. A sensor information acquisition means for acquiring sensor information from one or more sensors; a map information generating means for generating map information of the vicinity of the sensor based on the sensor information; a map suitability estimation means for estimating suitability of the map information based on at least one of the sensor information and the map information generated by the map information generation means; a notification means for notifying the appropriateness, the sensor information acquisition means acquires position and orientation information of the sensor; The information processing apparatus is characterized in that the map suitability estimation means calculates a similarity between the position and orientation information at two points in time acquired by the sensor information acquisition means, and estimates the suitability based on the similarity.

12. A sensor information acquisition means for acquiring sensor information from one or more sensors; a map information generating means for generating map information of the vicinity of the sensor based on the sensor information; a map suitability estimation means for estimating suitability of the map information based on at least one of the sensor information and the map information generated by the map information generation means; a notification means for notifying the appropriateness, The information processing device is characterized in that the map suitability estimation means estimates the suitability based on a difference between a first movement amount of the sensor estimated from the map information and a second movement amount of the sensor estimated from the sensor information.

13. An information processing device according to any one of claims 1 to 12; a movement control means for controlling movement based on the map information; A mobile device having:

14. a sensor information acquisition step of acquiring sensor information from one or more sensors; a map generation step of generating map information of the vicinity of the sensor based on the sensor information; a map suitability estimation step of estimating suitability of the map information based on at least one of the sensor information and the map information; a notification step of notifying the appropriateness, The information processing method is characterized in that the map suitability estimation step estimates the suitability based on the sensor information and on whether or not a closed path is formed on a movement trajectory of the sensor.

15. A computer program for controlling each means of the information processing device according to any one of claims 1 to 12 or the mobile device according to claim 13 by a computer.

Images