Information processing apparatus, movable body, information processing method, computer program, and storage medium

Abstract

To provide an information processing apparatus capable of creating a high-quality map with reduced measurement omissions.SOLUTION: In an information processing apparatus, sensor information including shape information is acquired from a three-dimensional sensor mounted on a movable body, position and orientation information of the three-dimensional sensor when the three-dimensional sensor has measured the shape information is acquired, map information for creating a shape map is created based on the shape information and the position and orientation information, a portion of a movable object is recognized based on the sensor information, and a measurement omission region indicating a measurement omission in the shape map is set based on the sensor information, the position and orientation information, and the portion.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a mobile object, an information processing method, a computer program, and a storage medium. [Background technology]

[0002] There is a technology that creates a shape map (hereafter simply referred to as a map) that shows the shape of the entire surrounding environment based on shape information measured from multiple points by moving and rotating a 3D sensor. Generally, it is desirable that this map be composed only of static objects that do not physically move.

[0003] In Patent Document 1, dynamic objects such as people and cars are recognized in camera images and their corresponding shape information is removed, thereby making it possible to create a map excluding the dynamic objects. A method for creating a map showing occupied and unoccupied areas created in this way is described, for example, in Non-Patent Document 1, and a method for recognizing people based on machine learning is described, for example, in Non-Patent Document 2. A method for estimating the position and orientation of a sensor based on a range image and a method for creating a three-dimensional map are described, for example, in Non-Patent Document 3. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-87407 [Non-patent literature]

[0005] [Non-Patent Document 1] Sebastian Thrun, Arno Bucken, “Integrating grid-based and topological maps for mobile robot navigation,” Proceedings of the Thirteenth National Conference on Artificial Intelligence, 199. [Non-patent document 2] Jamie Shotton,et al,”Real-time Human Pose Recognition in Parts from Single Depth Images”,CVPR201 [Non-patent document 3] R. Newcombe,et al,”Kinect Fusion:Real-Time Dense Surface Mapping and Tracking”,EEE International Symposium Mixed and Augmented Reality(ISMAR),201 Summary of the Invention [Problem to be solved by the invention]

[0006] However, if a dynamic object appears when measuring the environment by moving and rotating a 3D sensor, not only will the dynamic object be intruded into the map, but the area hidden behind the object will not be measurable, resulting in measurement omissions in the map and reduced quality. Patent Document 1 and Non-Patent Documents 1-3 do not recognize this problem.

[0007] The present invention has been made in consideration of the above problems, and has as its object to provide an information processing device capable of creating a high-quality map with reduced measurement omissions. [Means for solving the problem]

[0008] One aspect of the present invention is an information processing device, a sensor information acquisition means for acquiring sensor information including shape information from a three-dimensional sensor mounted on the moving object; a position and orientation information acquisition means for acquiring position and orientation information of the three-dimensional sensor when the three-dimensional sensor measures the shape information; a map information creating means for creating a shape map based on the shape information and the position and orientation information; an object recognition means for recognizing a part of a dynamic object based on the sensor information; The method is characterized by comprising a setting unit that sets a measurement omission area indicating a measurement omission in the shape map based on the sensor information, the position and orientation information, and the part. [Effects of the Invention]

[0009] According to the present invention, it is possible to realize an information processing device capable of creating a high-quality map with reduced measurement omissions. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a scene setting of a use case in the first embodiment. [Figure 2] 1 is a diagram illustrating a configuration of a mobile robot according to a first embodiment. [Figure 3] 2 is a functional block diagram showing the configuration of the inside of an information processing device 500 and related modules according to the first embodiment. FIG. [Figure 4] 4 is a flowchart showing a processing procedure in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a map in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating the positional relationship between a three-dimensional sensor and a dynamic object in the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of displaying an unmeasured area on a map in the first embodiment. [Figure 8] FIG. 10 is a diagram showing another example of displaying an unmeasured region on a map in the first embodiment. [Figure 9] 10A and 10B are diagrams illustrating an example in which a measurement omitted region is remeasured in the second embodiment. [Figure 10] FIG. 10 is a diagram showing an example in which a re-measured portion of a measurement omission region is deleted in the second embodiment. [Figure 11] FIG. 11 is a functional block diagram showing the configuration of the inside of an information processing device 501 and related modules according to a third embodiment. [Figure 12] 10 is a flowchart showing a processing procedure in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicate descriptions will be omitted or simplified.

[0012] First Embodiment In the first embodiment, a case will be described in which a mobile robot such as an AGV (Automatic Guided Vehicle) or an AMR (Autonomous Mobile Robot) equipped with a three-dimensional sensor is used. The mobile robot in this embodiment uses SLAM (Simultaneous Localization and Mapping) technology, which measures the surrounding environment by moving and rotating and creates a map based on shape information obtained from multiple points.

[0013] Figure 1 is a diagram showing a scene setting for a use case in the first embodiment. In Figure 1, a mobile robot 200 as a moving body is equipped with a 3D sensor 100, and a user 300 controls its movement using a controller 400. A display device 410 displays a shape map created based on shape information measured along the route up to the current point in time.

[0014] The user 300 refers to the map displayed on the display device 410 and controls the mobile robot 200 to add more measurement points to create the desired map. In this embodiment, when a dynamic object is captured, the part of the shape information acquired by the 3D sensor that corresponds to the dynamic object is recognized, and the area behind the object that cannot be measured by the 3D sensor (hereinafter referred to as the measurement omission area) is displayed on the display device 410.

[0015] The user 300 checks the information displayed here, and if he finds an area that has not been measured, he performs processing such as excluding dynamic objects. After that, he moves and rotates the mobile robot 200 so that the 3D sensor 100 can measure this area and measures it again, thereby creating a high-quality map with no measurement omissions.

[0016] This embodiment will be described in detail below. First, the module configuration of this embodiment will be described. FIG. 2 is a diagram showing the configuration of a mobile robot in the first embodiment, in which 200 represents the mobile robot, 100 represents a three-dimensional sensor that acquires shape information, 210 represents a control unit that drives and controls the wheels of the mobile robot 200, and 500 represents an information processing device in this embodiment.

[0017] The information processing device 500 has a built-in CPU as a computer, which functions as a control means for controlling the operation of each part of the entire mobile robot 200 based on a computer program stored in a memory as a storage medium. Also, the control unit 210 controls the movement or rotation (steering) of the mobile body based on a shape map formed by the information processing device 500.

[0018] Fig. 3 is a functional block diagram showing the configuration of the inside of an information processing device 500 and related modules in the first embodiment. Note that some of the functional blocks shown in Fig. 3 are realized by causing a computer (not shown) included in the information processing device 500 to execute a computer program stored in a memory (not shown) serving as a storage medium. However, some or all of these may be realized by hardware. Examples of hardware that can be used include a dedicated circuit (ASIC) and a processor (reconfigurable processor, DSP).

[0019] Furthermore, the functional blocks shown in FIG. 3 do not have to be housed in the same housing, and may be configured as separate devices connected to each other via signal paths. The above description regarding FIG. 3 also applies to FIG. In FIG. 3, 510 denotes a sensor information acquisition unit, 520 denotes a position and orientation information acquisition unit, 530 denotes a map information creation unit, 540 denotes an object recognition unit, 550 denotes a setting unit, and 560 denotes a display control unit.

[0020] The three-dimensional sensor (3D sensor) 100 is a sensor that acquires shape information of an object. In this embodiment, it is a depth sensor that captures a distance image of the object, and is configured by, for example, a stereo camera.

[0021] Mobile robot 200 is a robot that moves, for example, on a horizontal plane, and drives motors and the like based on control signals from control unit 210 to move and rotate (steer) within the horizontal plane. The relative position and orientation of mobile robot 200 and 3D sensor 100 are set in advance to have a predetermined relationship. Therefore, the position and orientation of the 3D sensor provided in the mobile robot can be calculated based on control signals from control unit 210.

[0022] The display device 410 is a monitor that outputs images. The sensor information acquisition unit 510 acquires sensor information including shape information from the 3D sensor 100 mounted on the moving object. The position and orientation information acquisition unit 520 acquires, from the position and orientation information estimation unit 230, the position and orientation information of the 3D sensor when the 3D sensor 100 measured the shape information.

[0023] The position and orientation information estimation unit 230 is for estimating the position and orientation of the three-dimensional sensor 100, and estimates the position and orientation of the three-dimensional sensor 100 based on, for example, a control signal for driving control of a moving body, a gyro sensor, etc. The map information creation unit 530 creates a shape map of the surroundings based on the shape information acquired by the sensor information acquisition unit 510 and the position and orientation information acquired by the position and orientation information acquisition unit 520.

[0024] The object recognition unit 540 recognizes a portion corresponding to a dynamic object from the shape information acquired by the 3D sensor 100. The setting unit 550 sets a measurement omission region indicating a measurement omission in the shape map based on the sensor information including the shape information acquired by the sensor information acquisition unit 510, the position and orientation information acquired by the position and orientation information acquisition unit 520, and the portion recognized by the object recognition unit 540. The display control unit 560 generates display information that distinguishes between the map created by the map information creation unit 530 and the measurement omission region set by the setting unit 550.

[0025] Next, a specific processing procedure of this embodiment will be described. Fig. 4 is a flowchart showing the processing procedure in the first embodiment. Note that the computer in the information processing device 500 executes a computer program stored in memory to perform the operation of each step in the flowchart of Fig. 4. The flow of Fig. 4 starts, for example, by turning on the power of the mobile robot 200 and setting it to map creation mode.

[0026] In step S600, the information processing device 500 performs initialization for image acquisition and calculation. Specifically, the information processing device 500 performs processes such as starting up programs and sensors, and acquiring and loading parameters required for processing from a storage unit (not shown) included in the information processing device 500.

[0027] In step S610 (sensor information acquisition step), information processing device 500 acquires sensor information (shape information) from 3D sensor 100 using sensor information acquisition unit 510. Specifically, a distance image is acquired. The distance image acquired at each time point i is defined as Di (i = 0 to t). Here, i = 0 represents the time of initialization, and t represents the current time. Also, the 3D coordinates indicated by each pixel j included in Di are defined as Rij. Note that Rij is a 3D coordinate with 3D sensor 100 as the origin.

[0028] In step S620 (position and orientation information acquisition step), information processing device 500 acquires position and orientation information of 3D sensor 100 from position and orientation information estimation unit 230 using position and orientation information acquisition unit 520. In this embodiment, the position and orientation of 3D sensor 100 is calculated based on a control signal from control unit 210. The position and orientation acquired at each time point i is designated as Pi. Note that control unit 210 is a module that controls the wheels of mobile robot 200 based on a control signal including the amount of movement and (horizontal) rotation of mobile robot 200.

[0029] The position and orientation information estimation unit 230 estimates the position and orientation Qi of the mobile robot 200 based on information on the amount of movement and amount of rotation of the mobile robot 200. Then, Qi is coordinate-transformed based on the relative position and orientation between the mobile robot 200 and the 3D sensor 100, which has been calculated in advance through calibration, to estimate the position and orientation Pi of the 3D sensor.

[0030] Here, the position and orientation Pi includes three-dimensional position parameters and three-dimensional orientation parameters, and is expressed in a coordinate system O centered on the position of the mobile robot 200 at the time of initialization. The XY plane of the coordinate system O is defined as the plane of the map to be output, and the X and Y coordinates of the coordinate system O represent two-dimensional coordinates on the map. For example, the X and Y coordinate components of the position and orientation Pi represent two-dimensional coordinates on the map. The two-dimensional coordinates of the position and orientation Pi on the map are defined as Qi.

[0031] In step S630 (map information creation step), the information processing device 500 creates a shape map of the environment using the map information creation unit 530 based on the shape information acquired by the sensor information acquisition unit 510 and the position and orientation acquired by the position and orientation information acquisition unit 520. FIG. 5 is a diagram showing an example of a map in the first embodiment. In this embodiment, the map information is a two-dimensional map of a room in which a mobile robot moves, as seen from above, as shown in 700 in FIG.

[0032] In map 700, black areas represent occupied areas where objects exist, and white areas represent unoccupied areas where no objects exist. Here, occupied / unoccupied can be determined using distance image Di. Specifically, based on the three-dimensional coordinates Rij of each pixel j included in distance image Di, it can be determined that the area from the center of three-dimensional sensor 100 to those coordinates is an unoccupied area where no objects exist, and the area beyond those coordinates is an occupied area where objects exist.

[0033] To reflect this determination result on a map, first, the 3D coordinates Rij of each pixel j contained in the range image Di are projected onto the map using the position and orientation Pi acquired at each time point i. The 2D coordinates projected onto the map are called SiJ. The area on the map from the 2D coordinates Qi to Sij of the 3D sensor can then be determined as an unoccupied area where no objects are present, and the area beyond Sij can be determined as an occupied area where objects are present.

[0034] By calculating this for all distance images Di (i = 0 to t), it is possible to determine whether an area is occupied or unoccupied for the entire range measured by moving and rotating the mobile robot 200. By representing these occupied and unoccupied areas in two colors, black and white, a map 700 such as that shown in Figure 5 can be created.

[0035] Such a map 700 showing occupancy and non-occupancy is called an Occupancy Map. The method for creating an Occupancy Map is well known, and it may be created using the method described in the aforementioned Non-Patent Document 1, for example.

[0036] In step S640 (object recognition step), information processing device 500 uses object recognition unit 540 to recognize a portion corresponding to a dynamic object from the shape information acquired by 3D sensor 100. In this embodiment, a dynamic object is defined as, for example, a person, and is recognized based on a distance image acquired by sensor information acquisition unit 510. A method based on machine learning, for example, can be used to recognize a person from a distance image.

[0037] Specifically, by inputting various distance images showing people and learning the areas where people exist, it is possible to prepare a program that recognizes similar areas in a given distance image Di. Methods for recognizing people based on machine learning are well known, and the method described in the aforementioned Non-Patent Document 2 may be used, for example. Note that the three-dimensional coordinates of a pixel included in the area recognized in the distance image are designated Tik, where k represents the pixel number.

[0038] In step S650 (setting step), the information processing device 500 has a setting unit 550 that sets a measurement omission area based on the shape information acquired by the sensor information acquisition unit 510, the position and orientation acquired by the position and orientation information acquisition unit 520, and the part recognized by the object recognition unit 540.

[0039] Specifically, first, when a dynamic object is recognized by the object recognition unit 540 at time i, the three-dimensional coordinates Tik included in the recognized part are projected onto the map. The projected two-dimensional coordinates are designated as Uik. FIG. 6 is a diagram showing the positional relationship between the three-dimensional sensor and the dynamic object in the first embodiment. Here, as shown in FIG. 6, the area behind Uik on the line from the three-dimensional sensor position Qi (810) toward Uik (point cloud 820) is designated as the measurement omission area. That is, based on the position and orientation of the three-dimensional sensor, the area behind (behind) the recognized part in the direction toward it is set as the measurement omission area.

[0040] In this embodiment, a line segment within a predetermined length L from the position of Uik on a straight line is defined as Vik (830), and an area (840) within a predetermined range M from the line segment Vik (830) is defined as a measurement omission area. The area defined at each time i is defined as Ai. Area Ai is stored in a storage unit (not shown).

[0041] In step S660, the information processing device 500 generates, via the display control unit 560, display information that distinguishes between the map created by the map information creation unit 530 and the measurement omission area information indicating measurement omissions set by the setting unit 550, and displays the information on the display device.

[0042] FIG. 7 is a diagram showing an example of displaying an unmeasured area on a map in the first embodiment. In this embodiment, as shown in FIG. 7, measurement omission areas are displayed with different patterns compared to occupied areas and unoccupied areas on the map. Here, 710 represents a map, and 711 represents an unmeasured area. The information to be displayed is displayed on a display device 410 on a controller 400 operated by a user 300 via a communication unit (not shown).

[0043] If the user 300 checks the information displayed here and finds any areas that have not been measured, he or she can delete the dynamic object from the map, and then move and rotate the mobile robot 200 to measure again so that the 3D sensor 100 can measure the areas that have not been measured. This allows for the creation of a high-quality map with no areas that have not been measured.

[0044] In step S670, information processing device 500 repeats the processes of steps S610 to S660 until the user issues an instruction to end measurement. In each repetition, sensor information acquisition unit 510 acquires and processes sensor information at a new time.

[0045] As described above, in this embodiment, areas that have not been measured due to dynamic objects can be displayed on a map. By viewing this display, the user can control the position and direction of the 3D sensor to compensate for the measurement omissions, thereby creating a high-quality map with no measurement omissions.

[0046] (Another example of a sensor information acquisition unit) In the first embodiment, the 3D sensor 100 is a depth sensor that acquires a distance image. However, the 3D sensor may be any sensor that acquires shape information of an object. As described above, a distance image may be created by stereo matching using a stereo camera, or LiDAR (Light Detection And Ranging) may be used to acquire shape information using a laser. Alternatively, an imaging element such as a CMOS image sensor with a DAF function may be used. DAF stands for Dual Pixel Auto Focus.

[0047] Furthermore, the sensor information acquisition unit 510 may acquire information from multiple sensors in addition to the 3D sensor 100. For example, information may be acquired from additional sensors such as a 2D camera that acquires images, a gyro sensor, or a magnetic sensor that recognizes the position of magnetic tape attached to the floor. The acquired information from multiple sensors may be used by the position and orientation information acquisition unit 520 and the object recognition unit 540, as will be described in detail later.

[0048] (Another example of a position and orientation information acquisition unit) Furthermore, in the first embodiment, the position and orientation information acquisition unit 520 acquires the position and orientation estimated by the position and orientation information estimation unit 230. In this embodiment, the position and orientation information estimation unit 230 estimates the position and orientation of the 3D sensor 100 based on a control signal for driving the mobile robot 200 as a moving body. However, the position and orientation acquired by the position and orientation information acquisition unit 520 may be estimated by any method.

[0049] For example, the position and orientation information estimation unit 230 may estimate the position and orientation of the 3D sensor based on a range image acquired from the 3D sensor by the sensor information acquisition unit 510. For example, the position and orientation of the 3D sensor at each time i can be estimated and acquired by inputting a range image Di at each time i, calculating the relative position and orientation between images using ICP, and calculating the cumulative value of the relative position and orientation. ICP stands for Iterative Closed Point.

[0050] Methods for estimating the position and orientation of a sensor based on a range image are well known, and for example, the method described in the aforementioned Non-Patent Document 3 may be used. Furthermore, when the sensor information acquisition unit 510 acquires additional sensor information such as a two-dimensional camera, gyro sensor, or magnetic sensor, the position and orientation may be estimated by further utilizing this sensor information. In other words, the position and orientation information may be estimated using at least one of a camera capturing a two-dimensional image, a gyro sensor, a magnetic sensor, or the like.

[0051] (Another example of a map information creation section) In the first embodiment, the map information creation unit 530 creates a two-dimensional map by projecting three-dimensional coordinates included in the distance image acquired by the sensor information acquisition unit 510 onto two-dimensional coordinates on a map. However, the map created by the map information creation unit 530 may be a three-dimensional map. In other words, the shape map created by the map information creation unit 530 may be a three-dimensional or two-dimensional shape map.

[0052] Similarly, in the case of three dimensions, a map can be created by determining occupied and unoccupied areas in a space. The method of creating a three-dimensional map can be the method described in the aforementioned Non-Patent Document 3. In this case, a map showing occupied and unoccupied areas may be used, or a map showing the boundary between occupied and unoccupied areas with lines may be used.

[0053] Furthermore, the map information creation unit 530 may create a map using three-dimensional coordinates from which three-dimensional coordinates Tik corresponding to the parts of dynamic objects recognized by the object recognition unit 540 have been removed. By removing the parts of dynamic objects, the map that can be created can be composed of only static objects. The method of Patent Document 1 mentioned above may be used as a method for removing dynamic objects from a map.

[0054] (Another example of an object recognition unit) Furthermore, in the first embodiment, the object recognition unit 540 recognized a portion corresponding to a dynamic object from the shape information acquired by the 3D sensor 100. Specifically, the dynamic object was recognized based on the shape information acquired from the 3D sensor from the sensor information acquired by the sensor information acquisition unit 510. However, the object recognition unit 540 may use any method as long as it can recognize a dynamic object. The sensor information acquisition unit 510 may acquire two-dimensional image data from a camera that captures two-dimensional image information, and recognize the portion of the dynamic object based on the acquired two-dimensional image data.

[0055] Furthermore, in the first embodiment, a person is set to be recognized as a dynamic object, but the dynamic object may be a non-human object, such as a small animal. Alternatively, another mobile robot may be assumed, or a temporarily placed object that is likely to move may be assumed. Regardless of the object, by inputting various range images or 2D images showing the assumed dynamic object and learning the areas where the dynamic object exists, it is possible to recognize similar areas in a given range image Di.

[0056] (Another example of the setting section) In the first embodiment, the tertiary coordinates Tik included in the part recognized by the object recognition unit 540 are projected onto the map (Uik, 810 in FIG. 6), and the area behind (behind) it is set as a measurement omission area indicating a measurement omission. However, the part recognized by the object recognition unit 540, which is the source of the measurement omission area, itself may be set as the measurement omission area. In other words, Uik itself projected onto the map may be set as shown in 721 in FIG. 8.

[0057] 8 is a diagram showing another example of displaying an unmeasured area on a map in the first embodiment. Although 721 does not directly indicate an unmeasured area, the user 300 can see it and indirectly know that there is an unmeasured area behind it.

[0058] (Another example of a display control unit) In the first embodiment, the display control unit 560 displays occupied and unoccupied areas on the map in a different pattern from measurement omission areas to distinguish them from each other. However, the display control unit 560 may use any method as long as it generates a display signal for distinguishing between a shape map made up of occupied and unoccupied areas on the map and measurement omission areas indicating measurement omissions. The occupied and unoccupied areas on the map may be displayed in different colors from the measurement omission areas, or the measurement omission areas may be displayed so as to be distinguishable by blinking.

[0059] Furthermore, the displayed information may include additional information other than the occupied area, unoccupied area, and measurement-omitted area. For example, the position and orientation of the mobile robot 200 at the current time may be displayed on a map. By displaying the information about the mobile robot 200 on a map, the user can easily confirm the relationship between the area indicating measurement omission and the position and orientation of the mobile robot 200, and can easily understand how to move and rotate the mobile robot 200 to measure the measurement-omitted area.

[0060] <Second embodiment> In the first embodiment, a measurement-omitted region is displayed by recognizing a dynamic object, and the measurement-omitted region indicates that the region could not be measured at the time the dynamic object was recognized. In other words, if the region can be measured at another time, there is a possibility that the measurement-omitted region will disappear.

[0061] Specifically, user 300 may be able to eliminate an unmeasured area displayed on display device 410 by moving and rotating mobile robot 200 so that 3D sensor 100 faces the unmeasured area and then remeasures it. Since an area that can be remeasured in this way no longer needs to be displayed as an unmeasured area, it is desirable to remove it from the map displayed on display device 410.

[0062] In this embodiment, the setting unit 550 first calculates a measurement omission area when a dynamic object is recognized, as in the first embodiment. Here, the time when the dynamic object is recognized is set to x, and the set measurement omission area is set to Ax. Then, if this area Ax can be measured at any of the other times (0 to t, excluding x), this area Ax is determined to have been measured. Then, the setting unit 550 sets the measurement omission area excluding the area determined to have been measured.

[0063] Specifically, under the condition that no dynamic object is recognized at all times (0 to t, excluding x), the area measured by 3D sensor 100 is searched for an overlapping portion with unmeasured area Ax. If an overlapping portion is found, the overlapping portion is determined as having been measured in the information set by setting unit 550, and only the remaining area is set as an unmeasured area.

[0064] 9 is a diagram showing an example of re-measuring an unmeasured area in the second embodiment, with 730 being an example of a map in which an unmeasured area has been re-measured at a certain time u. 731 represents the position of 3D sensor 100 at time u, and 732 represents the re-measurement area re-measured by 3D sensor 100. As shown in FIG. 9, if re-measurement area 732 overlaps with measurement unmeasured area 711, the overlapping area is determined to have been measured. Furthermore, map information creation unit 530 determines whether the area is occupied or unoccupied based on the shape information at time u, and expresses this as black and white information.

[0065] Fig. 10 is a diagram showing an example in which the remeasured portion of the measurement omission area in the second embodiment has been deleted, and shows an example in which the overlapping area in Fig. 9 has been hidden. In 740 of Fig. 10, area 741 indicates the remaining measurement omission area that has not yet been remeasured.

[0066] As described above, in this embodiment, if an unmeasured area can be remeasured at a different time, it is determined to have been measured and is deleted or hidden from the map. In other words, among the unmeasured areas, areas that were measured by the 3D sensor at a time different from the time when the object recognition unit 540 recognized the object are excluded from the unmeasured areas. This allows the user to check whether an unmeasured area was remeasured and prevents forgetting to remeasure. This makes it possible to create a high-quality map with no unmeasured areas.

[0067] (Removal of manual settings) As described above, in the second embodiment, if there is a portion of the measurement range of three-dimensional sensor 100 that overlaps with area Ai, that overlapping portion is deleted from the settings of setting unit 550 so that it is not displayed on the map. However, the display of the measurement omission area that has been set on the map may also be deleted based on input from user 300.

[0068] For example, the user 300 specifies an area on the map that has not been measured using the controller 400, and the display control unit 560 controls the area so that it is not displayed. By performing this operation, the user 300 can delete each area that has not been measured while checking whether it has been remeasured, and can prevent forgetting to remeasure an area that has not been measured.

[0069] <Third embodiment> In the first embodiment, the user 300 looked at the displayed measurement omission area and controlled the movement and rotation of the mobile robot 200 so that the three-dimensional sensor 100 could measure that area. However, the control of the mobile robot 200 may be automatic rather than manual by the user 300. Note that in this embodiment, since the user 300 does not need to manually control the mobile robot 200 to remeasure, the display control unit 560 does not need to control the display of information about the measurement omission to the user 300.

[0070] Fig. 11 is a functional block diagram showing the configuration of the inside of an information processing device 501 and related modules in embodiment 3. In Fig. 11, a sensor information acquisition unit 510, a position and orientation information acquisition unit 520, a map information creation unit 530, an object recognition unit 540, and a setting unit 550 have the same functions as the functional modules with the same names in embodiment 1. Also, unlike embodiment 1, a display control unit 560 is not included, but instead a control amount calculation unit 570 is provided, and the control unit 210 is controlled by the output of the control amount calculation unit 570.

[0071] Fig. 12 is a flowchart showing a processing procedure in the third embodiment. Note that the computer in the information processing device 500 executes a computer program stored in the memory to perform the operation of each step in the flowchart in Fig. 12.

[0072] Steps S900 to S950 and step S970 are the same as steps S600 to S650 and step S670 in the first embodiment, respectively, and therefore will not be described here. However, the display control process by display control unit 560 in step S660 in the first embodiment is not performed, and instead, control amount calculation process in step S960 is performed. The following describes the process of calculating the control amount in the control amount calculation unit 570 in step S960.

[0073] In step S960, the information processing device 500 automatically calculates the movement and rotation amounts of the mobile robot 200 using the control amount calculation unit 570 so that the 3D sensor 100 can measure the unmeasured area. Specifically, first, the unmeasured area to which the 3D sensor 100 is to move is automatically selected. In this embodiment, the unmeasured area closest to the current position and orientation of the 3D sensor 100 is selected.

[0074] The selected measurement omission area is designated as As. Next, a virtual viewpoint for re-measurement is set near the selected area As. In this embodiment, the viewpoint is set in an area within a predetermined range N from the measurement omission area As, in a location that is unoccupied on the map (i.e., a location where there is no object), and that is closest to the current position and orientation of the 3D sensor. Note that this viewpoint is set in the direction of the measurement omission area As.

[0075] Next, the amount of movement and (horizontal) rotation of the mobile robot 200 are calculated so that the difference between the current position and orientation of the 3D sensor acquired by the position and orientation information acquisition unit 520 and the set viewpoint becomes zero. The calculated amount of movement and amount of rotation are then output to the control unit 210, and the mobile robot 200 automatically moves and rotates toward the measurement-omitted area. This allows the measurement-omitted area to be remeasured.

[0076] As described above, in this embodiment, the mobile robot is automatically controlled so that the missed measurement area can be remeasured, and the missed measurement area is remeasured. That is, based on the position and orientation information and the missed measurement area, the movement or rotation of the mobile robot is controlled so that the missed measurement area can be measured using a 3D sensor. This reduces the user's effort and makes it possible to create a high-quality map without missed measurement areas.

[0077] (Automatically calculates and displays the amount of movement and rotation on the map) In the third embodiment, the mobile robot 200 is automatically controlled based on the movement amount and rotation amount calculated by the control amount calculation unit 270, but the calculated movement amount and rotation amount may be displayed on the display device 410 so that the user 300 can input the control amount by referring to them. In this case, the display device 410 may display the numerical values ​​of the movement amount and rotation amount, or may indicate the control amount by showing the position and direction on a map.

[0078] The first to third embodiments described above can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. Alternatively, the programs may be provided by recording them on a computer-readable recording medium.

[0079] <Effects> In the first embodiment, areas that have not been measured due to dynamic objects are displayed on a map. By viewing this display, the user can move and rotate the 3D sensor of the mobile robot to fill in the measurement omissions, thereby creating a high-quality map without any measurement omissions.

[0080] In the second embodiment, if an unmeasured area can be measured at a different time, it is determined to have been measured and is not displayed on the map. This allows the user to easily check whether an unmeasured area has been remeasured, and prevents forgetting to remeasure. This also makes it possible to create a high-quality map with no unmeasured areas.

[0081] In the third embodiment, the mobile robot is automatically controlled so that areas that have not been measured can be remeasured, thereby reducing the user's workload and creating a high-quality map with no areas that have not been measured.

[0082] <Definition, etc.> In the above embodiments, the 3D sensor may be any sensor that can acquire shape information of an object, such as a depth sensor that acquires a distance image, a LiDAR that acquires shape information using a laser, or a stereo camera that can create a distance image by stereo matching.

[0083] The sensor information acquisition unit may acquire information from multiple sensors in addition to the 3D sensor. For example, it may acquire information from sensors such as a 2D camera that acquires images, a gyro sensor, and a magnetic sensor that recognizes the position of magnetic tape attached to the floor.

[0084] The position and orientation information acquisition unit may acquire the estimated position and orientation by any method, including a method based on a control signal of the mobile robot, or a method based on sensor information acquired by a sensor information acquisition unit.

[0085] The map created by the map information creation unit can be any map that can represent the shape of the environment. It can be a two-dimensional map or a three-dimensional map. It can also be a map that shows occupied and unoccupied areas, or a map that represents their boundaries with lines.

[0086] The object recognition unit may use any method that can recognize a dynamic object. The object may be recognized based on shape information acquired from a three-dimensional sensor, or may be recognized based on two-dimensional image information.

[0087] The method of setting the measurement omission area indicating the measurement omission in the setting unit may be any method as long as it can indicate the area behind the part of the dynamic object recognized by the object recognition unit. It is also possible to set the part of the dynamic object recognized by the object recognition unit and indirectly indicate the area behind it. The display control unit may use any method as long as it can distinguish between the shape map and the measurement omission area. The display may be performed by changing the pattern or color, or by changing whether or not to blink.

[0088] In the above-described embodiment, an example has been described in which an image processing system is mounted on a mobile robot, such as an AGV or an AMR. However, the mobile body in this embodiment is not limited to a mobile robot, and may be a drone or any other mobile device that can move. Furthermore, the information processing system in this embodiment includes those mounted on such mobile bodies.

[0089] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. [Explanation of symbols]

[0090] 100 3D sensor 200 Mobile Robots 300 users 400 Controller 410 Display device

Claims

1. a sensor information acquiring means for acquiring sensor information including shape information from a three-dimensional sensor mounted on the moving object; a position and orientation information acquisition means for acquiring position and orientation information of the three-dimensional sensor when the three-dimensional sensor measures the shape information; an object recognition means for recognizing a part of a dynamic object based on the sensor information; a setting means for setting, when the part is recognized, an area within a predetermined range from the part behind the part when viewed from the three-dimensional sensor based on the sensor information, the position and orientation information, and the part, as a measurement omission area that indicates a measurement omission and should be remeasured; a map information generating means for generating a shape map of the surrounding environment in which the unmeasured area can be identified based on the shape information, the position and orientation information, and the unmeasured area information; 13. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, further comprising a display control means for generating a display signal for distinguishing and displaying the shape map and the measurement omission region.

3. 3. The information processing apparatus according to claim 1, wherein the setting unit sets the part as the measurement omission region.

4. The information processing device according to any one of claims 1 to 3, characterized in that the setting means excludes from the measurement omission area an area that the three-dimensional sensor was able to measure at a time different from the time at which the object recognition means recognized the dynamic object.

5. A camera for acquiring two-dimensional image information is provided.

5. The information processing apparatus according to claim 1, wherein the object recognition means recognizes the portion of the dynamic object based on the two-dimensional image information.

6. The information processing device according to any one of claims 1 to 5, further comprising a control means for controlling a movement or rotation of the moving body for measuring the measurement omission area with the three-dimensional sensor based on the position and orientation information and the measurement omission area.

7. 7. The information processing device according to claim 1, wherein the three-dimensional sensor includes an imaging element having a stereo camera, LiDAR, or DAF function.

8. 8. The information processing apparatus according to claim 1, wherein the position and orientation information acquisition means acquires the position and orientation information of the three-dimensional sensor based on a control signal for driving the moving body.

9. 8. The information processing apparatus according to claim 1, wherein the position and orientation information acquisition means acquires the position and orientation information of the three-dimensional sensor based on a range image acquired from the three-dimensional sensor.

10. 10. The information processing apparatus according to claim 1, wherein the position and orientation information acquisition means acquires the position and orientation information using at least one of a camera for capturing a two-dimensional image, a gyro sensor, and a magnetic sensor.

11. 11. The information processing apparatus according to claim 1, wherein the map information generating means generates the shape map in three dimensions or two dimensions.

12. An information processing device according to any one of claims 1 to 11, The three-dimensional sensor; and a control means for controlling the movement or rotation of the moving body based on the shape map.

13. a sensor information acquisition step of acquiring sensor information including shape information from a three-dimensional sensor mounted on the moving object; a position and orientation information acquisition step of acquiring position and orientation information of the three-dimensional sensor when the three-dimensional sensor measures the shape information; an object recognition step of recognizing a part of a dynamic object based on the sensor information; a setting step of setting, when the part is recognized, an area within a predetermined range from the part behind the part when viewed from the three-dimensional sensor based on the sensor information, the position and orientation information, and the part, as a measurement omission area that indicates a measurement omission and should be re-measured; a map information creation step of creating a shape map of the surrounding environment in which the unmeasured area can be identified based on the shape information, the position and orientation information, and the information on the unmeasured area; 13. An information processing method comprising:

14. An information processing device according to any one of claims 1 to 11 or a computer program for controlling each means of a moving body according to claim 12 by a computer.

15. A computer readable storage medium storing the computer program according to claim 14.

Images