Information processing device, control method, and program

Abstract

To generate a three-dimensional model of a real space that reproduces a continuous surface.SOLUTION: An information processing device comprises: three-dimensional reference surface information input means for inputting three-dimensional reference surface information obtained from information representing a real space on a two-dimensional surface or three-dimensional information presenting the real space; three-dimensional point cloud input means for inputting three-dimensional point cloud information having measured the real space; three-dimensional point cloud selection means for selecting, from the three-dimensional point cloud, a three-dimensional point cloud related to a specific surface included in the real surface; and three-dimensional surface information calculation means for calculating three-dimensional surface information representing a shape of the real space on the basis of the three-dimensional reference surface information and the selected three-dimensional point cloud.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method, and a program. [Background technology]

[0002] Three-dimensional shapes in real space are acquired based on distance information and image information measured by sensors such as cameras and LiDAR. LiDAR is an abbreviation for LIGHT DETECTION AND RANGING.

[0003] In Patent Document 1, a three-dimensional shape in real space is reconstructed as a three-dimensional model in a computer from a plurality of depth images captured by a depth camera capable of measuring the distance to an object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5906258 [Non-patent literature]

[0005] [Non-Patent Document 1] M. Labbe et al., “RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operation,” Journal of Field Robotics, 2019 [Non-Patent Document 2] GRISETTI ET AL., IMPROVED TECHNIQUES FOR GRID MAPPING WITH RAO-BLACKWELLIZED PARTICLE FILTERS, TRANS ON ROBOTICS 23.1(2007) [Non-Patent Document 3] R.RAGURAM ET.AL., “USAC: A UNIVERSAL FRAMEWORK FOR RANDOM SAMPLE CONSENSUS”, PAMI, 2013 [Non-Patent Document 4] CARLOS ET AL, ORB-SLAM3: AN ACCURATE OPEN-SOURCE LIBRARY FOR VISUAL, VISUAL-INERTIAL AND MULTI-MAP SLAM, IEEE TRANS. ROBOTICS, 2021. Summary of the Invention [Problem to be solved by the invention]

[0006] In the method of Patent Document 1, structures that are actually continuous surfaces may be estimated as discontinuous three-dimensional models with irregularities or holes due to measurement errors. In addition, surfaces such as walls that are actually smooth may be estimated as surfaces with small irregularities.

[0007] In view of the above, an object of the present invention is to generate a three-dimensional model of a real space in which surface information is appropriately reproduced. [Means for solving the problem]

[0008] An information processing device according to one embodiment of the present invention has a three-dimensional reference plane information input means for inputting three-dimensional reference plane information obtained from information representing a real space on a two-dimensional plane or three-dimensional information representing the real space, a three-dimensional point cloud input means for inputting three-dimensional point cloud information obtained by measuring the real space, a three-dimensional point cloud selection means for selecting a three-dimensional point cloud relating to a specific surface contained in the real space from the three-dimensional point clouds, and a three-dimensional surface information calculation means for calculating three-dimensional surface information representing the shape of the real space based on the three-dimensional reference plane information and the selected three-dimensional point cloud. Effect of the Invention

[0009] According to the present invention, a three-dimensional model of real space in which surface information is appropriately reproduced can be generated. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a situation in which the device according to the first embodiment of the present invention is used. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of a moving body according to a first embodiment of the present invention. [Diagram 3] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment of the present invention. [Figure 4] FIG. 1 is a diagram showing a concept of processing according to a first embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram illustrating a configuration of an apparatus according to a first embodiment of the present invention. [Figure 6] 4 is a flowchart showing a process flow of the device according to the first embodiment of the present invention. [Figure 7] FIG. 4 is a diagram illustrating the configuration of an apparatus according to a second embodiment of the present invention. [Figure 8] 10 is a flowchart showing a process flow of an apparatus according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing an example of a GUI of an apparatus according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicated descriptions are omitted.

[0012] [First embodiment] The first embodiment of the present invention relates to the generation of a three-dimensional model of a real space. The three-dimensional model is also called a three-dimensional map or a three-dimensional cartography.

[0013] Fig. 1 is a diagram for explaining a situation in which the device according to the first embodiment of the present invention is used. Fig. 1 is a diagram for explaining a use case in which a three-dimensional model generated by the device according to the first embodiment of the present invention is used.

[0014] A space U10 indicates a real space to which the information processing device 1 according to the first embodiment of the present invention is applied. A moving object U101 is a moving object that moves in the space U10. A sensor is disposed in the moving object U101, and information on surrounding structures and position information and orientation information of the moving object U101 can be acquired based on the measurement value of this sensor. The structural information refers to the arrangement and shape of structures such as a wall U102 that is a wall in the space U10, a pillar U103 that is a column in the space U10, and fixtures (not shown). The position information and orientation information of the moving object U101 represent the position and orientation of the moving object in the real space, and are represented by three parameters, namely, two-dimensional position coordinates (X, Y) and a direction (θ).

[0015] The space U20 is a space where a user U201 is present. The user U201 is in a remote location different from the location where the moving object U101 is located. FIG. 1 shows a state where a three-dimensional model generated using the first embodiment of the present invention is displayed on a tablet U202 operated by the user U201, and the moving object U101 is remotely controlled. The three-dimensional model of the real space generated using the first embodiment of the present invention and the three-dimensional model of the moving object U101 are displayed on the GUI of the tablet U202. GUI is an abbreviation for Graphical User Interface. While looking at the GUI of the tablet U202, the user U201 remotely controls the moving object U101 so that the moving object U101 does not come into contact with or collide with a structure. The GUI of the tablet U202 will be described in detail later.

[0016] The mobile object U101 is, for example, an AMR such as a cleaning robot, an AGV, an autonomous vehicle, a drone, etc. AMR is an abbreviation for Autonomous Mobile Robot. AGV is an abbreviation for Automatic Guides Vehicle. In this embodiment, a case where an information processing device is arranged in an AMR will be described.

[0017] 2 is a diagram illustrating the configuration of a moving body U101 according to the first embodiment of the present invention. The moving body U101 includes an information processing device Y130. In this embodiment, a moving body system for generating a three-dimensional model of the surroundings while moving the moving body U101 and controlling the moving body U101 will be described.

[0018] The information processing device Y130 is connected to various sensors such as an imaging device arranged on the moving body U101. Here, the various sensors arranged on the moving body U101 include a color camera Y100, a distance camera Y101, a stereo camera Y102, an IMU·Y103, and a LiDAR·Y104. IMU is an abbreviation for Inertial Measurement Unit. LiDAR is an abbreviation for Light Detection And Ranging.

[0019] The stereo camera Y102 captures a stereo image consisting of two monochrome images, and outputs the captured stereo image to the information processing device Y130. The IMU·Y103 measures inertial information, which is acceleration and angular velocity generated when the moving body U101 moves, and outputs the measured inertial information to the information processing device Y130. The LiDAR·Y104 outputs scan information, which measures the distance from the moving body U101 to structures such as walls and fixtures present in the vicinity, to the information processing device Y130. Note that the LiDAR·Y104 in this embodiment is a 2D-LiDAR (two-dimensional LiDAR) that measures a point cloud at the same height. The information processing device Y130 calculates the position and orientation of the moving body U101 in real space using the input stereo images, inertial information, scan information, and information from the encoders Y112 and Y122 described later. In this embodiment, the information processing device Y130 calculates the position and orientation of the moving body U101 in real space from a plurality of sensor information by the method of Labbe et al. (see Non-Patent Document 1).

[0020] The color camera Y100 captures a color image. The captured color image is used to add color information and texture information to a three-dimensional model. The distance camera Y101 captures a depth image (also called a depth image or distance image). The captured depth image is used to generate a three-dimensional model, as described below.

[0021] Next, a method for controlling the moving object U101 will be described. The information processing device Y130 transmits drive signals to the motors Y111 and Y121 through the motor controllers Y110 and Y120 to rotate the motors Y111 and Y121. This causes the wheels Y113 and Y123 to rotate, and the moving object U101 moves.

[0022] The encoders Y112, Y122 detect the amount of actual rotation of the wheels Y113, Y123, and output rotation information to the information processing device Y130. The information processing device Y130 uses the input rotation amount to calculate the position and orientation of the moving body U101 in real space.

[0023] The tablet U202 is connected to the information processing device Y130 via wireless communication, and presents information of the information processing device Y130 to the user U201 through a GUI displayed on the screen of the tablet U202. The user remotely controls the mobile object U101 using the information on the presented GUI.

[0024] The information storage device Y140 holds the layout information and BIM information of structures in the real space acquired in advance using the mobile body U101. BIM is an abbreviation for Building Information Modeling. The information processing device Y130 is connected to the information storage device Y140 via wireless communication. The information processing device Y130 reads out two-dimensional map information from the information storage device Y140 and uses it in the processing described below.

[0025] In this embodiment, an example will be described in which the information processing device Y 130 is disposed in the mobile body U 101. However, a part or the whole of the information processing device Y 130 can also be provided on a server connected via a network.

[0026] 3 is a diagram showing a hardware configuration of an information processing device according to a first embodiment of the present invention. The information processing device Y130 has a CPU H11, a ROM H12, a RAM H13, an external memory H14, an input unit H15, a display unit H16, a communication I / F H17, an I / O H18, and a system bus H21. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. I / F is an abbreviation for interface. I / O is an abbreviation for Input / Output.

[0027] The CPU H11 executes a control program, thereby carrying out the processing of this embodiment. The CPU H11 also controls various devices connected to the system bus H21. The ROM H12 stores the BIOS program and boot program. BIOS is an abbreviation for Basic Input Output System. The RAM H13 is used as the main storage device for the CPU H11. The external memory H14 stores the programs processed by the information processing device Y130 in this embodiment.

[0028] The input unit H15 performs processing related to input of information and the like from an input device such as a keyboard or a mouse. In this embodiment, the input device corresponds to the touch display of the tablet U202. The display unit H16 outputs the calculation results of the information processing device Y130 to a display device according to instructions from the CPU H11. In this embodiment, the display device corresponds to the display of the tablet U202. In particular, the three-dimensional model generated by applying this embodiment is displayed on the display. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator.

[0029] The communication I / F H17 is a communication interface that communicates information via a network. The communication I / F H17 can be of any type, such as Ethernet, USB, serial communication, or wireless communication. USB is an abbreviation for Universal Serial Bus. A three-dimensional model is transmitted from the information processing device Y130 to the tablet U202 via the display unit H16 via the communication I / F H17. The I / O H18 inputs information from sensors, such as an imaging device, disposed on the mobile object U101 (described later), to the information processing device Y130.

[0030] FIG. 4 is a diagram showing the concept of processing according to the first embodiment of the present invention, and is a diagram showing the concept of a method for generating a three-dimensional model. FIG. 4(A) is a diagram showing an example of a real space in which a three-dimensional model is generated. FIG. 4(B) is a diagram showing two-dimensional space information in which a structure is projected two-dimensionally. FIG. 4(C) is a diagram explaining the concept of calculating a pseudo three-dimensional surface from two-dimensional reference surface information. FIG. 4(D) is a diagram showing the concept of generating a local plane by plane fitting a three-dimensional point group obtained from a depth image. FIG. 4(E) is a schematic diagram in which FIG. 4(D) is projected onto a floor surface. FIG. 4(F) is a conceptual diagram of calculated three-dimensional surface information.

[0031] In Fig. 4(A), M10 shows an example of a real space for generating a three-dimensional model in this embodiment. M101 is a wall, which is an example of a structure in the real space. M102 is a pillar, which is an example of a structure in the real space. M103 is outside the space in which the moving object moves (outside the measurement range). M104 is the space in which the moving object moves (within the measurement range).

[0032] In FIG. 4B, M20 indicates a two-dimensional map that is two-dimensional spatial information in which a structure is projected two-dimensionally. In this embodiment, a two-dimensional map is used that is generated based on scan information obtained by scanning a two-dimensional plane in real space based on a LiDAR sensor arranged on a moving body U101. The two-dimensional map is an occupancy grid map, and is generated by a method of GRISETTI et al. (see Non-Patent Document 2). M201 indicates a boundary surface between a wall and a space in the occupancy grid map. M202 indicates a boundary surface between a pillar and a space in the occupancy grid map. M203 is an unmeasured area in the occupancy grid map, and is an area in which it is not yet determined whether an object is located in the space. M204 is a hollow area in the space, and indicates an area in which no object exists (i.e., an area in which a moving body can move). Hereinafter, in this embodiment, information indicating the boundary surface between a structure such as a wall M201 or a pillar M202 in the two-dimensional map and a space M204, which is a space in which no object exists, is called two-dimensional reference surface information.

[0033] In FIG. 4(C), M30 shows a conceptual diagram of calculating a pseudo three-dimensional surface from two-dimensional reference plane information in an area surrounded by a dashed line M205 in M20. M301 shows the two-dimensional reference plane information of M201 three-dimensionally. As shown by the arrow M302, the three-dimensional surface shown in M303 can be calculated by extending the two-dimensional reference plane information in a direction perpendicular to the floor surface. In this embodiment, the three-dimensional surface calculated from the two-dimensional reference plane information is called three-dimensional reference plane information.

[0034] In FIG. 4(D), M40 is a diagram showing a concept of plane fitting of a three-dimensional point group M401 obtained from a depth image acquired by a depth camera arranged on a moving body U101 to generate local planes M402 to M405. For plane estimation, for example, a RANSAC algorithm (see Non-Patent Document 3) is used, which calculates a plane by obtaining plane parameters based on a small number of samples and voting the number of points that match the plane parameters. Since the three-dimensional point group M401 contains noise, it is calculated by dividing one plane into multiple local planes as shown in FIG. 4(D). M403 indicates a low-height wall surface. M406 is a hole like a window in a wall surface that is not included in the three-dimensional reference plane information.

[0035] In Fig. 4(E), M50 is a schematic diagram of M40 projected onto a floor surface. M501 is a three-dimensional point group corresponding to the three-dimensional point group M401. M502 is two-dimensional reference plane information corresponding to M301. M503 to M506 are three-dimensional plane information calculated based on the three-dimensional point group, and correspond to M402 to M405, respectively.

[0036] The two-dimensional reference plane information indicated by M502 is a plane measured by LiDAR, and there is a positional deviation (deviation) from the original position of the plane in real space. On the other hand, the three-dimensional point cloud indicated by M501 includes noise, and indicates that the plane includes irregularities, that is, the plane is divided as shown in M503 to M506. In this embodiment, the two-dimensional reference plane information and the three-dimensional point cloud are used together. As a result, in this embodiment, three-dimensional information (here, holes) not included in the two-dimensional reference plane information is added to the reference plane information, and noise (here, irregularities) included in the three-dimensional point cloud is reduced. As a result, in this embodiment, three-dimensional plane information M507 is calculated by appropriately using the surface information.

[0037] In Fig. 4(F), M60 is a conceptual diagram of the calculated three-dimensional surface information 601. The position of the three-dimensional surface information M601 has been corrected based on the measured three-dimensional point cloud for the two-dimensional reference plane information M301 and the three-dimensional reference plane information M303, which are misaligned. In addition, a low-height area of ​​the wall surface is reflected in the three-dimensional plane of M601. Furthermore, a window (M602), which is a three-dimensional shape that was not included in the three-dimensional reference plane information, is reflected.

[0038] In this embodiment, the surface is a plane. The two-dimensional reference surface information is stored as line segment information that stores the two-dimensional coordinates (X, Y) of points corresponding to the vertices of the surface. The three-dimensional reference surface information and the three-dimensional surface information are stored as mesh model data that stores the three-dimensional coordinates (X, Y, Z) of each vertex of the surface for each surface. The three-dimensional point group is a point cloud (point group), and each holds three parameters of the three-dimensional coordinates (X, Y, Z).

[0039] Fig. 5 is a diagram for explaining the configuration of an apparatus according to the first embodiment of the present invention. Fig. 5 shows the functional configuration of an information processing apparatus 1. The information processing apparatus 1 in Fig. 5 corresponds to the information processing apparatus Y130 in Fig. 2. In this embodiment, the information processing apparatus 1 has a three-dimensional reference plane information input unit 101, a three-dimensional point group input unit 102, a three-dimensional point group selection unit 103, and a three-dimensional plane information calculation unit 104. The three-dimensional reference plane information input unit 101, the three-dimensional point group input unit 102, the three-dimensional point group selection unit 103, and the three-dimensional plane information calculation unit 104 are realized by executing a program in the CPU·H11.

[0040] A three-dimensional reference plane information input unit 101 inputs three-dimensional reference plane information held in a holding unit (not shown), and outputs the input three-dimensional reference plane information to a three-dimensional plane information calculation unit 104 .

[0041] A three-dimensional point cloud input unit 102 inputs a three-dimensional point cloud held in a holding unit (not shown), and outputs the input three-dimensional point cloud to a three-dimensional point cloud selection unit 103 .

[0042] The three-dimensional point cloud selection unit 103 performs plane clustering on the three-dimensional point cloud from the three-dimensional point cloud input unit 102, and divides the three-dimensional point cloud for each plane. The three-dimensional point cloud selection unit 103 selects a three-dimensional point cloud that belongs to one of the divided planes. The three-dimensional point cloud selection unit 103 outputs the selected three-dimensional point cloud to the three-dimensional surface information calculation unit 104.

[0043] The three-dimensional surface information calculation unit 104 calculates three-dimensional surface information using the three-dimensional reference surface information from the three-dimensional reference surface information input unit 101 and the three-dimensional point group from the three-dimensional point group information selection unit 103. The three-dimensional surface information calculation unit 104 outputs the calculated three-dimensional surface information to a storage unit (not shown) and also to a display unit (not shown). The display unit (not shown) displays the three-dimensional surface information to the user.

[0044] Fig. 6 is a flowchart showing the flow of processing of the device according to the first embodiment of the present invention. Fig. 6 is a flowchart explaining the operation of the information processing device 1. The processing explained in Fig. 6 is automatically started when the information processing device 1 is powered on and the information processing device 1 starts up.

[0045] In step S101, the information processing device 1 initializes each component within the information processing device 1. That is, the CPU H11 reads a program from the external memory H14 and makes the information processing device 1 operable. Furthermore, the CPU H11 reads program setting parameters and the like from the external memory H14 into the RAM H13 as necessary. Once a series of initialization processes is completed, the process proceeds to step S102.

[0046] In step S102, the three-dimensional reference plane information input unit 101 inputs three-dimensional reference plane information via the communication I / F H17. The three-dimensional reference plane information input unit 101 outputs the input three-dimensional reference plane information to the three-dimensional plane information calculation unit 104. Then, the process of step S103 is executed.

[0047] In step S103, the three-dimensional point cloud input unit 102 inputs a three-dimensional point cloud. That is, the three-dimensional point cloud input unit 102 inputs a three-dimensional point cloud calculated from a depth image acquired by a depth camera (not shown) via the I / O·H 18. The three-dimensional point cloud input unit 102 outputs the input three-dimensional point cloud to the three-dimensional point cloud selection unit 103. Then, the process of step S104 is executed.

[0048] In step S104, the three-dimensional point group selection unit 103 performs plane clustering on the three-dimensional point group from the three-dimensional point group input unit 102. The RANSAC algorithm is used for the plane clustering. Specifically, the three-dimensional point group selection unit 103 calculates a plane equation based on three randomly selected points, calculates the distance between the remaining three-dimensional point groups that were not selected and the calculated plane, and counts the number of points within a predetermined distance. The three-dimensional point group selection unit 103 selects the point group of the plane with the largest calculated number as the first cluster, repeats the same plane clustering on the remaining point groups, and clusters the point groups on the second, third, and so on. In this way, the three-dimensional point group selection unit 103 clusters and selects the three-dimensional point group for each candidate that becomes a plane included in the three-dimensional point group. The three-dimensional point group selection unit 103 outputs the selected three-dimensional point group to the three-dimensional plane information calculation unit 104. Next, the process of step S105 is executed.

[0049] In step S105, the three-dimensional surface information calculation unit 104 judges whether the three-dimensional point group from the three-dimensional point group selection unit 103 corresponds to the three-dimensional reference surface information from the three-dimensional reference surface information input unit 101. That is, the three-dimensional surface information calculation unit 104 judges whether the three-dimensional surface generated based on the selected three-dimensional point group coincides with the three-dimensional reference surface or is included therein. Specifically, the three-dimensional surface information calculation unit 104 sets two conditions: that the inner product of the normal between the three-dimensional plane information obtained from the three-dimensional point group and the three-dimensional reference surface is a predetermined value or more, and that the distance is a predetermined value or less. Then, when these two conditions are satisfied, the three-dimensional surface information calculation unit 104 judges that the selected three-dimensional point group is a three-dimensional point group that corresponds to the three-dimensional reference surface. When the three-dimensional surface information calculation unit 104 judges that the three-dimensional point group corresponds to the three-dimensional reference surface information, the process of step S106 is executed. If the three-dimensional surface information calculation unit 104 determines that the three-dimensional point group does not correspond to the three-dimensional reference surface information, the process proceeds to step S107.

[0050] In step S106, the three-dimensional surface information calculation unit 104 calculates three-dimensional surface information using the three-dimensional point group from the three-dimensional point group selection unit 103. Specifically, the three-dimensional surface information calculation unit 104 projects the three-dimensional point group onto a plane obtained from the three-dimensional point group using RANSAC. The three-dimensional surface information calculation unit 104 obtains a circumscribing rectangle for the projected point group, and calculates a mesh connecting the vertices as three-dimensional surface information. The three-dimensional surface information calculation unit 104 replaces the three-dimensional surface information calculated in this manner with the selected three-dimensional point group, and holds it in a holding unit (not shown). When a series of calculation processes for the three-dimensional surface information is completed, the process of step S107 is executed.

[0051] In step S107, the three-dimensional surface information calculation unit 104 judges whether or not to end the calculation of the three-dimensional plane information from the input three-dimensional point group. Specifically, the three-dimensional surface information calculation unit 104 judges to end the calculation of the three-dimensional plane information if the number of remaining points of the three-dimensional point group replaced in step S106 is less than a predetermined number. The three-dimensional surface information calculation unit 104 judges not to end the calculation of the three-dimensional plane information if the number of remaining points of the three-dimensional point group replaced in step S106 is equal to or greater than a predetermined number. If the three-dimensional surface information calculation unit 104 judges to end the calculation of the three-dimensional plane information, the process of step S108 is executed. If the three-dimensional surface information calculation unit 104 judges not to end the calculation of the three-dimensional plane information, the process of step S104 is executed, and the calculation of the three-dimensional plane information is repeated for the remaining points.

[0052] In step S108, the CPU H11 determines whether or not to end the process. If an instruction to end the calculation process of the three-dimensional surface information is input from the user via an input unit (not shown), the CPU H11 determines to end the process and shuts down the system. If an instruction to end the process is not input via the input unit, the CPU H11 determines not to end the process, and executes the process of step S103, repeatedly performing the calculation process of the three-dimensional surface information every time a three-dimensional point group is input.

[0053] <Effects> As described above, in the first embodiment, three-dimensional plane information is calculated by selecting three-dimensional point cloud information that corresponds to the three-dimensional reference plane information acquired in advance. The three-dimensional reference plane information includes positional errors, and when a measured three-dimensional point cloud is used, a discontinuous three-dimensional model such as unevenness or holes may be generated due to measurement errors. However, this embodiment makes it possible to generate a three-dimensional model that appropriately uses surface information. In particular, according to this embodiment, by using three-dimensional point cloud information that corresponds to the three-dimensional reference plane information acquired in advance, it becomes possible to generate a three-dimensional model while suppressing the generation of an erroneous plane due to measurement errors.

[0054] <Variation 1-1> In the first embodiment, various sensors (color camera Y100, distance camera Y101, stereo camera Y102, IMU·Y103, and LiDAR·Y104) are arranged on the moving body U101. The present invention is not limited to this, and sensors that are determined to be unnecessary depending on the situation do not need to be arranged on the moving body U101. Furthermore, in order to measure the surrounding real space in a short period of time, for example, multiple color cameras Y100 may be arranged, and the necessity and number of sensors can be freely selected depending on the situation.

[0055] The calculation of position and orientation using multiple sensor information is not limited to the method described in Non-Patent Document 1, as long as the position and orientation can be calculated using sensors such as IMU·Y103 and LiDAR·Y104. When some sensors are not arranged on the moving body U101 as described above, for example, the method of CARLOS et al. (see Non-Patent Document 4) that calculates the position and orientation based on the measurement values ​​of a camera or a camera and an IMU can be used. When only LiDAR is used, the method described in Non-Patent Document 2 can be used. In this way, any method of calculating the position and orientation can be used depending on the surrounding environment and the purpose.

[0056] In the first embodiment, the position and orientation of the moving body U101 are represented by parameters with three degrees of freedom, X, Y, and θ. As long as the position and orientation can be represented, they may be stored in another representation, such as a 3×3 rigid body transformation matrix. The position and orientation may be stored with six degrees of freedom including the height position (Z) and the rotation parameters of PITCH and ROLL, or may be stored in the form of a 4×4 matrix. By storing the position and orientation with six degrees of freedom, it becomes possible to calculate the height position and orientation when the moving body U101 moves up a slope, for example.

[0057] The two-dimensional map, which is two-dimensional spatial information, does not have to be an occupancy grid map created based on scan information measured by LiDAR, as long as it can calculate two-dimensional reference plane information. The two-dimensional map may be CAD data that stores a list of vertices of wall surfaces. The format of the CAD data is also arbitrary, such as straight lines and line segments, as long as it can express a two-dimensional reference plane. It is not limited to CAD, and may be a scanned image of an architectural drawing or a simple floor map.

[0058] In the first embodiment, the three-dimensional reference plane information and the three-dimensional reference plane information are stored as three-dimensional coordinates of the vertices of the plane. The three-dimensional reference plane information and the three-dimensional reference plane information only need to store information about the plane, and may be stored as parameters such as an equation of the plane or an equation of two straight lines including the plane. The three-dimensional reference plane information and the three-dimensional reference plane information may store the vertices of a circumscribing rectangle as a circumscribing rectangle that surrounds the selected point group.

[0059] In the first embodiment, the surface is described as a plane, but the present invention can be applied to any continuous surface such as a curved surface, not limited to a plane. For example, the two-dimensional reference surface information is stored as line segment information, but if three-dimensional surface information can be calculated, it can also be stored as parameters of a curve such as a Bezier curve or a spline curve. If two-dimensional reference surface information consisting of such a curve is extended in the height direction, three-dimensional reference surface information can be calculated as a curved surface. That is, the three-dimensional reference surface information may also be stored as parameters such as a NURBS surface. The three-dimensional surface information can also be generated as a similar curved surface. In addition, the curved surface may be configured to be approximated and stored with multiple small meshes. In addition, a primitive shape defined in advance may be applied to each piece of information. In addition, if a small number of points are selected and the parameters of the curved surface are calculated, the RANSAC algorithm can be applied to cluster the surfaces, not limited to the planes described in the first embodiment. When the primitive shape is applied to a three-dimensional point group, the applied point group may be clustered as one cluster.

[0060] A 3D point cloud is not limited to a point cloud, as long as it is a data structure that represents the presence or absence of a structure in a 3D space, but can also be stored in a voxel representation that represents whether or not an object is filled in the voxel space. Also, a 3D point cloud can be stored in any data structure, such as a 3D OCCUPANCY GRID MAP or a TSDF, which is a spatial field representation that can represent the surface of a structure (object). TSDF is an abbreviation for TRUNCATED SIGNED DISTANCE FUNCTION.

[0061] The reference plane information can also be obtained from a plane in the Z direction obtained from a point cloud acquired by 3D LiDAR (three-dimensional LiDAR). In the first embodiment, the three-dimensional reference plane information is created based on two-dimensional reference plane information obtained from scan information measured and created by LiDAR, but the present invention is not limited to this as long as surface information in real space can be acquired. For example, as a method that does not use two-dimensional reference plane information, a three-dimensional surface detected by three-dimensionally scanning the real space using 3D LiDAR can be used as the three-dimensional reference plane information. In addition, building wall surface information can be acquired from three-dimensional information such as BIM (Building Information Modeling) and 3D CAD to be used as the three-dimensional reference plane information.

[0062] In the first embodiment, two motor controllers Y110 and Y120 are arranged inside the moving body U101, but a device capable of driving multiple motors may be used for one motor controller. Also, although there are two pairs of motors and wheels, multiple wheels may be driven by one motor, and any combination is acceptable as long as the moving body U101 can be driven to any position and direction.

[0063] In the first embodiment, the information of the encoders Y112 and Y122 may be used to realize closed-loop control by inputting operation information of the wheels Y113 and Y123 to the motor controllers Y110 and Y120. The information of the encoders Y112 and Y122 may not be the rotation amount measured by the encoder, but may be a command value for rotating the motor.

[0064] The wireless communication in the first embodiment may be WiFi, BLUETOOTH (registered trademark), 5G (fifth generation mobile communication system), etc. Any method may be used as long as the information processing device 1 can communicate bidirectionally with the tablet U202 and the information storage device Y140.

[0065] In the first embodiment, the system is such that a user U201 remotely controls a moving object U101 using a tablet U202. However, the device for remotely controlling the moving object U101 is not limited to the tablet U201, and may be another mobile terminal or a general computer. That is, the device operated by the user U201 may be a mouse or a keyboard, and the device for displaying information on the information processing device 1 may be a liquid crystal or an OLED display.

[0066] The information storage device Y140 may be located on a cloud server or on a server in an in-house network. Furthermore, the information storage device Y140 may be located within the information processing device 1. Any method may be used as long as the information storage device Y140 can read and write predetermined information.

[0067] In the first embodiment, the distance between the surfaces and the normals are used to match the three-dimensional reference surface information with the three-dimensional surface information generated based on the three-dimensional point group. The present invention is not limited to this method as long as it can determine that the three-dimensional reference surface information and the three-dimensional surface information represent the same surface in real space. For example, any method can be used, such as the distance between the vertex coordinates of the mesh or the number of common voxels through which both surfaces pass when the space is sliced ​​into voxels. As described above, if the surface is a curved surface, it may be determined whether or not the parameters representing the shape of the curved surface, such as the curvature, are similar to or greater than a predetermined value.

[0068] Furthermore, the present invention may be configured such that the three-dimensional surface information calculation unit calculates a plurality of pieces of three-dimensional surface information, and merges a plurality of pieces of surface information that match the three-dimensional surface information. Specifically, when a plurality of surfaces are calculated as described in M40 of FIG. 4, the three-dimensional reference surface information is used to determine that the surfaces are the same, and the surfaces are integrated. In this way, by integrating three-dimensional surfaces calculated as a plurality of surfaces due to measurement errors, it is possible to integrate three-dimensional surface information divided into a plurality of local planes, and generate three-dimensional surface information that is held as a wide surface. The present invention may be configured such that the space is divided into subspaces such as voxels in advance, and a plane calculated from a three-dimensional point group included in the subspace is determined to be the same surface as the three-dimensional reference surface information for each subspace as described above.

[0069] The present invention may be configured to generate a three-dimensional mesh based on a three-dimensional point group in advance, and smooth these as a single plane. Specifically, the three-dimensional plane is created by selecting three points that are in contact with a sphere of a predetermined radius from the three-dimensional point group to create a triangular mesh. The three-dimensional mesh is created by a ball pivoting method in which the sphere is rolled so as to include two of the points, and the triangle including the newly contacted point is meshed repeatedly. The mesh created in this way has irregularities, but the mesh is then corrected using three-dimensional reference plane information so that the irregularities of the mesh determined to be in the same plane are reduced. Specifically, the vertices of the three-dimensional mesh are moved in the normal direction of the three-dimensional reference plane information so that the vertices of the three-dimensional mesh are on the same plane. In this way, the irregularities of the three-dimensional mesh can be reduced.

[0070] In step S105, whether or not to calculate three-dimensional surface information is determined based on whether or not the three-dimensional point group corresponds to a three-dimensional reference surface. The present invention may be configured not to calculate three-dimensional surface information if the three-dimensional reference surface is less than a predetermined length or area. This makes it possible to prevent the generation of localized minute surfaces.

[0071] The present invention may be configured such that three-dimensional reference plane information is input to the three-dimensional point group selection unit 103, and a three-dimensional point group whose distance from the three-dimensional reference plane information is equal to or less than a predetermined value is selected. Three-dimensional surface information is generated from the three-dimensional point group selected in this manner. That is, three-dimensional surface information is generated in which the positional deviation of the three-dimensional reference plane information is corrected by an actually measured three-dimensional point group. In this manner, the generation accuracy of the three-dimensional surface information is improved.

[0072] In the present invention, when generating three-dimensional surface information, three-dimensional surface information can be obtained by using the surface parameters of the three-dimensional reference surface information as initial values. Specifically, for example, a point-to-surface ICP algorithm is used to update parameters so that the three-dimensional reference surface information approaches the three-dimensional point group to calculate the three-dimensional surface information. ICP is an abbreviation for Iterative Closest Point. That is, the parameters of the reference surface information are used as the initial values ​​of the ICP. In addition, a method may be adopted in which voting is terminated and not adopted when the difference between the three-dimensional reference surface information and the distance or normal is greater than a predetermined value during the plane search of RANSAC. By using such three-dimensional reference surface information as initial values ​​or prior information, three-dimensional surface information can be obtained at high speed when there is no initial value. In addition, it is possible to suppress the generation of three-dimensional surface information whose parameters deviate from the three-dimensional reference surface information.

[0073] [Second embodiment] In the first embodiment, three-dimensional surface information is calculated using three-dimensional reference surface information held in a holding unit (not shown) and a three-dimensional point cloud measured by an imaging device (not shown). In the second embodiment, a configuration will be described in which an information processing device includes a configuration for calculating the three-dimensional reference surface information and the three-dimensional point cloud, and further includes a three-dimensional model generation system that corrects the generated three-dimensional surface information and holds it as a three-dimensional model.

[0074] FIG. 7 is a diagram for explaining the configuration of an apparatus according to the second embodiment of the present invention. FIG. 7 shows the functional configuration of an information processing apparatus 2. The information processing apparatus 2 in FIG. 7 corresponds to the information processing apparatus Y130 in FIG. 2. In this embodiment, the information processing apparatus 2 has a two-dimensional space information acquisition unit 201, a two-dimensional reference plane information calculation unit 202, and a three-dimensional reference plane information calculation unit 203 in addition to the configuration of the information processing apparatus 1 in the first embodiment. The information processing apparatus 2 further has a three-dimensional information measurement unit 204, a three-dimensional point group generation unit 205, a three-dimensional plane information correction unit 206, a three-dimensional model generation unit 207, and a storage unit 208. Each configuration of the information processing apparatus 2 shown in FIG. 7 is realized by executing a program by the CPU·H11.

[0075] The measuring device 21 is a 2D LiDAR. The measuring device 21 outputs measured scan information to the information processing device 2. The imaging device 22 is a depth sensor. The imaging device 22 outputs acquired depth information to the information processing device 2. In the second embodiment, the measuring device 21, the imaging device 22, and the information processing device 2 are disposed in a moving body U101.

[0076] The information processing device 2 calculates three-dimensional reference plane information based on scan information from the measuring device 21. The information processing device 2 calculates a three-dimensional point group based on depth information from the imaging device 22. The information processing device 2 also corrects the three-dimensional plane information calculated in the first embodiment to generate a three-dimensional model. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals and detailed description thereof will be omitted.

[0077] The two-dimensional space information acquisition unit 201 inputs scan information measured by the measurement device 21. The two-dimensional space information acquisition unit 201 generates two-dimensional space information, which is an occupied lattice map as a two-dimensional map, based on the input scan information. The two-dimensional space information acquisition unit 201 outputs the generated two-dimensional space information to the two-dimensional reference plane information calculation unit 202.

[0078] The two-dimensional reference plane information calculation unit 202 calculates two-dimensional reference plane information representing the surface of the structure based on the two-dimensional space information from the two-dimensional space information acquisition unit 201. The two-dimensional reference plane information calculation unit 202 outputs the calculated two-dimensional reference plane information to the three-dimensional reference plane information calculation unit 203.

[0079] The three-dimensional reference plane information calculation unit 203 calculates three-dimensional reference plane information by extending in the height direction the two-dimensional reference plane information from the two-dimensional reference plane information calculation unit 202. The three-dimensional reference plane information calculation unit 203 outputs the calculated three-dimensional reference plane information to the three-dimensional reference plane information input unit 101.

[0080] The three-dimensional information measuring unit 204 receives a depth image captured by the imaging device 22 as three-dimensional measurement information. The three-dimensional information measuring unit 204 outputs the received three-dimensional measurement information to the three-dimensional point cloud generating unit 205.

[0081] The three-dimensional point cloud generation unit 205 performs SLAM processing based on the three-dimensional measurement information from the three-dimensional information measurement unit 204, and generates a three-dimensional point cloud by integrating the three-dimensional measurement information acquired at multiple positions and orientations at multiple times. SLAM is an abbreviation for SIMULTANEOUS LOCALIZATION AND MAPPING. The three-dimensional point cloud generation unit 205 outputs the generated three-dimensional point cloud to the three-dimensional point cloud input unit 102 and the storage unit 208.

[0082] The three-dimensional surface information correction unit 206 corrects the three-dimensional surface information using the three-dimensional reference surface information from the three-dimensional reference surface information input unit 101, the three-dimensional point group from the three-dimensional point group selection unit 103, and the three-dimensional surface information from the three-dimensional surface information calculation unit 104. Here, the correction of the three-dimensional surface information means determining the height, width, hole, and contour shape not included in the three-dimensional reference surface information based on the three-dimensional point group, and reflecting them in the three-dimensional surface information. The three-dimensional surface information correction unit 206 outputs the corrected three-dimensional surface information to the three-dimensional model generation unit 207.

[0083] The three-dimensional model generating unit 207 associates the three-dimensional surface information from the three-dimensional surface information correction unit 206 with the three-dimensional point group held by the holding unit 208 and holds it in the holding unit 208. The association means linking which point group among the three-dimensional point groups held by the holding unit 208 corresponds to the three-dimensional surface information. The association is used to select a display method, such as whether to display the three-dimensional surface information, display the three-dimensional point group, or not display the three-dimensional point group associated with the three-dimensional surface information among the three-dimensional point groups, in a GUI described in a modified example below. The three-dimensional model generating unit 207 generates a three-dimensional model by this association.

[0084] The storage unit 208 stores the three-dimensional model generated by the three-dimensional model generation unit 207. The three-dimensional model includes a three-dimensional point group and three-dimensional surface information.

[0085] Fig. 8 is a flowchart showing the flow of processing of the device according to the second embodiment of the present invention. Fig. 8 is a flowchart explaining the operation of the information processing device 2. The processing explained in Fig. 8 starts when the information processing device 2 is started up as a result of powering on the information processing device 2 and an instruction to start the processing of generating a three-dimensional model is received.

[0086] The second embodiment differs from the first embodiment in that steps S201, S202, S203, S204, S205, S206, and S207 are added. Step S201 is a two-dimensional space information acquisition process. Step S202 is a two-dimensional reference plane information calculation process. Step S203 is a three-dimensional reference plane information calculation process. Step S204 is a three-dimensional information measurement process. Step S205 is a three-dimensional point cloud generation process. Step S206 is a three-dimensional plane information correction process. Step S207 is a three-dimensional model generation process. In the following explanation, explanations of processes similar to those in the first embodiment will be omitted.

[0087] In step S201, the two-dimensional space information acquisition unit 201 performs scan matching on the scan information measured by the measurement device 21 using the method described in Non-Patent Document 2, and generates two-dimensional space information that is a two-dimensional map (occupancy grid map). Then, the process of step S202 is executed.

[0088] In step S202, the two-dimensional reference plane information calculation unit 202 calculates two-dimensional reference plane information based on the two-dimensional space information generated in step S201. Specifically, the two-dimensional reference plane information calculation unit 202 performs straight line fitting on the wall surface of the occupancy grid map, and calculates line segments of the wall surface that correspond to planes on the occupancy grid map. The two-dimensional reference plane information calculation unit 202 holds the coordinates of the endpoints of the calculated line segments as two-dimensional reference plane information. Next, the process of step S203 is executed.

[0089] In step S203, the three-dimensional reference plane information calculation unit 203 calculates three-dimensional reference plane information by extending the two-dimensional reference plane information calculated in step S202 in the height direction. Specifically, the three-dimensional reference plane information calculation unit 203 converts the two-dimensional coordinates (X, Y) of the coordinates of the end points of the line segment held as the two-dimensional reference plane information into three-dimensional coordinates (X, Y, 0) and (X, Y, H) to which height is added. Here, H is a parameter of the height of the wall surface, and is assumed to be input in advance as a setting parameter. As the parameter H of the height of the wall surface, for example, the height from the floor to the ceiling of the room is set. In this way, the three-dimensional reference plane information calculation unit 203 calculates the line segment, which is the two-dimensional reference plane, as three-dimensional reference plane information including the height direction. Then, the process of step S204 is executed.

[0090] In step S204, the three-dimensional information measuring unit 204 inputs, as three-dimensional measurement information, a depth image captured by the imaging device 22. Then, the process of step S205 is executed.

[0091] In step S205, the three-dimensional point cloud generating unit 205 executes SLAM on the three-dimensional measurement information input in step S204 to generate a three-dimensional point cloud. The three-dimensional point cloud generating unit 205 converts the depth information (depth information) obtained based on the depth image into a world coordinate system using the camera position coordinates calculated using the method described in Non-Patent Document 1, and generates a three-dimensional point cloud by integrating multiple viewpoints. Specifically, integrating multiple viewpoints means expressing the coordinates of each point based on a common coordinate system. Then, the process of step S102 is executed.

[0092] In step S206, the three-dimensional surface information correction unit 206 corrects the three-dimensional surface information calculated in step S106. In step S106, the three-dimensional surface information calculation unit 104 calculates a circumscribing rectangle surrounding the three-dimensional point group as three-dimensional surface information, but in step S206, the three-dimensional surface information correction unit 206 further corrects the height, width, hole, and contour shape using the three-dimensional point group. Specifically, regarding the height, width, and contour, the three-dimensional point group projected onto the plane calculated based on the three-dimensional point group is first meshed finer. For meshing, a ball pivoting method is used. In the above-mentioned ball pivoting, a hole shape is expressed by not generating a mesh for an area in which three points are not included in a circle of a predetermined radius. In this way, the height, width, hole, and contour shape of the three-dimensional surface information are corrected. Then, the process of step S107 is executed.

[0093] In step S107, a determination is made as to whether or not the calculation of the three-dimensional surface information is to be completed, as described in the first embodiment. If the calculation is to be continued, the process returns to step S104, and if the calculation is to be completed, the process proceeds to step S207.

[0094] In step S207, the three-dimensional model generating unit 207 generates a three-dimensional model using the three-dimensional surface information corrected in step S206 and the three-dimensional point group in the storage unit 208. Specifically, the three-dimensional model generating unit 207 first assigns an ID to each piece of three-dimensional surface information. Next, the three-dimensional model generating unit 207 extracts a three-dimensional point group that is less than a predetermined distance for each piece of three-dimensional surface information, and assigns an ID of the three-dimensional surface information corresponding to the extracted three-dimensional point group. By linking the three-dimensional point group with the ID in this way, the three-dimensional model generating unit 207 assigns information on which three-dimensional surface information it is related to and which three-dimensional surface information it is not related to for each three-dimensional point group. In this way, the three-dimensional model generating unit 207 generates a three-dimensional model including the three-dimensional point group to which the association with the three-dimensional surface information is assigned. Then, the process of step S108 is executed.

[0095] <Effects> As described above, the information processing device 2 of the second embodiment generates three-dimensional reference plane information based on two-dimensional space information generated based on measurement information measured by the LiDAR. The information processing device 2 also generates a three-dimensional point cloud based on imaging information measured by an imaging device, and generates three-dimensional plane information using these. The information processing device 2 also corrects the generated three-dimensional plane information. In this way, the information processing device 2 can generate a three-dimensional model that appropriately uses plane information by a sensor arranged on the moving body U101. In addition, according to the information processing device 2, the reproduction accuracy of shapes such as height, width, holes, and contours of the three-dimensional plane information is also improved. In addition, according to the information processing device 2, an ID of the three-dimensional plane information is assigned to the three-dimensional point cloud. As a result, when a three-dimensional model is displayed on a GUI described later, the three-dimensional plane information and the three-dimensional point cloud related thereto can be switched and displayed as necessary, making it easier for the user to grasp the surrounding space.

[0096] <Variation 2-1> In the second embodiment, the information processing device 2 includes all of the components for acquiring two-dimensional spatial information, measuring three-dimensional information, correcting three-dimensional surface information, and generating a three-dimensional model. However, the present invention is not limited to this, and it is not essential that the information processing device 2 includes all of the components, and the information processing device 2 may not include some of the components depending on the application. Furthermore, as long as the information processing device 2 can generate three-dimensional surface information using the three-dimensional point group and the three-dimensional reference surface information, the above components may be arranged externally and the information processing device 2 may acquire the processing results from the outside as necessary.

[0097] The three-dimensional surface information correction unit can also be configured to expand, complement, or shave off unmeasured areas of the three-dimensional point cloud using three-dimensional reference surface information. Specifically, it is the complement of areas such as mirror surfaces and glass. For example, an image from an RGB camera is input, and extracted using semantic area segmentation of the image such as FCN that has been trained to recognize and output mirror surfaces and mirror areas, and the three-dimensional surface information is filled in for the corresponding unmeasured areas of the three-dimensional point cloud using the three-dimensional reference surface information. FCN is an abbreviation for FULLY CONVOLUTIONAL NETWORKS. Semantic area segmentation is called semantic segmentation. In this way, it is also possible to fill in hole areas where three-dimensional surface information could not be generated. In addition, it is also possible to configure the three-dimensional surface information to be expanded using three-dimensional reference surface information so as to complement three-dimensional point clouds that are outside the field of view of the depth sensor and cannot be measured, not limited to mirror surfaces and mirror areas.

[0098] A configuration can also be realized in which mesh smoothing is performed using three-dimensional reference plane information for a mesh model in which mesh generation such as ball pivoting has been performed in advance for a three-dimensional point group. Specifically, the vertices of the three-dimensional mesh are moved with respect to the average position of the three-dimensional point group in the normal direction of the three-dimensional reference plane information. In this way, three-dimensional surface information can be generated for the three-dimensional mesh by appropriately using surface information. Specifically, a configuration can be realized in which the unevenness of the three-dimensional surface information is reduced. Note that a configuration may be adopted in which the flatness is calculated as the unevenness of the mesh, and the above-mentioned smoothing is performed if it is equal to or greater than a predetermined value. Note that the flatness may be anything that represents the flatness of the three-dimensional surface information, such as the variance of the three-dimensional point group in the normal direction of the plane obtained from the three-dimensional point group, or the variance of the normal of the mesh. In this way, by determining whether or not to perform smoothing according to the flatness, three-dimensional surface information with low flatness can be smoothed with higher priority. That is, the three-dimensional reference plane information is corrected when the difference in flatness between the three-dimensional reference plane information and the three-dimensional surface information is greater than a predetermined value as a difference in shape. The difference in shape may be any parameter that allows for comparing the shapes of the three-dimensional reference plane information and the three-dimensional plane information, such as height, width, contour length, contour length of holes included in the plane, or area.

[0099] Furthermore, an image captured by the color camera Y100 may be pasted onto the three-dimensional surface information as a texture. By displaying the three-dimensional surface information thus generated as a three-dimensional model on a display unit (not shown), the user can more intuitively grasp the surrounding space. Instead of a texture, a color attribute may be given to each vertex of the mesh.

[0100] In the second embodiment, the wall surface has been described, but the present invention can also be applied to the floor surface. That is, based on the prior information that the space in which the moving object in the two-dimensional space can move is flat, three-dimensional reference plane information representing the floor surface is calculated. By matching the three-dimensional reference plane information of the floor surface calculated in this way with the three-dimensional point cloud obtained by measuring the floor surface using the method of the second embodiment, three-dimensional plane information of a floor with few irregularities can be generated. The ceiling surface can also be generated in a similar manner.

[0101] In the second embodiment, the measuring device and the imaging device are configured to execute processing on input data acquired online, but the data may be stored in a recording device (not shown) and the processing of the second embodiment may be executed offline. An information processing device may be installed on a network, and each time predetermined input data is acquired, the data is transmitted to the information processing device and the processing of the second embodiment is executed. In this way, it is not necessary to execute processing on the mobile object U101, and it is not necessary to place a high-performance computer on the mobile object, and it is possible to reduce battery consumption of the mobile object U101.

[0102] The three-dimensional point cloud may be further corrected based on the three-dimensional surface information generated in the second embodiment. Specifically, the coordinates obtained by projecting the three-dimensional point cloud related to the three-dimensional surface information in the normal direction of the calculated three-dimensional surface information are set as new three-dimensional point cloud coordinates. In this way, a three-dimensional point cloud in real space that reproduces a continuous surface can be generated. When a three-dimensional model is held in a voxel representation, 3D OCCUPANCY GRID MAP, TSDF, or other representation as described above, these can also be corrected in the same way to generate a three-dimensional model in real space that reproduces a continuous surface.

[0103] <Variation 2-2> As Modification 2-2, the GUI of the device according to the second embodiment of the present invention will be described. Fig. 9 is a diagram showing an example of the GUI of the device according to the second embodiment of the present invention. GUI G10 shown in Fig. 9 is an example of display information presented by the display device according to the second embodiment of the present invention.

[0104] GUI G10 shows an example of a GUI for remotely controlling a moving body U101, a three-dimensional model of the real space explained in the second embodiment, the position and attitude of the moving body, and operation information.

[0105] G101 is an emergency stop button that allows the user to stop the movement of the moving object U101 by touching the emergency stop button G101 with a finger.

[0106] G102 is a cursor. The cursor G102 can be moved in response to a user's touch operation via a mouse, controller, or touch panel (not shown), and buttons and check boxes in the GUI can be operated by pressing a button pointed to by the cursor G102.

[0107] G103 is an example of a controller for a moving object U101. The user can move the circle inside the controller G103 up, down, left, and right to move the moving object U101 forward, backward, left, and right according to the input. The moving object U101 is, for example, an AMR.

[0108] G104 is an example showing the internal state of the moving object U101. In Fig. 9, the moving object U101 is shown as being in auto-run mode and moving at a speed of 0.5 m / sec.

[0109] G105 is a button for resetting the autopilot and operational settings.

[0110] G106 is a GUI for generating a 3D model of the real world and setting display information. With G106, it is possible to select whether or not to generate a 3D model by using a check box. Also, with G106, it is possible to select whether or not to display each element, such as a 3D point group, 3D surface information, 2D reference surface information, and 3D reference surface information, by using a check box.

[0111] G107 is an example of displaying operation information of a mobile unit U101. G107 displays the current position of the mobile unit U101 and the destination of the mobile unit.

[0112] G110 is a window that presents the three-dimensional model described in the second embodiment. In Fig. 9, the three-dimensional point group G111, the two-dimensional reference plane information G112, and the three-dimensional plane information G114 set in G106 described above are presented. In Fig. 9, a three-dimensional model G113 of the moving body U101 is also presented based on the position and posture of the moving body U101. Furthermore, the user can operate the viewpoint of the three-dimensional model by moving a finger up, down, left, or right while touching G110.

[0113] 9, by presenting a three-dimensional model of the real space, the user can intuitively and remotely control the mobile body U101 while recognizing the space around the mobile body U101. In particular, by presenting the three-dimensional surface information generated based on the three-dimensional reference surface information and the three-dimensional point cloud described in the second embodiment, the user can more intuitively control the mobile body U101 to avoid contact with obstacles when traveling in narrow passages or along walls, which are difficult to distinguish using the point cloud.

[0114] Furthermore, the GUI of the present invention may be configured to present the 3D reference plane information, 3D plane information, 3D point group, and their clusters in different colors, allowing the user to more intuitively grasp the surrounding space.

[0115] <Modification 2-3> Furthermore, in the second embodiment, an information processing device can be configured as a mobile body system having a control unit that controls the motor controllers Y110 and Y120 described in Fig. 2. In this case, the three-dimensional model generated in the second embodiment can be generated as a 3D OCCUPANCY GRID MAP by applying it to three-dimensional voxels, and this can be used to generate a route for the autonomous movement of the mobile body U101. In this case, by appropriately using surface information and using a hybrid three-dimensional model with an actually measured three-dimensional point cloud, it becomes possible to autonomously control the mobile body U101 with higher accuracy.

[0116] It can also be configured as a management system that presents a three-dimensional model to a user in a control room that manages the autonomously moving vehicle U101 and visualizes the autonomous movement state. In this way, the user can more intuitively grasp the position and posture of the vehicle in three-dimensional space.

[0117] <Variation 2-4> In the second embodiment, the measuring device 21, the imaging device 22, and the information processing device 2 are arranged on the moving body U101. The present invention is not limited to arranging these components on the moving body U101 as long as they are configured to generate a three-dimensional model, but may be configured to be carried by a person as a handheld measurement system. In addition, the present invention may be configured, for example, to install the measuring device 21 and the imaging device 22 at a predetermined location. In this case, if the imaging device 22 can measure the distance from the installation position to the surrounding structures, such as a 3D LiDAR, the three-dimensional point cloud generation unit can generate a three-dimensional model without calculating the position and orientation of the imaging device 22.

[0118] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0119] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0120] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a three-dimensional reference plane information input means for inputting three-dimensional reference plane information obtained from information expressing a real space on a two-dimensional plane or three-dimensional information expressing the real space; a three-dimensional point cloud input means for inputting three-dimensional point cloud information obtained by measuring the real space; a three-dimensional point cloud selection means for selecting, from the three-dimensional point cloud, a three-dimensional point cloud relating to a specific surface included in the real space; a three-dimensional surface information calculation means for calculating three-dimensional surface information representing a shape of the real space based on the three-dimensional reference surface information and the selected three-dimensional point group; 13. An information processing device comprising: (Configuration 2) the three-dimensional point cloud selection means selects the three-dimensional point cloud by performing surface-related clustering on the three-dimensional point cloud; the three-dimensional surface information calculation means calculates, based on the selected three-dimensional point group, three-dimensional surface information in which at least one of a distance to the three-dimensional reference surface information, a normal, and a curvature is equal to or smaller than a predetermined value; 2. The information processing device according to configuration 1. (Configuration 3) The three-dimensional point cloud selection means divides a space into a predetermined subspace, and selects the three-dimensional point cloud within the subspace; the three-dimensional surface information calculation means calculates three-dimensional surface information for each of the subspaces, and further generates three-dimensional surface information by integrating three-dimensional surface information in which at least one of a distance to the reference surface information, a normal, and a curvature is equal to or less than a predetermined value. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The three-dimensional point group selection means selects a three-dimensional point group within a predetermined distance from the three-dimensional reference plane information, the three-dimensional surface information calculation means calculates three-dimensional surface information based on the selected three-dimensional point group; 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) a two-dimensional space information acquisition means for acquiring two-dimensional space information, which is information on a structure in real space projected into two dimensions; a two-dimensional reference plane information calculation means for calculating, from the two-dimensional spatial information, two-dimensional reference plane information representing a boundary between a space in which the moving object moves and the structure; a three-dimensional reference plane information calculation means for calculating the three-dimensional reference plane information by extending the two-dimensional reference plane information in a height direction; 5. The information processing device according to any one of configurations 1 to 4, further comprising: (Configuration 6) The two-dimensional space information is generated based on a point cloud measured on a two-dimensional plane in real space. 6. The information processing device according to configuration 5. (Configuration 7) The two-dimensional space information is information of a structure generated based on CAD data or a drawing including the shape of the structure in real space. 7. The information processing device according to configuration 5 or 6. (Configuration 8) a three-dimensional surface information correcting means for correcting the three-dimensional surface information based on the three-dimensional reference surface information when a difference in shape between the three-dimensional surface information and the three-dimensional reference surface information is greater than a predetermined value; 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The shape refers to the flatness of the three-dimensional surface information, and when a difference in flatness between the three-dimensional surface information and the three-dimensional reference surface information is greater than a predetermined value, the three-dimensional surface information is smoothed using the three-dimensional reference surface information. 9. The information processing device according to configuration 8. (Configuration 10) The shape refers to at least one of the height, width, hole, and contour of the three-dimensional surface information, When a difference between a shape of the three-dimensional surface information and a shape of the three-dimensional surface information is greater than a predetermined value, the three-dimensional surface information is complemented or truncated based on the shape of the three-dimensional reference surface information. 9. The information processing device according to configuration 8. (Configuration 11) a three-dimensional information calculation means for inputting a captured image of a real space captured by an imaging device and calculating three-dimensional information of the real space depicted in the captured image; a three-dimensional point cloud generating means for calculating a position and an orientation of an imaging device using the captured image, and generating the three-dimensional point cloud in real space by integrating the three-dimensional information calculated from a plurality of viewpoints using the position and orientation; 11. The information processing device according to any one of configurations 1 to 10, further comprising: (Method 1) a three-dimensional reference plane information input step of inputting three-dimensional reference plane information obtained from information representing a real space on a two-dimensional plane or three-dimensional information representing the real space; a three-dimensional point cloud input step of inputting three-dimensional point cloud information obtained by measuring the real space; a three-dimensional point cloud selection step of selecting, from the three-dimensional point cloud, a three-dimensional point cloud relating to a specific surface included in the real space; a three-dimensional surface information calculation step of calculating three-dimensional surface information representing a shape of the real space based on the three-dimensional reference surface information and the selected three-dimensional point group; 13. A method for controlling an information processing apparatus comprising the steps of: (Program 1) Computer, a three-dimensional reference plane information input means for inputting three-dimensional reference plane information obtained from information expressing a real space on a two-dimensional plane or three-dimensional information expressing the real space; a three-dimensional point cloud input means for inputting three-dimensional point cloud information obtained by measuring the real space; a three-dimensional point cloud selection means for selecting, from the three-dimensional point cloud, a three-dimensional point cloud relating to a specific surface included in the real space; and a three-dimensional surface information calculation means for calculating three-dimensional surface information representing a shape of the real space based on the three-dimensional reference surface information and the selected three-dimensional point group; A program characterized by causing the program to function as a [Explanation of symbols]

[0121] U101: Mobile Y130: Information processing equipment

Claims

1. a three-dimensional reference plane information input means for inputting three-dimensional reference plane information obtained from information expressing a real space on a two-dimensional plane or three-dimensional information expressing the real space; a three-dimensional point cloud input means for inputting three-dimensional point cloud information obtained by measuring the real space; a three-dimensional point cloud selection means for selecting, from the three-dimensional point cloud, a three-dimensional point cloud relating to a specific surface included in the real space; a three-dimensional surface information calculation means for calculating three-dimensional surface information representing a shape of the real space based on the three-dimensional reference surface information and the selected three-dimensional point group; 13. An information processing device comprising:

2. the three-dimensional point cloud selection means selects the three-dimensional point cloud by performing surface-related clustering on the three-dimensional point cloud; the three-dimensional surface information calculation means calculates, based on the selected three-dimensional point group, three-dimensional surface information in which at least one of a distance to the three-dimensional reference surface information, a normal, and a curvature is equal to or smaller than a predetermined value; 2. The information processing apparatus according to claim 1,

3. The three-dimensional point cloud selection means divides a space into a predetermined subspace, and selects the three-dimensional point cloud within the subspace; the three-dimensional surface information calculation means calculates three-dimensional surface information for each of the subspaces, and further generates three-dimensional surface information by integrating three-dimensional surface information in which at least one of a distance to the reference surface information, a normal, and a curvature is equal to or smaller than a predetermined value.

2. The information processing apparatus according to claim 1,

4. The three-dimensional point group selection means selects a three-dimensional point group within a predetermined distance from the three-dimensional reference plane information, the three-dimensional surface information calculation means calculates three-dimensional surface information based on the selected three-dimensional point group; 2. The information processing apparatus according to claim 1,

5. a two-dimensional space information acquisition means for acquiring two-dimensional space information, which is information on a structure in real space projected into two dimensions; a two-dimensional reference plane information calculation means for calculating, from the two-dimensional spatial information, two-dimensional reference plane information representing a boundary between a space in which the moving object moves and the structure; a three-dimensional reference plane information calculation means for calculating the three-dimensional reference plane information by extending the two-dimensional reference plane information in a height direction; 2. The information processing apparatus according to claim 1, further comprising:

6. The two-dimensional space information is generated based on a point cloud measured on a two-dimensional plane in real space.

6. The information processing apparatus according to claim 5,

7. The two-dimensional space information is information of a structure generated based on CAD data or a drawing including a shape of a structure in real space.

6. The information processing apparatus according to claim 5,

8. a three-dimensional surface information correcting means for correcting the three-dimensional surface information based on the three-dimensional reference surface information when a difference in shape between the three-dimensional surface information and the three-dimensional reference surface information is greater than a predetermined value; 2. The information processing apparatus according to claim 1,

9. The shape refers to the flatness of the three-dimensional surface information, and when a difference in flatness between the three-dimensional surface information and the three-dimensional reference surface information is greater than a predetermined value, the three-dimensional surface information is smoothed using the three-dimensional reference surface information.

9. The information processing apparatus according to claim 8,

10. The shape refers to at least one of the height, width, hole, and contour of the three-dimensional surface information, and when a difference between the shape of the three-dimensional surface information and the shape of the three-dimensional surface information is greater than a predetermined value, the three-dimensional surface information is supplemented or truncated based on the shape of the three-dimensional reference surface information.

9. The information processing apparatus according to claim 8,

11. a three-dimensional information calculation means for inputting a captured image of a real space captured by an imaging device and calculating three-dimensional information of the real space depicted in the captured image; a three-dimensional point cloud generating means for calculating a position and an orientation of an imaging device using the captured image, and generating the three-dimensional point cloud in real space by integrating the three-dimensional information calculated from a plurality of viewpoints using the position and orientation; 2. The information processing apparatus according to claim 1, further comprising:

12. a three-dimensional reference plane information input step of inputting three-dimensional reference plane information obtained from information representing a real space on a two-dimensional plane or three-dimensional information representing the real space; a three-dimensional point cloud input step of inputting three-dimensional point cloud information obtained by measuring the real space; a three-dimensional point cloud selection step of selecting, from the three-dimensional point cloud, a three-dimensional point cloud relating to a specific surface included in the real space; a three-dimensional surface information calculation step of calculating three-dimensional surface information representing a shape of the real space based on the three-dimensional reference surface information and the selected three-dimensional point group; 13. A method for controlling an information processing apparatus comprising the steps of:

13. Computer, a three-dimensional reference plane information input means for inputting three-dimensional reference plane information obtained from information expressing a real space on a two-dimensional plane or three-dimensional information expressing the real space; a three-dimensional point cloud input means for inputting three-dimensional point cloud information obtained by measuring the real space; a three-dimensional point cloud selection means for selecting, from the three-dimensional point cloud, a three-dimensional point cloud relating to a specific surface included in the real space; and a three-dimensional surface information calculation means for calculating three-dimensional surface information representing a shape of the real space based on the three-dimensional reference surface information and the selected three-dimensional point group; A program characterized by causing the program to function as a

Images