Information processing device, method, and program

Abstract

To generate, in a short time, a three-dimensional model for recognizing the state of a surrounding space including a distance.SOLUTION: An information processing device comprises: two-dimensional reference surface information input means for inputting information on a surface in a two-dimensionally inputted space as two-dimensional reference surface information; three-dimensional reference surface generation means for generating a three-dimensional reference surface on the basis of the two-dimensional reference surface information; captured image input means for inputting a captured image in which a space is captured by an imaging device; position attitude input means for inputting a position and attitude of the imaging device when the captured image was captured; coordinate calculation means for calculating coordinates on the three-dimensional reference surface corresponding to coordinates in the captured image by using the position and attribute of the imaging device; and texture information generation means for generating texture information for the three-dimensional reference surface on the basis of the coordinates calculated by the coordinate calculation means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] There is a demand for creating maps for autonomous movement of moving objects such as trackless AGVs (Automated Guided Vehicles) and AMRs (Autonomous Mobile Robots) while remotely operating the moving objects. AGV is an abbreviation for Automated Guided Vehicle. AMR is an abbreviation for Autonomous Mobile Robot. When remotely operating a moving object, maps can be created efficiently by operating the object while visually checking the space around it using color images, etc.

[0003] Meanwhile, a technique has been known that uses a device capable of capturing depth images and color images to generate a three-dimensional model for the purpose of grasping the state of the surrounding space.

[0004] Patent Document 1 discloses a technology for reconstructing the surrounding space as a three-dimensional model using depth images and color images, and creating the three-dimensional model. During remote operation, images with different viewpoints are generated and displayed based on the three-dimensional model, allowing the user to check the state of the surrounding space, and thus grasp the next destination of the moving object and information on obstacles. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5906258 [Non-patent literature]

[0006] [Non-Patent Document 1] M. Labbe et al., “RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operation,” Journal of Field Robotics, 2019 [Non-Patent Document 2] Grisetti et al., Improved Techniques for Grid Mapping with Rao-Blackwellized Particle Filters, Trans on Robotics 23.1(2007)) [Non-Patent Document 3] Carlos et al, ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM, IEEE Trans. Robotics, 2021. Summary of the Invention [Problem to be solved by the invention]

[0007] In the technology of Patent Document 1, depending on the size of the space for which the 3D map model is to be generated, it may take a long time to generate a 3D map model for recognizing the state of the surrounding space.

[0008] The present invention has been made in consideration of the above problems. The object of the present invention is to generate a three-dimensional model in a short time period in order to recognize the state of the surrounding space, including distant locations. [Means for solving the problem]

[0009] An information processing device according to one embodiment of the present invention has a two-dimensional reference plane information input means for inputting information about a surface in a space projected into two dimensions as two-dimensional reference plane information, a three-dimensional reference plane generation means for generating a three-dimensional reference plane based on the two-dimensional reference plane information, an image input means for inputting an image of a space captured by an imaging device, a position and orientation input means for inputting the position and orientation of the imaging device when the image was captured, a coordinate calculation means for calculating coordinates on the three-dimensional reference plane corresponding to coordinates on the captured image using the position and orientation of the imaging device, and a texture information generation means for generating texture information for the three-dimensional reference plane based on the coordinates calculated by the coordinate calculation means. Effect of the Invention

[0010] According to the present invention, a three-dimensional model of a real space, including distant objects, can be generated in a short period of time, allowing a user to quickly recognize the state of the space. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a situation in which the device according to the first embodiment of the present invention is used. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of a moving body according to a first embodiment of the present invention. [Diagram 3] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment of the present invention. [Figure 4] FIG. 1 is a diagram showing a concept of processing according to a first embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram illustrating a configuration of an apparatus according to a first embodiment of the present invention. [Figure 6] 4 is a flowchart showing a process flow of the device according to the first embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing a concept of processing according to a second embodiment of the present invention. [Figure 8] FIG. 4 is a diagram illustrating the configuration of an apparatus according to a second embodiment of the present invention. [Figure 9]10 is a flowchart showing a process flow of an apparatus according to a second embodiment of the present invention. [Figure 10] FIG. 13 is a diagram illustrating the configuration of an apparatus according to a third embodiment of the present invention. [Figure 11] 13 is a flowchart showing a process flow of an apparatus according to a third embodiment of the present invention. [Figure 12] FIG. 13 is a diagram showing an example of a GUI of an apparatus according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present invention will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same or similar configurations are denoted by the same reference numerals, and duplicated explanations are omitted.

[0013] [First embodiment] A first embodiment of the invention involves the generation of a three-dimensional model of a space, also called a three-dimensional map or cartography.

[0014] Fig. 1 is a diagram for explaining a situation in which the device according to the first embodiment of the present invention is used. Fig. 1 is a diagram for explaining a use case in which a three-dimensional model generated by the device according to the first embodiment of the present invention is used.

[0015] A space U10 indicates a real space to which an information processing device according to a first embodiment of the present invention is applied. A moving body U101 is a moving body that moves in the space U10. A sensor is arranged in the moving body U101, and information on surrounding structures and position information and posture information of the moving body U101 can be obtained based on the measurement values ​​of this sensor. The structure information refers to the arrangement and shape of structures such as a wall U102 that is a wall in the space U10, a pillar U103 that is a column in the space U10, and fixtures (not shown). The position information and posture information of the moving body U101 represent the position and orientation of the moving body in the space, and are represented by three parameters: two-dimensional position coordinates (X, Y) and direction (θ).

[0016] The space U20 is a space where a user U201 is present. The user U201 is in a remote location different from the space where the moving object U101 is located. FIG. 1 shows a state where a three-dimensional model generated using the first embodiment of the present invention is displayed on a tablet U202 operated by the user U201, and the moving object U101 is remotely controlled. The three-dimensional model of the space generated using the first embodiment of the present invention and the three-dimensional model of the moving object U101 are displayed on the GUI of the tablet U202. GUI is an abbreviation for Graphical User Interface. While looking at the GUI of the tablet U202, the user U201 remotely controls the moving object U101 so that the moving object U101 does not come into contact with or collide with a structure. The GUI of the tablet U202 will be described in detail later.

[0017] The mobile object U101 is, for example, an AMR such as a cleaning robot, an AGV, an autonomous vehicle, a drone, etc. AMR is an abbreviation for Autonomous Mobile Robot. AGV is an abbreviation for Automatic Guides Vehicle. In this embodiment, a case where an information processing device is arranged in an AMR will be described.

[0018] 2 is a diagram illustrating the configuration of a moving body U101 according to the first embodiment of the present invention. The moving body U101 includes an information processing device Y130. In this embodiment, a moving body system for generating a three-dimensional model of the surrounding space while moving the moving body U101 and controlling the moving body U101 will be described.

[0019] The information processing device Y130 is connected to various sensors such as an imaging device arranged on the moving body U101. Here, the various sensors arranged on the moving body U101 include a color camera Y100, a distance camera Y101, a stereo camera Y102, an IMU·Y103, and a LiDAR·Y104. IMU is an abbreviation for Inertial Measurement Unit. LiDAR is an abbreviation for Light Detection And Ranging.

[0020] The stereo camera Y102 captures a stereo image consisting of two monochrome images, and outputs the captured stereo image to the information processing device Y130.

[0021] The IMU·Y103 measures inertial information, such as acceleration and angular velocity, that occurs when the mobile object U101 moves, and outputs the measured inertial information to the information processing device Y130.

[0022] The LiDAR·Y104 outputs scan information measuring the distance from the moving body U101 to structures such as walls and fixtures present in the surrounding space to the information processing device Y130. Note that the LiDAR·Y104 in this embodiment is a 2D-LiDAR (two-dimensional LiDAR) that measures a point cloud at the same height.

[0023] The information processing device Y130 calculates the position and orientation of the moving body U101 in space by using the input stereo images, information from the IMU·Y103 and LiDAR·Y104, and information from the encoders Y112 and Y122 described below. In this embodiment, the information processing device Y130 calculates the position and orientation of the moving body U101 in space from multiple sensor information by the method of Labbe et al. (see Non-Patent Document 1).

[0024] The color camera Y100 captures a color image, which is used to add color information and texture information to a three-dimensional reference surface in space, which will be described later.

[0025] The distance camera Y101 captures a depth image (also called a depth image or distance image). The captured depth image is used to generate a three-dimensional reference plane, which will be described later. The distance camera Y101 is an example of a distance image measurement device.

[0026] Next, a method for controlling the moving object U101 will be described. The information processing device Y130 transmits drive signals to the motors Y111 and Y121 through the motor controllers Y110 and Y120 to rotate the motors Y111 and Y121. This causes the wheels Y113 and Y123 to rotate, and the moving object U101 moves.

[0027] The encoders Y112, Y122 detect the amount of actual rotation of the wheels Y113, Y123, and output rotation information to the information processing device Y130. The information processing device Y130 uses the input amount of rotation to calculate the position and orientation of the moving body U101 in space.

[0028] The tablet U202 is connected to the information processing device Y 130 via wireless communication, and presents information of the information processing device Y 130 to the user U201 through a GUI displayed on the screen of the tablet U202. The user remotely controls the moving object U101 using the presented GUI.

[0029] The information storage device Y140 holds the layout information of structures in a space and BIM information acquired in advance using the mobile body U101 as two-dimensional model information. BIM is an abbreviation for Building Information Modeling. The information processing device Y130 connects to the information storage device Y140 via wireless communication. The information processing device Y130 reads out the above-mentioned two-dimensional map information from the information storage device Y140 and uses it in the processing described below.

[0030] In this embodiment, an example will be described in which the information processing device Y 130 is disposed in the mobile body U 101. However, a part or the whole of the information processing device Y 130 can also be provided on a server connected via a network.

[0031] 3 is a diagram showing a hardware configuration of an information processing device according to a first embodiment of the present invention. The information processing device Y130 has a CPU H11, a ROM H12, a RAM H13, an external memory H14, an input unit H15, a display unit H16, a communication I / F H17, an I / O H18, and a system bus H21. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. I / F is an abbreviation for interface. I / O is an abbreviation for Input / Output.

[0032] The CPU H11 executes a control program, thereby carrying out the processing of this embodiment. The CPU H11 also controls various devices connected to the system bus H21. The ROM H12 stores the BIOS program and boot program. BIOS is an abbreviation for Basic Input Output System. The RAM H13 is used as the main storage device for the CPU H11. The external memory H14 stores the programs processed by the information processing device Y130 in this embodiment.

[0033] The input unit H15 performs processing related to input of information and the like from an input device such as a keyboard or a mouse. In this embodiment, the input device corresponds to the touch display of the tablet U202. The display unit H16 outputs the calculation results of the information processing device Y130 to a display device according to instructions from the CPU H11. In this embodiment, the display device corresponds to the display of the tablet U202. The display device displays a three-dimensional model generated by applying this embodiment in particular. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator.

[0034] The communication I / F H17 is a communication interface that communicates information via a network. The communication I / F H17 can be of any type, such as Ethernet, USB, serial communication, or wireless communication. USB is an abbreviation for Universal Serial Bus. A three-dimensional model is transmitted from the information processing device Y130 to the tablet U202 via the display unit H16 via the communication I / F H17. The I / O H18 inputs information from each measuring device, as well as images captured by a color camera Y100 disposed on a mobile object U101 (described later), to the information processing device Y130.

[0035] FIG. 4 is a diagram showing the concept of processing according to the first embodiment of the present invention, and is a diagram showing the concept of a three-dimensional model generation method. FIG. 4(A) is a diagram showing an example of a real space in which a three-dimensional model is generated. FIG. 4(B) is a diagram showing a secondary model in which the structure in the real space of FIG. 4(A) is projected two-dimensionally. FIG. 4(C) is a diagram explaining the concept of calculating a pseudo three-dimensional surface from two-dimensional reference surface information in the area surrounded by the dashed line M205 in M20 of FIG. 4(B). FIG. 4(D) is a diagram showing a state when texture information of a color image acquired by a color camera Y100 arranged on a moving body U101 is superimposed on a three-dimensional reference surface.

[0036] In Fig. 4(A), M10 indicates an example of a real space for generating a three-dimensional model in this embodiment. M101 is a wall, which is an example of a structure in the real space. M102 is a pillar, which is an example of a structure in the real space. M103 is outside the space in which the moving object moves (outside the measurement range). M104 is the space in which the moving object moves (within the measurement range).

[0037] In FIG. 4B, M20 indicates a two-dimensional model in which a structure is projected two-dimensionally. In this embodiment, a two-dimensional map is generated based on scan information obtained by scanning a real space based on a LiDAR sensor arranged on a moving body U101. The two-dimensional map is an occupancy grid map, and is generated by a method of Grisetti et al. (see Non-Patent Document 2). M201 indicates a boundary surface with a wall in the occupancy grid map. M202 indicates a boundary surface with a pillar in the occupancy grid map. M203 is an unmeasured area in the occupancy grid map, and is an area in which it is not yet determined whether an object is located in the space. M204 is a hollow area in the space, and indicates an area in which no object exists (i.e., an area in which a moving body can move). Hereinafter, in this embodiment, information indicating a boundary surface between a structure such as a wall M201 or a pillar M202 in the two-dimensional map and a space M204 in which no object exists is called two-dimensional reference surface information.

[0038] In FIG. 4(C), M30 shows a conceptual diagram of calculating a pseudo three-dimensional surface from two-dimensional reference plane information in an area surrounded by a dashed line M205 in M20. M301 shows the two-dimensional reference plane information of M201 three-dimensionally. As shown by the arrow M302, the three-dimensional surface shown in M303 can be calculated by extending the two-dimensional reference plane information in a direction perpendicular to the floor surface. In this embodiment, the three-dimensional surface calculated from the two-dimensional reference plane information is called three-dimensional reference plane information.

[0039] In FIG. 4(D), M40 is a diagram showing a state when texture information of a color image acquired by a color camera Y100 arranged on a moving body U101 is superimposed on a three-dimensional reference plane. When the information processing device Y130 calculates the position and orientation in space of the moving body U101 by the above-mentioned method, the information processing device Y130 also calculates the position and orientation in space of the color camera Y100. However, it is assumed that the relative position and orientation of the moving body U101 and the color camera Y100 have been measured in advance. M402 is a schematic diagram showing color image data captured by the color camera Y100 at a certain time and the direction of the color camera Y100 at that time. At this time, the information processing device Y130 calculates the correspondence between the three-dimensional coordinates on the three-dimensional reference plane and the coordinates of the color image, and superimposes the texture information by projecting the texture information on the three-dimensional reference plane. M401 is a schematic diagram when texture information is superimposed on a part of the three-dimensional reference plane. By performing this process while moving the mobile body U101, continuously photographing the surrounding space with color camera Y100 and projecting the texture information onto a three-dimensional reference surface, the user U201 can obtain the surrounding space as a three-dimensional model.

[0040] In this embodiment, the surface is assumed to be a plane. The two-dimensional reference surface information is stored as line segment information that stores the two-dimensional coordinates (X, Y) of points corresponding to the vertices of the surface. The three-dimensional reference surface information and the three-dimensional surface information are stored as mesh model data that stores the three-dimensional coordinates (X, Y, Z) of each vertex of the surface for each surface.

[0041] Fig. 5 is a diagram for explaining the configuration of an apparatus according to the first embodiment of the present invention. Fig. 5 shows the functional configuration of an information processing device Y130. In this embodiment, the information processing device Y130 has a two-dimensional reference plane information input unit 501 and a three-dimensional reference plane generation unit 502. The information processing device Y130 further has a measurement data input unit 503, a position and orientation input unit 504, a coordinate calculation unit 505, a captured image input unit 506, a texture information generation unit 507, and a storage unit 508. Each configuration in Fig. 5 is realized by executing a program in the CPU H11.

[0042] The two-dimensional reference plane information input unit 501 inputs two-dimensional reference plane information held in the information storage device Y140 to the information processing device Y130. The two-dimensional reference plane information input unit 501 outputs the input two-dimensional reference plane information to a three-dimensional reference plane generation unit 502 and a position and orientation input unit 504. The two-dimensional reference plane information may be reference plane information generated based on design data when a spatial structure or the like is designed.

[0043] The three-dimensional reference plane generating unit 502 generates three-dimensional reference plane information using the input two-dimensional reference plane information. The three-dimensional reference plane generating unit 502 outputs the generated three-dimensional reference plane information to the coordinate calculating unit 505.

[0044] A measurement data input unit 503 inputs measurement data, which is information from each measuring instrument (not shown), to the information processing device Y 130. The measurement data input unit 503 outputs the input measurement data to a position and orientation input unit 504.

[0045] The position and orientation input unit 504 uses the input measurement data to calculate the position and orientation in space of the moving body U101. Furthermore, the position and orientation input unit 504 also calculates the position and orientation in space of the color camera Y100 installed on the moving body U101, and outputs the calculated position and orientation information of the color camera Y100 to the coordinate calculation unit 505.

[0046] The coordinate calculation unit 505 performs correspondence between the coordinate data of the three-dimensional reference plane and the coordinate data of the color image based on the input three-dimensional reference plane information and the position and orientation information of the color camera Y100. Furthermore, the coordinate calculation unit 505 outputs the calculated correspondence information of the coordinate data to the texture information generation unit 507. At this time, specification characteristic information of the color camera Y100 is required for the correspondence of the coordinate data. For example, the angle of view, resolution, cell size, and internal parameters indicating lens distortion of the color camera Y100 correspond to the specification characteristic information of the color camera Y100. This specification characteristic information of the color camera Y100 is stored in a storage device (not shown). The coordinate calculation unit 505 performs correspondence of the coordinate data after acquiring this specification characteristic information from the storage device in advance.

[0047] The captured image input unit 506 inputs image data from a color camera Y100 (not shown) to the information processing device Y130. The captured image input unit 506 outputs the input image data to a texture information generating unit 507.

[0048] The texture information generating unit 507 performs a process of superimposing texture information onto the three-dimensional reference plane based on the correspondence information of the input coordinate data and the image data. When superimposing the texture information, the texture information generating unit 507 performs the process while interpolating information between pixels of the color image using a known technique such as bilinear interpolation or bicubic interpolation. This allows the texture information generating unit 507 to generate a three-dimensional reference plane to which texture information with reduced unnaturalness has been added.

[0049] Here, we will explain the weighting of texture information. For example, when considering the distance from the color camera Y100 to a structure, the closer the distance is, the denser the texture information of the structure becomes. For this reason, a color image taken at a position closer to the structure can capture the texture information of the structure more accurately. Therefore, using texture information taken when the color camera Y100 is closer to the structure makes it possible to generate a more detailed 3D model.

[0050] To achieve this, the texture information generating unit 507 temporarily stores the distance between the color camera Y100 and the structure when the texture is superimposed on the three-dimensional reference plane. After that, the texture information generating unit 507 compares the distance between the color camera Y100 and the three-dimensional reference plane corresponding to the captured color image and adopts the shorter distance as the texture information.

[0051] Here, the weighting factor is the distance between the color camera Y100 and the structure. In addition, in the present invention, the texture information generating unit 507 may adopt as texture information the vector of the optical axis of the color camera Y100 and the normal vector of the three-dimensional reference plane that are more parallel to each other. Furthermore, the texture information to be adopted by the texture information generating unit 507 may be determined by a combination of these geometric conditions. That is, the texture information generating unit 507 may weight the captured image based on the geometric relationship between the color camera Y100 (imaging device) and the three-dimensional reference plane. The geometric relationship here may be, for example, the distance between the three-dimensional reference plane and the imaging device, or the angle between the normal vector of the three-dimensional reference plane and the optical axis vector of the imaging device, or a combination of these.

[0052] Furthermore, for three-dimensional reference surfaces on which texture is not superimposed, the transparency of the three-dimensional reference surface may be increased to make the three-dimensional reference surface visually less visible to the user, thereby intentionally indicating that texture information is not superimposed. Alternatively, an artificial texture such as a lattice pattern may be superimposed on the three-dimensional reference surface on which texture is not superimposed to indicate that no texture is superimposed. This allows the user to visually confirm whether or not three-dimensional model information of the structure has been generated.

[0053] In the above-mentioned method, a pseudo three-dimensional surface is calculated from two-dimensional reference plane information in the three-dimensional reference plane generating unit 502. In addition, the present invention may reconstruct two-dimensional reference plane information based on a distance sensor disposed on the moving body U101.

[0054] The storage unit 508 stores data of the three-dimensional reference plane onto which the texture generated by the texture information generation unit 507 is superimposed.

[0055] Fig. 6 is a flowchart showing the flow of processing of the device according to the first embodiment of the present invention. Fig. 6 is a flowchart explaining the operation of the information processing device Y130. The processing explained in Fig. 6 is automatically started by starting up the information processing device Y130 when the power of the information processing device Y130 is turned on.

[0056] In step S600, the information processing device Y130 initializes each component within the information processing device Y130. That is, the CPU H11 reads a program from the external memory H14 and makes the information processing device Y130 operable. In addition, the CPU H11 reads program setting parameters and the like from the external memory H14 to the RAM H13 as necessary. Once a series of initialization processes is completed, the process of step S601 is executed.

[0057] In step S601, the two-dimensional reference plane information input unit 501 inputs two-dimensional reference plane information via the communication I / F H17. The two-dimensional reference plane information input unit 501 outputs the input two-dimensional reference plane information to the three-dimensional reference plane generation unit 502 and the position calculation unit 504. Then, the process of step S602 is executed.

[0058] In step S602, the three-dimensional reference plane generating unit 502 generates a three-dimensional reference plane. That is, the three-dimensional reference plane generating unit 502 generates a three-dimensional reference plane by extending the two-dimensional reference plane in a direction perpendicular to the floor based on the two-dimensional reference plane information input in step S601. At this time, the three-dimensional reference plane generating unit 502 determines the extension distance of the two-dimensional reference plane based on a preset distance, and generates a three-dimensional reference plane. The three-dimensional reference plane generating unit 502 outputs the generated three-dimensional reference plane to the coordinate calculation unit 505. Then, the process of step S603 is executed.

[0059] The three-dimensional reference plane generating unit 502 may have a three-dimensional reference plane dividing unit that divides the generated three-dimensional reference plane. The three-dimensional reference plane generating unit 502 may adjust the size of the model by excluding from the three-dimensional reference plane a reference plane that does not correspond to a position on the captured image, for example, among the three-dimensional reference planes divided by the three-dimensional reference plane dividing unit. The three-dimensional reference plane generating unit 502 may output the three-dimensional reference plane after the size adjustment to the coordinate calculation unit 505. Furthermore, the transparency of the reference plane that does not correspond to a position on the captured image, for example, among the three-dimensional reference planes divided by the three-dimensional reference plane dividing unit, may be increased when displayed.

[0060] The information processing device Y130 may have a distance image input unit that inputs a distance image that is an image measured by the distance camera Y101 and indicates the distance in space from the distance camera Y101 to the structure. The three-dimensional reference plane generating unit 502 may generate a measured three-dimensional reference plane (measured three-dimensional reference plane) based on the distance image. In this case, the three-dimensional reference plane generating unit 502 may integrate a three-dimensional reference plane generated by extending a two-dimensional reference plane in a direction perpendicular to the floor with the measured three-dimensional reference plane, and use the integrated three-dimensional reference plane as the three-dimensional reference plane used in the coordinate calculation unit 505 and the texture information generating unit 507. Here, integration may mean replacing the three-dimensional reference plane generated based on the two-dimensional reference plane information with the measured three-dimensional reference plane, and using the measured three-dimensional reference plane as the three-dimensional reference plane used in the coordinate calculation unit 505 and the texture information generating unit 507.

[0061] In step S603, the information processing device Y 130 selects the images input by the captured image input unit 506. The selection method is to sequentially select the input color images by a first-in first-out method. Then, the process of step S604 is executed.

[0062] In step S604, the coordinate calculation unit 505 associates the coordinates of the three-dimensional reference plane with the image coordinates. That is, the coordinate calculation unit 505 calculates the self-position and orientation of the color camera Y100 based on the data of each measuring instrument input by the measurement data input unit 503 via the I / O·H18 and the two-dimensional reference plane information. The coordinate calculation unit 505 associates the coordinates of the three-dimensional reference plane with the image coordinates based on the calculated positional relationship between the self-position and orientation of the color camera Y100 and the three-dimensional reference plane. After calculating the coordinate association information, the coordinate calculation unit 505 outputs the coordinate association information to the texture information generation unit 507. Next, the process of step S605 is executed.

[0063] In step S605, the texture information generating unit 507 superimposes texture information onto the three-dimensional reference plane. That is, the texture information generating unit 507 superimposes texture information onto the three-dimensional reference plane based on the above-mentioned coordinate correspondence information and the color image selected in step S603. Furthermore, information on the superimposed three-dimensional reference plane is held in the holding unit 508 as a three-dimensional model.

[0064] In step S606, the information processing device Y130 judges whether or not to end a series of texture information generation. Here, the judgment is made based on whether or not an image is input to the information processing device Y130 by the captured image input unit 506. That is, if an image is stored in the information processing device Y130, the process returns to step S603 again to select an image. On the other hand, if an image is not stored in the information processing device Y130, the process ends. Alternatively, if the series of texture information generation is sufficiently fast with respect to the cycle of the images input from the captured image input unit 506, a method is considered in which the user notifies the information processing device Y130 of a signal to end the process by some method. For example, a check box for selecting whether or not to generate texture information is prepared on the GUI displayed on the tablet U202, and the generation of texture information is ended by turning off the check in the check box.

[0065] The effects of the present invention are realized by the CPU·H11 executing a program that realizes the above-mentioned processing.

[0066] <Effects> As described above, in the first embodiment, three-dimensional reference plane information is calculated based on two-dimensional reference plane information acquired in advance. In addition, a three-dimensional model of the surrounding space can be generated by superimposing an image captured by a color camera on this three-dimensional reference plane. This allows the three-dimensional model of the surrounding space to be generated in a short time.

[0067] <Variation 1-1> In the first embodiment, the measuring devices Y100 to Y104 are disposed on the moving body U101. As long as the moving body U101 is equipped with a camera capable of capturing color images and a measuring instrument for measuring the position and orientation of the moving body U101, other measuring devices may be excluded from the moving body U101. Furthermore, in order to obtain surrounding spatial information in a short period of time, for example, multiple color cameras Y100 may be installed on the moving body U101, and the necessity and number of measuring devices can be freely selected according to the situation.

[0068] The calculation of position and orientation from multiple sensor information is not limited to the method of Labbe et al. (see Non-Patent Document 1) as long as the position and orientation can be calculated using measurement devices such as IMU·Y103 and LiDAR·Y104. When some sensors are not installed as described above, for example, the method of Carlos et al. (see Non-Patent Document 3) that calculates the position and orientation from a camera or a camera and an IMU can be used. When only LiDAR is used, the above-mentioned method of Grisetti et al. (see Non-Patent Document 2) can be used. In this way, any method of calculating the position and orientation can be used depending on the space and purpose.

[0069] In the first embodiment, the position and orientation of the moving object are represented by parameters with three degrees of freedom: X, Y, and θ (yaw). As long as the position and orientation can be represented, they may be stored in another format, such as a 3×3 matrix. They may be stored with six degrees of freedom including the height position (Z) and the rotation parameters of pitch and roll, or may be stored in the form of a 4×4 matrix. By storing them in six degrees of freedom, it becomes possible to calculate the height position and orientation when the moving object moves up a slope, for example.

[0070] The two-dimensional map does not have to be an occupancy grid map created based on scan information measured by LiDAR, as long as it is possible to calculate two-dimensional reference plane information. It may be CAD data that stores a list of wall vertices. The format of the CAD data is also arbitrary, such as straight lines and line segments, as long as it can express a two-dimensional reference plane. It is not limited to CAD, and may be a scanned image of an architectural drawing or a simple floor map.

[0071] In the first embodiment, the three-dimensional reference surface information is stored as the three-dimensional coordinates of the vertices of the surface. However, as long as the surface information can be stored, it may be stored as parameters such as an equation of the surface or an equation of two straight lines including the surface.

[0072] In the first embodiment, the surface is described as a plane, but the present invention can be applied to any continuous surface, such as a curved surface, without being limited to a plane. For example, the two-dimensional reference surface information is stored as line segment information, but if three-dimensional surface information can be calculated, it can also be stored as parameters of curves such as Bezier curves and spline curves. The three-dimensional reference surface information and three-dimensional surface information may also be stored as parameters such as NURBS surfaces. A curved surface may be approximated by multiple small meshes and stored.

[0073] In the first embodiment, two motor controllers Y110 and Y120 are provided in the moving body U101, but a device capable of driving multiple motors may be used for one motor controller. Also, although there are two pairs of motors and wheels, multiple wheels may be driven by one motor, and any combination is acceptable as long as the moving body U101 can be driven to any position and direction.

[0074] In the first embodiment, the information from the encoders Y112 and Y122 may be used to realize closed-loop control by inputting operation information of the wheels Y113 and Y123 to the motor controllers Y110 and Y120. (Instruction values ​​may be used instead of actual rotation amounts.)

[0075] In addition, the wireless connection in the first embodiment may be Bluetooth (registered trademark), 5G (fifth generation mobile communication system), etc., other than WiFi. WiFi is an abbreviation for Wireless Fidelity. Any method may be used as long as the information processing device Y130 can communicate bidirectionally with the tablet U202 and the information storage device Y140.

[0076] In the first embodiment, the system is one in which a user U201 remotely controls a mobile object U101 using a tablet U202. The device for remotely controlling the mobile object U101 is not limited to the tablet U202, but may be other portable terminals or a general computer. That is, the device operated by the user U201 may be a mouse or a keyboard, and the device for displaying information of the information processing device Y130 may be a liquid crystal or an OLED display. OLED is an abbreviation for Organic Light Emitting Diode. Also, the device is not limited to a tablet or a computer, but may be a smartphone, or any other device that can communicate bidirectionally with the mobile object U101.

[0077] The information storage device Y140 may be located on a cloud server or on a server in an in-house network. Furthermore, the information storage device Y140 may be located in the information processing device Y130. Any method may be used as long as the information storage device Y140 can read and write predetermined information.

[0078] In the first embodiment, the storage unit 508 is provided in the information processing device Y130, but it does not have to be provided in the information processing device Y130. The storage unit 508 may be a storage device on a network, and may further be the same as the information storage device Y140.

[0079] [Second embodiment] In the first embodiment, two-dimensional reference plane information is read from an information storage device, and a three-dimensional reference plane is generated. In the second embodiment of the present invention, an information processing device having a configuration in which a three-dimensional reference plane corresponding to a floor is generated in addition to the three-dimensional reference plane, and texture information is also superimposed on these reference planes will be described. In the second embodiment, the configuration within the information processing device is different from that of the information processing device Y130 of the first embodiment, and therefore the information processing device of the second embodiment will be referred to as information processing device 700.

[0080] Fig. 7 is a conceptual diagram of a three-dimensional reference plane in the second embodiment. Fig. 7(A) is a diagram showing a state where a three-dimensional reference plane of M303 is pseudo-generated from two-dimensional reference plane information of M301, and a three-dimensional floor reference plane of M501 is generated. Fig. 7(B) is a diagram showing a state where texture information of a color image acquired by a color camera Y100 mounted on a moving body U101 is superimposed on the three-dimensional reference plane and the three-dimensional floor reference plane.

[0081] In Fig. 7(A), M501 is a three-dimensional plane of the floor, which is called a three-dimensional floor reference plane in the second embodiment. M50 shows how the three-dimensional reference plane of M303 is pseudo-generated from the two-dimensional reference plane information of M301, and how the three-dimensional floor reference plane of M501 is generated.

[0082] In FIG. 7B, M60 shows the state when the texture information of the color image acquired by the color camera Y100 mounted on the moving body U101 is superimposed on the three-dimensional reference plane and the three-dimensional floor reference plane. M602 is a schematic diagram showing the color image data captured by the color camera Y100 at a certain time and the orientation of the color camera at that time. At this time, the information processing device 2 calculates the correspondence between the three-dimensional coordinates on the three-dimensional reference plane and the three-dimensional coordinates on the three-dimensional floor reference plane and the image coordinates, and projects the texture information onto the three-dimensional reference plane and the three-dimensional floor reference plane to generate a three-dimensional model with the texture information superimposed. This process is performed by moving the moving body U101, continuously capturing the surrounding space with the color camera Y100, and projecting the texture information onto the three-dimensional reference plane and the three-dimensional floor reference plane. This allows the user U201 to acquire the surrounding space including the floor surface as a three-dimensional model. The same reference symbols are used for the same configurations as in the first embodiment, and the description is omitted.

[0083] 8 is a diagram showing the configuration of an information processing device 700 according to the second embodiment. The information processing device 700 differs from the first embodiment in that, in addition to the configuration described in the first embodiment, a three-dimensional floor reference plane generating unit 801 is added. The same components as those in the first embodiment are denoted by the same reference numerals and will not be described.

[0084] The three-dimensional reference plane generating unit 801 generates three-dimensional floor reference plane information, and outputs the generated three-dimensional floor reference plane information to the coordinate calculating unit 505. The coordinate calculating unit 505 performs correspondence between the coordinate data of the color image, including the three-dimensional floor reference plane information in addition to the input three-dimensional reference plane information.

[0085] Fig. 9 is a flowchart for explaining the operation of the information processing device 700 according to the second embodiment. The process explained in Fig. 9 differs from the first embodiment in that a process for generating a three-dimensional floor reference surface (step S900) and a process for calculating the coordinates of the three-dimensional floor reference surface (step S901) are added. The same processes as those in the first embodiment are denoted by the same reference numerals and will not be explained.

[0086] In step S900, the three-dimensional floor reference plane generating unit 801 generates a three-dimensional floor reference plane. At this time, the three-dimensional floor reference plane generating unit 801 generates the three-dimensional floor reference plane based on preset height information in the three-dimensional space. After the three-dimensional floor reference plane generating unit 801 generates the three-dimensional floor reference plane, the process of step S603 is executed.

[0087] In step S901, the coordinate calculation unit 505 associates the coordinates of the three-dimensional floor reference plane with the image coordinates. That is, the coordinate calculation unit 505 associates the coordinates of the three-dimensional reference plane with the image coordinates based on the positional relationship between the self-position and orientation of the color camera Y100, which is also used in step S604, and the three-dimensional floor reference plane. After calculating the coordinate association information, the coordinate calculation unit 505 outputs the coordinate association information to the texture information generation unit 507. Then, the process of step S605 is executed.

[0088] In step S605, the texture information generating unit 507 superimposes texture information in the same manner as in the first embodiment, but the target reference planes include the three-dimensional floor reference plane in addition to the three-dimensional reference plane.

[0089] <Effects> As described above, in the second embodiment, in addition to the configuration of the first embodiment, a three-dimensional floor reference plane is also added and texture information is added. This makes the three-dimensional model of the surrounding space of the moving body U101 closer to reality, and has the effect of making it easier for the user to visually recognize the surrounding space in a short time.

[0090] <Variation 2-1> In the second embodiment, a three-dimensional floor reference plane is added to the three-dimensional reference plane in the first embodiment, but in addition to this, a three-dimensional ceiling reference plane may be added to hold the texture information of the ceiling. Here, the distance between the three-dimensional floor reference plane and the three-dimensional ceiling reference plane is set in advance through data such as BIM, and the three-dimensional ceiling reference plane is generated in the same manner as the three-dimensional floor reference plane by reading the distance in advance from the information storage device Y140.

[0091] In addition, the coordinate calculation unit 505 calculates the correspondence between the three reference planes, the three-dimensional reference plane, the three-dimensional floor reference plane, and the three-dimensional ceiling reference plane, and the coordinates of the image captured by the color camera, and projects the texture information onto each reference plane. In this way, according to this modification, a three-dimensional model including the ceiling can be generated.

[0092] [Third embodiment] In the first and second embodiments, the process of superimposing texture information on each reference surface, generating a three-dimensional model, and storing the data in a storage unit has been described. In the third embodiment, the process of displaying a three-dimensional model on a GUI so that a user can check the generated three-dimensional model in real time will be described.

[0093] 10 is a diagram showing the configuration of an information processing device 800 according to the third embodiment and its peripherals. The configuration of the third embodiment includes a display unit 1001 connected to the information processing device 800 in addition to the configuration of the second embodiment. The display unit 1001 is configured to display a three-dimensional model stored in the storage unit 508 of the information processing device 800. The display unit 1001 is, for example, a display of a tablet U202. The other configurations are similar to those of the second embodiment, and therefore description thereof will be omitted.

[0094] FIG. 11 is a flowchart for explaining the operation of the information processing device 800 according to the third embodiment. The processing according to the third embodiment in FIG. 11 is different from the second embodiment in that a display processing of step S1100 is added. That is, in the third embodiment, after each reference plane information to which texture information is added is generated in step S605, the information processing device 800 displays a three-dimensional model using the display unit 1001 in step S1100. In addition, every time an image is selected in step S603 and the texture information is updated thereafter, the three-dimensional model displayed on the display unit 1001 is updated in step S1100. The processing of each of the other steps is similar to that of the second embodiment, and therefore will not be described.

[0095] The GUI of the device according to the third embodiment of the present invention will be described. Fig. 12 is a diagram showing an example of a GUI displayed on the display unit 1001 according to the third embodiment of the present invention. GUI G10 shown in Fig. 12 is an example of display information presented by the display device according to the third embodiment of the present invention.

[0096] GUI G10 shows an example of a GUI for remotely controlling the moving body U101, a three-dimensional reference plane of the space described in the first and second embodiments, the position and attitude of the moving body U101, and display of operation information.

[0097] G101 is an emergency stop button that allows the user to stop the movement of the moving object U101 by touching the emergency stop button G101 with a finger.

[0098] G102 is a cursor. The cursor G102 can be moved in response to a user's touch operation via a mouse, controller, or touch panel (not shown), and buttons and check boxes in the GUI can be operated by pressing a button pointed to by the cursor G102.

[0099] G103 is an example of a controller for a moving object U101. The user can move the circle inside the controller G103 up, down, left, and right to move the moving object U101 forward, backward, left, and right according to the input. The moving object U101 is, for example, an AMR.

[0100] G104 is an example presenting the internal state of the moving object U101. In Fig. 12, the moving object U101 is shown as being in auto-run mode and moving at a speed of 0.5 m / sec.

[0101] G105 is a button for resetting the autopilot and operational settings.

[0102] G106 is a GUI for setting display information such as three-dimensional reference planes for space. G106 allows you to select whether or not to display each element, such as two-dimensional reference planes, three-dimensional reference planes, and texture information, using check boxes.

[0103] G107 is an example of displaying operation information of a mobile unit U101. G107 displays the current position of the mobile unit U101 and the destination of the mobile unit.

[0104] G110 is a GUI that presents the three-dimensional reference plane described in the third embodiment as a three-dimensional model. G111 is a three-dimensional reference plane on which no texture information has been superimposed. G112 is a three-dimensional reference plane on which texture information has already been superimposed. Here, no texture information has been superimposed on the floor surface, but when a three-dimensional floor reference plane is used, texture information is also superimposed on the floor surface, and when a three-dimensional ceiling reference plane is used, a three-dimensional model is displayed in which texture information is also superimposed on the ceiling.

[0105] In addition, the user can control the viewpoint of the 3D model by moving his / her finger up, down, left, or right while touching GUI-G110. In this way, a GUI is provided that allows the user to visually check whether texture information is added to each 3D reference surface.

[0106] <Variation Example 3-1> In addition, if there is a three-dimensional reference surface on which you do not want to overlay a texture, there may be a function that allows the user to touch and select that three-dimensional reference surface in the GUI and press a button (not shown) to delete the display of the three-dimensional reference surface.

[0107] <Effects> As described above, in the third embodiment, in addition to the second embodiment, a three-dimensional model is displayed in real time on the display unit, so that information about the surrounding space can be presented to the user in real time. This allows the user to grasp the information about the space in a short time, and to create a model of the space in a short time.

[0108] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0109] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0110] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a two-dimensional reference plane information input means for inputting information on a surface in a two-dimensionally projected space as two-dimensional reference plane information; a three-dimensional reference plane generating means for generating a three-dimensional reference plane based on the two-dimensional reference plane information; A captured image input unit for inputting a captured image of a space captured by an imaging device; a position and orientation input unit for inputting a position and orientation of the imaging device when the captured image is captured; a coordinate calculation means for calculating coordinates on the three-dimensional reference plane corresponding to coordinates on the captured image by using the position and orientation of the imaging device; a texture information generating means for generating texture information for the three-dimensional reference surface based on the coordinates calculated by the coordinate calculating means; 13. An information processing device comprising: (Configuration 2) The two-dimensional reference plane information is information generated based on a point cloud measured on a two-dimensional plane. 2. The information processing device according to configuration 1. (Configuration 3) The three-dimensional reference plane generating means generates the three-dimensional reference plane based on the two-dimensional reference plane information and preset height information. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The two-dimensional reference plane information is reference plane information generated based on design data of a space. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) the texture information generating means weights the captured image based on a geometric relationship between the imaging device and the three-dimensional reference plane; 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The geometric relationship is a distance between the three-dimensional reference plane and the imaging device, or an angle between a normal vector of the three-dimensional reference plane and an optical axis vector of the imaging device, or a combination of the distance and the angle. 6. The information processing device according to configuration 5. (Configuration 7) The three-dimensional reference plane generating means generates at least one of a three-dimensional floor reference plane and a three-dimensional ceiling reference plane, the coordinate calculation means calculates, using the position and orientation of the imaging device, coordinates on at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generation means, which correspond to coordinates on the captured image; the texture information generating means generates texture information for at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generating means based on the coordinates calculated by the coordinate calculating means. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The three-dimensional reference plane generating means generates the three-dimensional floor reference plane based on preset height information in the three-dimensional space. 8. The information processing device according to configuration 7. (Configuration 9) The three-dimensional reference plane generating means generates the three-dimensional ceiling reference plane based on preset height information in the three-dimensional space. 9. The information processing device according to configuration 7 or 8. (Configuration 10) the three-dimensional reference plane generating means has a three-dimensional reference plane dividing means for dividing the three-dimensional reference plane, and a reference plane that does not correspond to a position on the captured image among the divided reference planes is excluded from the three-dimensional reference plane. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) displaying the three-dimensional reference surface and texture information for the three-dimensional reference surface on a display means; 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) a display means for displaying at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generating means, and texture information for at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generating means; 8. The information processing device according to configuration 7. (Configuration 13) a distance image input means for inputting a distance image which is an image showing a distance in space from the distance image measurement device to a structure, the distance image being measured by the distance image measurement device; the three-dimensional reference plane generating means generates a measured three-dimensional reference plane based on the distance image, and integrates the measured three-dimensional reference plane with the three-dimensional reference plane to generate a three-dimensional reference plane to be used by the coordinate calculating means and the texture information generating means; 14. The information processing device according to any one of configurations 1 to 13. (Configuration 14) The integration includes replacing a three-dimensional reference surface generated based on the two-dimensional reference surface information with an actually measured three-dimensional reference surface. 14. The information processing device according to configuration 13. (Method 1) a two-dimensional reference plane information input step of inputting information on a surface in a two-dimensionally projected space as two-dimensional reference plane information; a three-dimensional reference plane generating step of generating a three-dimensional reference plane based on the two-dimensional reference plane information; a captured image input step of inputting a captured image of a space captured by an imaging device; a position and orientation input step of inputting a position and orientation of the imaging device when the captured image is captured; a coordinate calculation step of calculating coordinates on the three-dimensional reference plane corresponding to coordinates on the captured image using the position and orientation of the imaging device; a texture information generating step of generating texture information for the three-dimensional reference surface based on the coordinates calculated by the coordinate calculating step; The method according to claim 1, further comprising: (Program 1) Computer a two-dimensional reference plane information input means for inputting information on a plane in a two-dimensionally projected space as a two-dimensional reference plane; a three-dimensional reference plane generating means for generating a three-dimensional reference plane based on the two-dimensional reference plane information; A captured image input means for inputting a captured image of a space captured by an imaging device; a position and orientation input unit for inputting a position and orientation of the imaging device when the captured image was captured; a coordinate calculation means for calculating coordinates on the three-dimensional reference plane corresponding to coordinates on the captured image by using the position and orientation of the imaging device; and a texture information generating means for generating texture information for the three-dimensional reference surface based on the coordinates calculated by the coordinate calculating means; A program characterized by causing the program to function as a [Explanation of symbols]

[0111] U101: Mobile Y130: Information processing equipment

Claims

1. a two-dimensional reference plane information input means for inputting information on a surface in a two-dimensionally projected space as two-dimensional reference plane information; a three-dimensional reference plane generating means for generating a three-dimensional reference plane based on the two-dimensional reference plane information; A captured image input unit for inputting a captured image of a space captured by an imaging device; a position and orientation input unit for inputting a position and orientation of the imaging device when the captured image is captured; a coordinate calculation means for calculating coordinates on the three-dimensional reference plane corresponding to coordinates on the captured image by using the position and orientation of the imaging device; a texture information generating means for generating texture information for the three-dimensional reference surface based on the coordinates calculated by the coordinate calculating means; 13. An information processing device comprising:

2. The two-dimensional reference plane information is information generated based on a point cloud measured on a two-dimensional plane.

2. The information processing apparatus according to claim 1,

3. The three-dimensional reference plane generating means generates the three-dimensional reference plane based on the two-dimensional reference plane information and preset height information.

2. The information processing apparatus according to claim 1,

4. The two-dimensional reference plane information is reference plane information generated based on design data of a space.

2. The information processing apparatus according to claim 1,

5. the texture information generating means weights the captured image based on a geometric relationship between the imaging device and the three-dimensional reference plane; 2. The information processing apparatus according to claim 1,

6. The geometric relationship is a distance between the three-dimensional reference plane and the imaging device, or an angle between a normal vector of the three-dimensional reference plane and an optical axis vector of the imaging device, or a combination of the distance and the angle.

6. The information processing apparatus according to claim 5,

7. The three-dimensional reference plane generating means generates at least one of a three-dimensional floor reference plane and a three-dimensional ceiling reference plane, the coordinate calculation means calculates, using the position and orientation of the imaging device, coordinates on at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generation means, which correspond to coordinates on the captured image; the texture information generating means generates texture information for at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generating means based on the coordinates calculated by the coordinate calculating means.

2. The information processing apparatus according to claim 1,

8. The three-dimensional reference plane generating means generates the three-dimensional floor reference plane based on preset height information in the three-dimensional space.

8. The information processing apparatus according to claim 7,

9. The three-dimensional reference plane generating means generates the three-dimensional ceiling reference plane based on preset height information in the three-dimensional space.

8. The information processing apparatus according to claim 7,

10. the three-dimensional reference plane generating means has a three-dimensional reference plane dividing means for dividing the three-dimensional reference plane, and a reference plane that does not correspond to a position on the captured image among the divided reference planes is excluded from the three-dimensional reference plane.

2. The information processing apparatus according to claim 1,

11. displaying the three-dimensional reference surface and texture information for the three-dimensional reference surface on a display means; 2. The information processing apparatus according to claim 1,

12. a display means for displaying at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generating means, and texture information for at least one of the three-dimensional floor reference plane and the three-dimensional ceiling reference plane generated by the three-dimensional reference plane generating means; 8. The information processing apparatus according to claim 7,

13. a distance image input means for inputting a distance image which is an image showing a distance in space from the distance image measurement device to a structure, the distance image being measured by the distance image measurement device; the three-dimensional reference plane generating means generates a measured three-dimensional reference plane based on the distance image, and integrates the measured three-dimensional reference plane with the three-dimensional reference plane to generate a three-dimensional reference plane to be used by the coordinate calculating means and the texture information generating means; 2. The information processing apparatus according to claim 1,

14. The integration includes replacing a three-dimensional reference surface generated based on the two-dimensional reference surface information with an actually measured three-dimensional reference surface.

14. The information processing apparatus according to claim 13,

15. a two-dimensional reference plane information input step of inputting information on a surface in a two-dimensionally projected space as two-dimensional reference plane information; a three-dimensional reference plane generating step of generating a three-dimensional reference plane based on the two-dimensional reference plane information; a captured image input step of inputting a captured image of a space captured by an imaging device; a position and orientation input step of inputting a position and orientation of the imaging device when the captured image is captured; a coordinate calculation step of calculating coordinates on the three-dimensional reference plane corresponding to coordinates on the captured image using the position and orientation of the imaging device; a texture information generating step of generating texture information for the three-dimensional reference surface based on the coordinates calculated by the coordinate calculating step; The method according to claim 1, further comprising:

16. Computer a two-dimensional reference plane information input means for inputting information on a plane in a two-dimensionally projected space as a two-dimensional reference plane; a three-dimensional reference plane generating means for generating a three-dimensional reference plane based on the two-dimensional reference plane information; A captured image input means for inputting a captured image of a space captured by an imaging device; a position and orientation input unit for inputting a position and orientation of the imaging device when the captured image was captured; a coordinate calculation means for calculating coordinates on the three-dimensional reference plane corresponding to coordinates on the captured image by using the position and orientation of the imaging device; and a texture information generating means for generating texture information for the three-dimensional reference surface based on the coordinates calculated by the coordinate calculating means; A program characterized by causing the program to function as a

Images