Information processing device, information processing method, and program

Abstract

This information processing device comprises: an acquisition unit that acquires a captured image obtained by capturing a region of interest; a determination unit (for example, a distance measurement point determination unit (22)) that, on the basis of the acquired captured image, determines a measurement point corresponding to each of one or more geometric models representing the geometric shape of the region of interest; and a processing unit (25) that acquires first distance information obtained by measuring the distance to the determined measurement point, and generates a three-dimensional model of the region of interest on the basis of the acquired first distance information.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] Patent Literature 1 discloses an information processing device that performs three-dimensional measurement using LiDAR (Light Detection and Ranging), which recognizes a target to be measured within a space based on the detection result of a space recognition sensor, and sets a scanning area by a ranging sensor that scans the space using a laser and measures distance three-dimensionally so as to include at least a part of the target to be measured. Specifically, Patent Literature 1 discloses an information processing device that recognizes a target to be measured within a space based on an image captured of the space to be measured, and measures distance three-dimensionally so as to include at least a part of the target to be measured, thereby enabling high-speed three-dimensional distance measurement.

[0003] International Publication No. 2019 / 239566

[0004] However, in the information processing device of Patent Document 1, in order to increase speed, the area to be measured based on the captured image is narrowed down, making it difficult to increase speed, for example, when measuring distance to a predetermined target area such as the entire space.

[0005] Therefore, the present disclosure provides an information processing device, an information processing method, and a program that can achieve high-speed distance measurement even when measuring the distance to a predetermined target area.

[0006] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires a captured image of a target area, a determination unit that determines measurement points corresponding to one or more geometric models that indicate the geometric shape of the target area based on the acquired captured image, and a processing unit that acquires first distance information obtained by measuring the distance from the determined measurement points and generates a three-dimensional model of the target area based on the acquired first distance information.

[0007] An information processing method according to one aspect of the present disclosure acquires an image of a target area, determines measurement points corresponding to one or more geometric models that represent the geometric shape of the target area based on the acquired image, acquires first distance information obtained by measuring the distance from the determined measurement points, and generates a three-dimensional model of the target area based on the acquired first distance information.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described information processing method.

[0009] According to one aspect of the present disclosure, it is possible to realize an information processing device or the like that can achieve high-speed distance measurement even when measuring the distance to a predetermined target area.

[0010] FIG. 1 is a diagram illustrating a schematic configuration of an information processing system according to Embodiment 1. FIG. 2 is a block diagram illustrating a functional configuration of the information processing system according to Embodiment 1. FIG. 3 is a sequence diagram illustrating the operation of the information processing system according to Embodiment 1. FIG. 4 is a flowchart illustrating the operation of the information processing device according to Embodiment 1. FIG. 5A is a diagram illustrating a measurement object according to Embodiment 1. FIG. 5B is a diagram illustrating a captured image according to Embodiment 1. FIG. 6 is a diagram illustrating a captured image according to Embodiment 1 in which the captured image is divided into geometric regions. FIG. 7 is a diagram illustrating a captured image in which a ceiling surface is one geometric region according to Embodiment 1. FIG. 8 is a diagram illustrating distance measurement points according to Embodiment 1. FIG. 9A is a diagram illustrating a first setting example of distance measurement points according to Embodiment 1. FIG. 9B is a diagram illustrating a second setting example of distance measurement points according to Embodiment 1. FIG. 10 is a diagram illustrating the effect of increasing the number of measurements at one distance measurement point according to Embodiment 1. FIG. 11A is a diagram illustrating distance measurement points for matching a geometric model with distance information according to Embodiment 1. FIG. 11B is a first diagram illustrating a process performed when the geometric model and distance information do not match according to Embodiment 1. FIG. 11C is a second diagram for explaining processing when a geometric model and distance information according to Embodiment 1 do not match. FIG. 12 is a diagram schematically illustrating a three-dimensional model according to Embodiment 1. FIG. 13A is a diagram illustrating ranging points displayed in a geometric region of a captured image according to Embodiment 1. FIG. 13B is a diagram illustrating ranging points displayed in a three-dimensional model according to Embodiment 1. FIG. 14A is a diagram illustrating a depth estimation result for a captured image according to Modification 1 of Embodiment 1. FIG. 14B is a diagram illustrating an example of setting ranging points based on a depth estimation result for a captured image according to Modification 1 of Embodiment 1. FIG. 15A is a diagram illustrating a first example of a region in which a geometric model is not set according to Modification 2 of Embodiment 1. FIG. 15B is a diagram illustrating a second example of a region in which a geometric model is not set according to Modification 2 of Embodiment 1. FIG. 16A is a diagram illustrating a state in which an accessory object is present in a geometric region according to Modification 3 of Embodiment 1. FIG. 16B is a diagram illustrating a state in which a missing region in which texture is missing is present in a geometric region according to Modification 3 of Embodiment 1. FIG. 16C is a diagram showing a state in which the texture of the defective region according to Modification 3 of Embodiment 1 is complemented.FIG. 16D is a diagram showing a state in which a texture complemented in a defective region according to Modification 3 of Embodiment 1 is clearly displayed. FIG. 17A is a diagram showing a first setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 17B is a diagram showing a second setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 17C is a diagram showing a third setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 18A is a diagram showing a fourth setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 18B is a diagram showing a fifth setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 18C is a diagram showing a sixth setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 18D is a diagram showing a seventh setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 18E is a diagram showing an eighth setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 18F is a diagram showing a ninth setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 19A is a diagram showing a tenth setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 19B is a diagram showing an eleventh setting example of ranging points according to Modification 4 of Embodiment 1. FIG. 20 is a diagram showing an example of setting ranging points according to Modification 5 of Embodiment 1. FIG. 21 is a sequence diagram showing the operation of an information processing system according to Embodiment 2. FIG. 22 is a flowchart showing the operation of an information processing device according to Embodiment 2. FIG. 23A is a diagram for explaining a first example of three-dimensionally modeling a portion of a space according to various other modified examples. FIG. 23B is a diagram for explaining a second example of three-dimensionally modeling a portion of a space according to various other modified examples. FIG. 24 is a diagram for explaining a first example of a method for determining a portion of a space to be three-dimensionally modeled according to various other modified examples. FIG. 25 is a diagram for explaining a second example of a method for determining a portion of a space to be three-dimensionally modeled according to various other modified examples. FIG. 26 is a diagram for explaining a display example of a three-dimensional model when a space region is limited and three-dimensionally modeled according to various other modified examples. FIG. 27 is a diagram for explaining measurement when a directly measurable region exists according to various other modified examples. FIG. 28A is a diagram showing the configuration of a LiDAR according to various other modified examples. FIG. 28B is a diagram for explaining deviation of the distance measurement position according to other various modified examples.FIG. 29A is a diagram showing an image in which the edges of a generated model are transformed onto image coordinates according to various other modified examples. FIG. 29B is a diagram showing an edge image of the entire periphery when measured from inside a space according to various other modified examples. FIG. 30A is a diagram showing the relationship between the light ray direction for distance measurement and the normal direction of the measurement object when the light ray direction and the normal direction of the measurement object are nearly orthogonal according to various other modified examples. FIG. 30B is a diagram showing the spot shape of laser light when the light ray direction for distance measurement and the normal direction of the measurement object are nearly orthogonal according to various other modified examples. FIG. 31A is a diagram showing the relationship between the light ray direction for distance measurement and the normal direction of the measurement object when the light ray direction and the normal direction of the measurement object are nearly parallel according to various other modified examples. FIG. 31B is a diagram showing the spot shape of laser light when the light ray direction for distance measurement and the normal direction of the measurement object are nearly parallel according to various other modified examples. FIG. 32 is a flowchart showing the operation of converting to BIM data according to various other modified examples. FIG. 33 is a diagram showing an example of BIM data according to various other modified examples. FIG. 34 is a diagram showing a table of correspondence between BIM classes and geometric models according to various other modified examples.

[0011] (Background to the Invention of the Present Disclosure) Before describing the embodiments of the present disclosure, the background to the invention of the present disclosure will be described.

[0012] It is sometimes necessary to measure a three-dimensional space and generate a three-dimensional model. In such cases, the entire three-dimensional space can be the target area for distance measurement in order to grasp the shape, etc., of the entire three-dimensional space.

[0013] On the other hand, it is sometimes desirable to achieve faster ranging, for example, when the time for ranging is limited, etc. However, when the entire three-dimensional space is the target area for ranging, it is difficult to achieve high speed with the technology of Patent Document 1.

[0014] Therefore, the inventors of the present application have conducted extensive research into information processing devices etc. that can achieve high-speed distance measurement even when measuring the distance to a predetermined target area such as the entire three-dimensional space, and have devised the following information processing device etc. Specifically, the inventors have devised an information processing device etc. that can perform high-speed measurement by determining distance measurement points (measurement points) required for modeling the spatial shape based on a captured image of the target area and performing distance measurement only at the determined distance measurement points.

[0015] Furthermore, as a further improvement, the inventors of the present application are also studying information processing devices and the like that can realize highly accurate measurements and reduce the amount of data.

[0016] An information processing device according to a first aspect of the present disclosure includes an acquisition unit that acquires an image of a target area, a determination unit that determines measurement points corresponding to one or more geometric models that indicate the geometric shape of the target area based on the acquired image, and a processing unit that acquires first distance information obtained by measuring the distance from the determined measurement points and generates a three-dimensional model of the target area based on the acquired first distance information.

[0017] This allows the measurement points in the target area to be set to measurement points corresponding to one or more geometric models. For example, by determining the measurement points required to create a three-dimensional model of one or more geometric models based on the captured image, the number of measurement points can be reduced, making it possible to achieve high-speed distance measurement even when measuring the distance to a predetermined target area.

[0018] Also, for example, an information processing device according to the second aspect may be an information processing device according to the first aspect, and when second distance information obtained by measuring additional measurement points necessary to determine the one or more geometric models is required in addition to the acquired first distance information, the processing unit may acquire the second distance information and further generate the three-dimensional model based on the second distance information.

[0019] This allows the second distance information to be acquired when additional measurements are required, making it possible to generate a more accurate 3D model. In addition, since the additional required amount of second distance information is acquired, it is possible to shorten the measurement time while suppressing an increase in the amount of data.

[0020] Also, for example, an information processing device according to the third aspect may be an information processing device according to the second aspect, and further include a determination unit that determines whether or not additional measurements are required for each of one or more geometric regions extracted from the captured image and corresponding to the one or more geometric models.

[0021] This allows a determination as to whether or not additional measurements are necessary for each region within the captured image, thereby enabling efficient three-dimensional modeling of the entire target region.

[0022] Also, for example, an information processing device according to a fourth aspect may be an information processing device according to the third aspect, and the determination unit may determine whether or not the additional measurement is necessary by determining whether or not at least one of the one or more geometric models is consistent with the first distance information for that geometric model.

[0023] This allows additional measurements to be performed only when there is a mismatch between the geometric model and the distance information, making it possible to obtain distance information that is effectively consistent, thereby enabling efficient 3D modeling.

[0024] Also, for example, an information processing device according to a fifth aspect may be the information processing device according to the fourth aspect, and the determination unit may determine, as measurement points for the at least one geometric model, measurement points for determining whether or not there is consistency, in addition to measurement points corresponding to the geometric model.

[0025] This allows a determination as to whether or not there is a match, thereby improving measurement accuracy.

[0026] Also, for example, an information processing device according to a sixth aspect may be an information processing device according to any one of the first to fifth aspects, wherein the acquisition unit acquires third distance information obtained by measuring the distance to the target area, and the processing unit determines, based on the captured image and the third distance information, whether the measurement points included in the third distance information are sufficient for determining the one or more geometric models, and if it is determined that they are not sufficient, the first distance information may be acquired.

[0027] Thus, by acquiring the third distance information together with the captured image (for example, simultaneously with the captured image), the overall measurement time can be shortened. In other words, it is possible to achieve even faster distance measurement.

[0028] Also, for example, an information processing device according to a seventh aspect may be an information processing device according to any one of the first to sixth aspects, and the geometric model may include at least one of a plane, a curved surface, a rectangular prism, a straight line, and a curve.

[0029] This makes it possible to efficiently realize three-dimensional modeling of artificial spaces.

[0030] Also, for example, an information processing device according to an eighth aspect is an information processing device according to any one of the first to seventh aspects, and the processing unit may cause a display unit to display the three-dimensional model and measurement points used to generate the three-dimensional model.

[0031] This makes it possible to visualize the measurement points used in three-dimensional modeling, thereby assisting humans in determining the validity of the measurement points.

[0032] Also, for example, an information processing device according to a ninth aspect may be an information processing device according to any one of the first to eighth aspects, in which the processing unit estimates a distance distribution within the target area by estimating monocular depth, and the determination unit determines measurement conditions corresponding to each of the one or more geometric models based on the distance distribution.

[0033] This allows appropriate measurement points to be determined according to the distance distribution before measurement.

[0034] Also, for example, an information processing device according to a tenth aspect is the information processing device according to the ninth aspect, and the determination unit may determine the measurement conditions by at least one of increasing the spatial density of measurement points and increasing the number of measurements of the measurement points as the distance increases.

[0035] This makes it possible to make the spatial density of the distance measurement points more uniform, thereby making it possible to make the measurement accuracy uniform over the entire target area.

[0036] Also, for example, an information processing device according to an eleventh aspect is an information processing device according to any one of the first to tenth aspects, and the determination unit may determine measurement conditions corresponding to each of the one or more geometric models based on the intensity distribution within the target area.

[0037] This allows appropriate measurement points to be determined according to the intensity distribution before measurement.

[0038] Also, for example, an information processing device according to a twelfth aspect may be the information processing device according to the eleventh aspect, and the determination unit may determine the measurement conditions by at least one of increasing the spatial density of measurement points and increasing the number of measurements of the measurement points as the intensity becomes lower.

[0039] This makes it possible to make the measurement accuracy of the distance measurement points more uniform, thereby making it possible to make the measurement accuracy uniform over the entire target area.

[0040] Also, for example, an information processing device according to a thirteenth aspect may be an information processing device according to any one of the third to fifth aspects, and may further include a recognition unit that extracts the one or more geometric areas for measurement based on image recognition of an image of the target area.

[0041] This allows for effective three-dimensional modeling by extracting two-dimensional regions through image recognition.

[0042] Also, for example, an information processing device according to a fourteenth aspect is the information processing device according to the thirteenth aspect, and the recognition unit may exclude an area in the target area where a moving object exists from the one or more geometric areas.

[0043] This makes it possible to eliminate objects that may be a cause of errors, thereby improving the accuracy of the generated three-dimensional model.

[0044] Also, for example, an information processing device according to a 15th aspect is an information processing device according to any one of the 1st to 14th aspects, and the processing unit may generate the three-dimensional model using the first distance information of a portion of the target area.

[0045] This makes it possible to generate a three-dimensional model of only a desired area or an area where distance accuracy can be obtained, thereby reducing the amount of processing required when generating a three-dimensional model.

[0046] Also, for example, an information processing device according to a 16th aspect is an information processing device according to any one of the 1st to 15th aspects, and the processing unit may output information on the 3D model indicating that an area of ​​the target area that is not subject to generation of the 3D model is the non-target area.

[0047] This allows a user checking a three-dimensional model to easily understand the areas that are not to be modeled, thereby effectively supporting the user in checking the three-dimensional model.

[0048] Furthermore, for example, an information processing device according to a 17th aspect is an information processing device according to any one of the 1st to 16th aspects, and when there is an error of a predetermined amount or more between the irradiation position of the laser light emitted by the ranging unit that measures the distance to the measurement point and the calculated irradiation position of the laser light emitted by the ranging unit, the processing unit may perform at least one of correcting the error or notifying the user.

[0049] This reduces the influence on the first distance information of errors caused by a drive mechanism for driving the distance measurement unit, an apparatus equipped with the distance measurement unit, etc. For example, by reducing the need to redo distance measurement due to the large influence of errors, it is possible to achieve faster distance measurement when measuring the distance to the target area.

[0050] Furthermore, for example, an information processing device according to an 18th aspect is an information processing device according to any one of the 1st to 17th aspects, and the processing unit may present an additional measurement position for measuring the distance to the blind spot when a blind spot exists based on the positional relationship between a ranging unit that measures the distance to the measurement point and an object that exists in the target area.

[0051] As a result, when distance measurements are performed multiple times in the target area, the additional measurement positions are automatically indicated, thereby reducing the time required for the additional distance measurements. In other words, even when distance measurements are performed multiple times in the target area, high-speed distance measurements can be achieved.

[0052] An information processing method according to a 19th aspect of the present disclosure includes acquiring a captured image of a target area, determining measurement points corresponding to one or more geometric models that represent the geometric shape of the target area based on the acquired captured image, acquiring first distance information obtained by measuring the distances to the determined measurement points, and generating a three-dimensional model of the target area based on the acquired first distance information.A program according to a 20th aspect of the present disclosure is a program for causing a computer to execute the information processing method according to the 19th aspect.

[0053] This provides the same effect as the above-described information processing method.

[0054] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.

[0055] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0056] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims are described as optional components.

[0057] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0058] In this specification and the drawings, the X-axis, Y-axis, and Z-axis represent the three axes of a right-handed three-dimensional Cartesian coordinate system.

[0059] Furthermore, in this specification, terms indicating relationships between elements, such as "same" and "match," terms indicating the shapes of elements, such as "cylinder" and "sphere," as well as numerical values ​​and numerical ranges, are not expressions that express only the strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about several percent (or about 10%).

[0060] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0061] First Embodiment An information processing device according to the present embodiment will be described below with reference to FIGS. 1 to 13B.

[0062] [1. Configuration of Information Processing Device] First, the configuration of an information processing device according to this embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram showing a schematic configuration of an information processing system 1 according to this embodiment. Note that Fig. 1 shows an exemplary functional configuration of the information processing system 1, and the functional configuration of the information processing system 1 is not limited to that shown in Fig. 1.

[0063] 1, the information processing system 1 includes a scanrider 100, a mobile terminal 200, and a server 300. The scanrider 100 and the mobile terminal 200 are connected to each other so that they can communicate with each other, and the mobile terminal 200 and the server 300 are connected to each other so that they can communicate with each other.

[0064] The information processing system 1 generates a three-dimensional model of a target area using an image of the target area captured by a camera (e.g., an omnidirectional (360-degree) camera) and distance information to a ranging point in the target area measured using a scanner (e.g., an omnidirectional scanner) and LiDAR (Light Detection and Ranging). The three-dimensional model is also referred to as a geometric model (three-dimensional geometric model), a mesh model (three-dimensional mesh model), etc. The camera is not limited to an omnidirectional camera and may be a camera with a limited field of view. The scanner is not limited to an omnidirectional scanner and may be a scanner with a limited control range for the irradiation direction of a laser light source. The use of a laser light source makes it possible to effectively speed up measurement over a wide area.

[0065] The ScanLIDAR 100 includes a camera, a scanner, and a LiDAR, and is configured to measure captured images of a target area and distance information (e.g., point cloud information). The point cloud information includes distance information from the ScanLIDAR 100 to a ranging point and color information of the ranging point. The camera, scanner, and LiDAR may be implemented as an integrated device or as separate devices. For example, if the target area is an indoor room, the ScanLIDAR 100 is installed inside the room or at the entrance to the room by a worker or the like. The room may be a room where a site survey, such as an electrical construction site, is to be conducted. The site survey involves, but is not limited to, acquiring the room structure through measurements and creating a blueprint, for example.

[0066] The mobile terminal 200 is a terminal device held by the worker, and, for example, controls the ScanLIDAR 100 and processes data measured by the ScanLIDAR 100. The mobile terminal 200 is, for example, but not limited to, a tablet terminal or a smartphone. The mobile terminal 200 includes a display unit 210 for displaying data measured by the ScanLIDAR 100. The display unit 210 is, for example, realized by, but not limited to, a liquid crystal display. The display unit 210 is an example of a presentation unit. The ScanLIDAR 100 and the mobile terminal 200 may be integrated.

[0067] The server 300 controls the ScanLIDAR 100 and processes data acquired by the ScanLIDAR 100. The server 300 may be implemented by a server device (PC) or a cloud server. The server 300 may be capable of communicating with the ScanLIDAR 100.

[0068] The information processing system 1 may include at least one of the mobile terminal 200 and the server 300 .

[0069] FIG. 2 is a block diagram showing the functional configuration of the information processing system 1 according to this embodiment.

[0070] As shown in FIG. 2 , the information processing system 1 includes a scan lidar 10 and an information processing device 20 .

[0071] The scanlidar 10 corresponds to the scanlidar 100 shown in FIG. 1 and performs imaging and ranging of an object 50 under the control of an information processing device 20, and transmits the obtained data to the information processing device 20. The scanlidar 10 includes an imaging unit 11, a control unit 12, a scanner 13, a LiDAR 14, and a signal processing unit 15. The control unit 12, the scanner 13, the LiDAR 14, and the signal processing unit 15 perform ranging based on a control plan. Note that the three-dimensional positional relationship between the imaging unit 11 and the LiDAR 14 is assumed to be known through a calibration performed in advance. Furthermore, the three-dimensional positional relationship between the imaging unit 11 and the LiDAR 14 may be updated by a calibration operation as necessary.

[0072] The photographing unit 11 is a camera (for example, an omnidirectional camera) and photographs the target area. In this embodiment, the photographing unit 11 photographs the target area before distance measurement by the LiDAR 14. The photographing unit 11 may be a monochrome camera or a color camera. That is, the photographed image may be a monochrome image or a color image. Furthermore, the photographed image may be a 360-degree image photographed by an omnidirectional camera, a planar projection image, or a fisheye projection image. Note that the image is a still image, but may also be, for example, a moving image.

[0073] The control unit 12 is a control device that controls each component of the scan lidar 10. The control unit 12 controls the scanner 13 based on a control plan (or a control signal based on the control plan) from the control planning unit 23. The control unit 12 may also control the operations of the imaging unit 11 and the LiDAR 14.

[0074] The scanner 13 controls the irradiation direction of the laser light emitted from the laser light source. The scanner 13 is realized by, for example, a two-axis galvanometer mirror, a MEMS (Micro Electro Mechanical Systems) mirror, or a two-axis pan head, but is not limited to these.

[0075] The LiDAR 14 emits laser light and measures the distance to the target object 50 based on information on the reflected light. The LiDAR 14 has a laser light source that emits laser light, and an optical member and a light receiving element for receiving the reflected light. The optical member includes, for example, a lens, but is not limited to this. The distance measurement method of the LiDAR 14 may be a ToF (Time Of Flight) method, an FMCW (Frequency Modulated Continuous Wave) method, or another method. The LiDAR 14 is an example of a distance measurement unit.

[0076] The signal processing unit 15 is a processing unit that generates point cloud data indicating the distance to the target object 50 based on the received light data of the reflected light from the LiDAR 14 and outputs the generated point cloud data to the information processing device 20.

[0077] The information processing device 20 is realized by at least one of the mobile terminal 200 and the server 300 shown in FIG. 1 , and includes, as functional components, a recognition unit 21, a ranging point determination unit 22, a control planning unit 23, a consistency determination unit 24, and a processing unit 25. The information processing device 20 also includes, as hardware components, a non-volatile memory storing a program, a volatile memory serving as a temporary storage area for executing the program, an input / output port, a communication interface, a processor for executing the program, and the like. The memory may be a read-only memory (ROM) or a random access memory (RAM), and can store a program to be executed by the processor. Each functional component of the information processing device 20 is realized by a processor that executes a program stored in the memory. The information processing device 20 may be realized by a mobile terminal such as a stationary personal computer (PC), a smartphone, or a tablet terminal, a dedicated computer, or the like, or by a server (e.g., a cloud server), or by a combination thereof.

[0078] The recognition unit 21 extracts a geometric region and a geometric model (e.g., a 3D geometric model) from the captured image by performing image recognition on the captured image. The geometric region is a region in space distinguished by a geometric shape unit and includes the region of the measurement target. The geometric model indicates the geometric shape of the target region (e.g., the geometric shape of the geometric region), and examples thereof include, but are not limited to, a plane, a curved surface, a rectangular parallelepiped, a straight line (e.g., a rod-shaped body such as a wire), a curved line, a spherical surface, a cylindrical surface (e.g., the side surface of a cylinder), etc.

[0079] The ranging point determination unit 22 determines ranging point positions necessary for generating a 3D model of the target area based on the captured image. For example, the ranging point determination unit 22 may determine ranging points so as to perform only measurements necessary for modeling the spatial shape of the target area. For each geometric area, the ranging point determination unit 22 determines ranging point positions corresponding to the geometric area. Furthermore, the ranging point determination unit 22 may further determine ranging point positions necessary for consistency determination, which determines whether the geometric model and corresponding distance information are consistent.

[0080] The control planning unit 23 creates a control plan including a measurement trajectory and a scanner drive sequence for efficiently measuring the distances to the plurality of distance measurement points based on the plurality of distance measurement points determined by the distance measurement point determination unit 22 .

[0081] The consistency determination unit 24 determines the consistency between the distance information of the ranging points in the geometric region and the geometric model. The consistency determination will be described later with reference to FIGS. 11A to 11C. The consistency determination unit 24 may also determine whether or not additional measurements are necessary for each of one or more geometric regions corresponding to one or more geometric models extracted from the captured image. The consistency determination unit 24 is an example of a determination unit.

[0082] The processing unit 25 generates geometric model parameters (geometric parameters) for each geometric region and a data set of boundary shapes and textures. The processing unit 25 integrates the geometric models of the entire measurement region to generate a 3D model of the entire space. The geometric parameters include, for example, the position (coordinates), size, and shape of the geometric model. The processing unit 25 may also perform processing such as removing measurement noise from multiple measurements of the same point.

[0083] 2. Operation of Information Processing System Next, the operation of the information processing system 1 configured as described above will be described with reference to Fig. 3 to Fig. 13B. Fig. 3 is a sequence diagram showing the operation (information processing method) of the information processing system 1 according to this embodiment.

[0084] 3, first, the information processing device 20 transmits a start signal to the control unit 12 of the scan lidar 10, and the control unit 12 receives the start signal (S11). The start signal includes a command to cause the imaging unit 11 to capture an image of the target area.

[0085] Next, the control unit 12 transmits a shooting instruction to the photographing unit 11 based on the start signal, and the photographing unit 11 receives the shooting instruction (S12).

[0086] Next, the photographing unit 11 photographs the target area (S13). The photographing unit 11 obtains a photographed image by photographing the entire target area, for example. Note that the photographed image may be one or multiple images as long as it includes the entire target area.

[0087] Next, the photographing unit 11 transmits image data representing the photographed image to the information processing device 20, and the information processing device 20 acquires the image data (S14).

[0088] Next, the recognition unit 21 of the information processing device 20 sets a region (geometric region) and a geometric model based on the image data (S15). Then, the ranging point determination unit 22 determines ranging points for each geometric region. As will be described in detail later, if the geometric model is a plane, three or more ranging points are determined, and if the geometric model is a sphere, one or more ranging points are determined. In this way, the ranging point determination unit 22 determines the number of ranging points according to the geometric model of the geometric region.

[0089] Next, the control planning unit 23 creates a control plan for measuring the distances of each of the distance measurement points determined by the distance measurement point determination unit 22 (S16). The control plan includes the order of the distance measurement points to be measured, i.e., the scanner drive sequence. The control planning unit 23 then transmits the control plan to the control unit 12 of the scan lidar 10, and the control unit 12 acquires the control plan (S17).

[0090] Next, the control unit 12 transmits a ranging instruction to the LiDAR 14, and the LiDAR 14 receives the ranging instruction (S18). The ranging instruction also includes control information for the scanner 13, such as a scanner drive sequence based on the control plan.

[0091] Next, the LiDAR 14 and the scanner 13 perform measurements at each ranging point determined by the ranging point determination unit 22 while scanning (S19).

[0092] Next, the LiDAR 14 transmits the point cloud data generated by the ranging to the information processing device 20, and the information processing device 20 acquires the point cloud data (S20).

[0093] Next, the information processing device 20 determines geometric parameters, which are parameters of each geometric model, based on the acquired point cloud data (S21). Once the geometric parameters are determined, a three-dimensional model can be generated. Note that the geometric parameters for a three-dimensional planar region may be a set of three-dimensional coordinates of vertices on the boundary of the three-dimensional planar region, or a set of parameters of an equation of a three-dimensional plane including the three-dimensional planar region and two-dimensional vertex coordinates representing the shape of a two-dimensional region on that plane. Similar parameter expressions may also be used for other geometric models.

[0094] Next, the operation executed by the information processing device 20 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the operation (information processing method) of the information processing device 20 according to this embodiment.

[0095] 4, first, the recognition unit 21 acquires an image (S110). The recognition unit 21 acquires a captured image obtained by the imaging unit 11 capturing an image of a target area. The timing at which the recognition unit 21 acquires a captured image is not particularly limited, and the image may be acquired every time the imaging unit 11 captures an image, or may be acquired periodically. The recognition unit 21 functions as an acquisition unit that acquires the captured image.

[0096] Next, the recognition unit 21 sets a geometric area and a geometric model included in the target area based on the acquired captured image (S120).

[0097] Fig. 5A is a diagram schematically showing an object to be measured according to this embodiment. Fig. 5B is a diagram schematically showing a captured image P1 according to this embodiment. Figs. 5A and 5B are schematic diagrams illustrating the operation of step S120.

[0098] 5A shows a room, which is a rectangular parallelepiped object 50. For convenience, no other objects are placed in the room.

[0099] FIG. 5B shows a captured image P1 (for example, an image showing only a part of the room) captured from the center of the room on the negative side of the X axis.

[0100] When a photographed image P1 as shown in FIG. 5B is acquired, the recognition unit 21 sets five regions A to E as geometric regions whose geometric model is a plane. Region A is the ceiling surface, regions B to D are wall surfaces, and region E is the floor surface. For example, the recognition unit 21 may input the image shown in FIG. 5B into a machine learning model that is trained by machine learning to input an image and output geometric regions contained in the image, thereby acquiring regions A to E as the output. In this way, the recognition unit 21 is configured to be able to identify each region (ceiling surface, floor surface, wall surface, etc.) through image recognition. Note that the boundary lines of the geometric regions can also be acquired from the photographed image. This eliminates the need for measurements to acquire the boundaries of the geometric regions, thereby enabling even faster processing.

[0101] Next, step S120 will be described with reference to FIGS. 6 and 7 for a more realistic environment. FIG. 6 is a diagram showing a captured image P2 according to this embodiment divided into geometric regions. FIG. 7 is a diagram showing a captured image P3 according to this embodiment in which the ceiling surface 510 is one geometric region. The hatching shown in FIGS. 6 and 7 does not indicate a cross section, but indicates that the geometric regions are different. The same applies to the hatching shown in FIG. 8. For convenience, the lighting devices provided on the ceiling surface 510 are not hatched in FIG. 6.

[0102] As shown in Fig. 6, the recognition unit 21 sets a plurality of geometric regions in the captured image P2 through image recognition. In Fig. 6, a plurality of geometric regions are set, including a ceiling surface 510, a wall surface 520, a desk surface 530, and a floor surface 540. Note that areas with different hatching patterns indicate different geometric regions. For example, a different geometric region is set for each of the plurality of desk surfaces 530. Furthermore, for example, a geometric region different from that of the wall surface 520 is set for an object (e.g., a whiteboard) placed on or near the wall surface 520.

[0103] 7, the recognition unit 21 may set an area including a lighting device or the like as one geometric area (in the case of FIG. 7, the geometric area corresponds to the ceiling surface 510). For example, the recognition unit 21 may set a geometric area by regarding another geometric area surrounded by (or sandwiched between) one geometric area as the one geometric area.

[0104] It should be noted that when one area surrounds another area, it is not essential to regard the other area as the one area. For example, the two areas can be determined to be on the same plane when distances between the two areas are determined to be equal after distance measurement is performed on the two areas as separate areas. Therefore, the two areas may be considered as one area after they are determined to be on the same plane.

[0105] In practice, the geometric regions are displayed in different colors, but this is not limiting.

[0106] Furthermore, the ranging point determination unit 22 determines ranging points for measuring distance for each geometric area set by the recognition unit 21. The ranging point determination unit 22 may determine a predetermined number of ranging points depending on the geometric model, the size of the geometric area, etc.

[0107] 8 is a diagram showing distance measurement points 61 according to this embodiment. In this embodiment, the distance measurement point determination unit 22 determines points necessary for generating a geometric model as distance measurement points 61.

[0108] FIG. 8 shows an example in which five ranging points 61 are set on the ceiling surface 510 by the ranging point determination unit 22. In other words, areas on the ceiling surface 510 other than the five ranging points 61 are not measured. The number of ranging points 61 is not limited to five. The number of ranging points 61 may be one or more, and may be at least three if the geometric model is flat, for example. The ranging point determination unit 22 may also set an excess number of ranging points. Increasing the number of ranging points beyond the minimum required number can reduce errors due to averaging and increase the reliability of the consistency evaluation.

[0109] Conventionally, measurements are performed by uniformly scanning an area (here, the ceiling surface 510), which means that a wide area such as the ceiling surface 510 is measured densely, which takes a long time to measure and results in a large amount of data being obtained.

[0110] On the other hand, in this embodiment, since only the points necessary for generating a geometric model are measured, the distances of only a limited number of measurement points are measured, which reduces the transfer time and the amount of data obtained. For example, it is possible to improve accuracy by increasing the number of measurements (number of distance measurements) of each point, while also shortening the measurement time.

[0111] Here, the determination of the ranging points 61 will be described with reference to FIGS. 9A and 9B. FIG. 9A is a diagram showing a first setting example of the ranging points 61 according to this embodiment. FIG. 9B is a diagram showing a second setting example of the ranging points 61 according to this embodiment. FIG. 9A shows the setting of the ranging points 61 when the ranging points can be set freely. FIG. 9B shows the setting of the ranging points 61 when the ranging points cannot be set at arbitrary positions due to the mechanical mechanism of the scanner 13. Note that FIGS. 9A and 9B show an example in which four ranging points 61 (three for distance measurement and one for consistency determination) are determined. Note that the dashed lines in FIG. 9B indicate the scanning path of the scanner 13.

[0112] 9A , when ranging points 61 can be freely set, ranging points 61 may be set at positions separated from each other within the geometric area 51 so that the geometric parameters of the geometric area 51 can be accurately determined. Since the geometric area 51 is trapezoidal, the ranging point determination unit 22 may determine each vertex (i.e., four corners) of the geometric area 51 as the ranging points 61. For example, the ranging point determination unit 22 may determine ranging points 61 according to the shape of the geometric area 51.

[0113] 9B , when the scanning path of the scanner 13 is limited to the geometric area 51 as shown by the dashed line, the ranging point determination unit 22 may set ranging points 61 on the dashed line and at positions spaced apart from each other. Note that the ranging point determination unit 22 may also set all discrete points on the dashed line as ranging points.

[0114] Furthermore, the ranging point determination unit 22 may further determine the number of measurements for one ranging point 61. Fig. 10 is a diagram for explaining the effect of increasing the number of measurements for one ranging point 61 according to this embodiment. In Fig. 10, the horizontal axis represents error distribution, and the vertical axis represents probability density. The solid line represents a graph in the case where a ranging point is measured once, and the dashed line represents a graph in the case where the same ranging point is measured 50 times. The number of measurements (the number of times the same ranging point is repeatedly measured) may be determined based on the reference measurement accuracy or the relative measurement accuracy.

[0115] As shown in Figure 10, the error distribution is narrower when the number of measurements is 50 than when the number of measurements is 1, and the probability density of the error being 0 is more than five times higher than when the number of measurements is 1. This shows that the measurement accuracy can be improved by increasing the number of measurements.

[0116] For example, since the number of measurements depends on the reflectivity of the laser light of the object, the number of measurements for ranging points whose reflectivity (e.g., reflectivity in the visible light region) in the captured image is estimated to be equal to or less than a predetermined value may be set to be greater than the number of measurements for ranging points whose reflectivity is estimated to be greater than the predetermined value. Alternatively, the same ranging point may be measured a predetermined number of times (e.g., three times) or more, the measurement accuracy for each ranging point may be estimated from the measurement results, and if the measurement accuracy does not meet a reference value, the ranging point may be additionally measured. An upper limit may also be set on the number of measurements. By taking measurement accuracy into consideration when determining a geometric model, the accuracy of the generated geometric model can be improved.

[0117] In this embodiment, a laser light source is used as the light source, and the number of distance measurement points is narrowed down, so it is possible to shorten the time required for measurement compared to the conventional method in which an area is uniformly scanned. Therefore, even if the number of measurements is increased, the measurement time is unlikely to become long. In other words, it is possible to improve the measurement accuracy without increasing the measurement time.

[0118] 4 , the control planning unit 23 creates a control plan based on the ranging points determined by the ranging point determination unit 22 (S130), and transmits the control plan to the control unit 12 to cause the scan lidar 10 to perform ranging and acquire ranging results (S140). The point cloud data acquired as ranging results in step S140 is an example of first distance information.

[0119] Next, the consistency determination unit 24 determines whether there are sufficient distance measurement points based on the acquired distance measurement results (S150). The consistency determination unit 24 determines whether there are sufficient distance measurement points by determining whether the distance information of the distance measurement points 61 in the geometric region is consistent with the geometric model. If the distance information is consistent with the geometric model, the consistency determination unit 24 determines that there are sufficient distance measurement points (there are enough distance measurement points). If the distance information is inconsistent with the geometric model, the consistency determination unit 24 determines that there are insufficient distance measurement points (there are not enough distance measurement points). Here, the determination of whether the distance information is consistent with the geometric model will be described with reference to FIGS. 11A to 11C.

[0120] FIG. 11A is a diagram showing distance measurement points for matching a geometric model with distance information according to this embodiment.

[0121] FIG. 11A shows ranging points determined by the ranging point determination unit 22 when the recognition unit 21 determines that the geometric region 52 is a plane. Three ranging points 61 (circles) are ranging points for distance measurement. One ranging point 62 (black dots) is a ranging point for determining whether the geometric model and distance information are consistent with each other, and is set, for example, at a position different from the three ranging points 61. When the geometric region 52 forms a three-dimensional plane, distance information should be obtained in which the ranging point 62 exists on the three-dimensional plane obtained from the three ranging points 61. Note that the number of ranging points 62 is not limited to one, and two or more ranging points may be set. Furthermore, when determining whether the geometric model and distance information are consistent with each other is not performed, the ranging point 62 need not be set.

[0122] If the distance information of the distance measurement points 62 does not match the geometric model, the consistency determination unit 24 determines that the geometric area 52 is not a single three-dimensional plane. In this case, any of the following processes (I) to (IV) may be performed.

[0123] (I) The geometric region 52 may be further divided into smaller regions, geometric models (e.g., planes) of these regions may be determined, and additional ranging points required to determine the parameters of each plane may be determined to determine whether additional ranging is necessary, i.e., whether there are enough ranging points.

[0124] FIG. 11B is a first diagram for explaining processing when the geometric model and distance information do not match according to this embodiment.

[0125] 11B, the consistency determination unit 24 divides the geometric region 52 into two or more geometric regions (here, geometric regions 52a and 52b). For example, the geometric region may be repeatedly divided until the distance information and the geometric model are consistent with each other.

[0126] (II) The geometric region 52 may display on a display unit or the like that the geometric model is inconsistent.

[0127] (III) The consistency determination unit 24 may select the geometric model with the highest consistency from among geometric models prepared in advance, and apply the selected geometric model in place of the geometric model determined to be inconsistent. In other words, the geometric model may be replaced with a geometric model with high consistency.

[0128] (IV) A geometric model of the geometric region 52 may be created by fitting a 3D mesh and a 3D curved surface based on distance information of points that have already been measured.

[0129] 11C is a second diagram illustrating a process when the geometric model according to the present embodiment does not match the distance information. It is assumed that the distances between the vertices of the geometric region 52b are known in relation to the surfaces of the adjacent geometric regions.

[0130] 11C , the consistency determination unit 24 may divide the geometric region 52 into a plurality of 3D (three-dimensional) meshes 70 so that the distances of the measured distance measurement points (here, three distance measurement points 61 and one distance measurement point 62) are satisfied. Fig. 11C shows an example in which the geometric region 52b is divided into 12 3D meshes 70. In this case, for example, no additional distance measurement is performed.

[0131] 4 , next, when the consistency determination unit 24 determines that there are insufficient ranging points (No in S150), the ranging point determination unit 22 determines additional ranging points (S160). Using FIG. 11B as an example, the ranging point determination unit 22 adds ranging points to each of the divided geometric areas 52a and 52b so that the number of ranging points conforms to the geometric model. Because the geometric area 52b is a plane (i.e., there are three or more ranging points) and ranging of the ranging points 61 and 62 has been completed, the ranging point determination unit 22 adds at least one ranging point 63. Furthermore, because the geometric area 52a is a plane (i.e., there are three or more ranging points) and ranging of the two ranging points 61 has been completed, the ranging point determination unit 22 adds at least one ranging point 63.

[0132] Note that step S160 is not limited to adding a new ranging point, and the same ranging point as a ranging point that has already been measured may be determined. In other words, step S160 may determine a ranging point to increase the number of measurements at the same ranging point.

[0133] If the consistency determination unit 24 determines that there are enough distance measurement points (Yes in S150), the process proceeds to step S190. In this case, no additional distance measurement is performed.

[0134] Next, the control planning unit 23 updates the additional control plan (S170). For example, the control planning unit 23 creates a control plan for the additional ranging point.

[0135] Next, the control planning unit 23 transmits a control plan for the additional ranging points to the control unit 12, thereby causing the scan lidar 10 to perform additional ranging and acquire ranging results (S180). The point cloud data acquired as ranging results in step S180 is an example of second distance information.

[0136] Next, the processing unit 25 sets parameters of each geometric model based on the acquired distance information of each ranging point (S190). Any known method may be used to set parameters of the geometric model from the distance information.

[0137] Next, the processing unit 25 determines whether measurement of all geometric regions has been completed (S200). If the processing unit 25 determines that measurement of all geometric regions has been completed, i.e., that parameters have been set for each geometric region (Yes in S200), it outputs a three-dimensional model based on the parameters of each geometric model (S210). For example, the processing unit 25 may transmit the three-dimensional model to a device equipped with a display unit, thereby displaying the three-dimensional model on the display unit. Furthermore, if the information processing device 20 is equipped with a display unit, the processing unit 25 may display the three-dimensional model on the display unit. By outputting the three-dimensional model, the amount of data can be reduced compared to when point cloud data of the target region is output.

[0138] As described above, if the processing unit 25 judges Yes in step S150 for all geometric regions, it generates a three-dimensional model based on the first distance information, and if there is a geometric region for which the processing unit 25 judges No in step S150, it generates a three-dimensional model for that geometric region based on the first distance information and the second distance information.

[0139] FIG. 12 is a diagram schematically illustrating a three-dimensional model M1 according to this embodiment. For example, FIG. 12 illustrates a three-dimensional model corresponding to the captured image P1 of FIG. 5B. Since such a three-dimensional model M1 is not composed of point cloud data, the amount of data can be reduced. Note that the surface pattern of the output three-dimensional model M1 may be acquired from the captured image P1, for example.

[0140] The processing unit 25 may display the ranging points on the display unit. An example of displaying the ranging points on the display unit will be described with reference to Fig. 13A and Fig. 13B. Fig. 13A is a diagram showing ranging points 61 displayed in a geometric region of a captured image P1 according to this embodiment. Fig. 13B is a diagram showing ranging points 61 displayed on a three-dimensional model M2 according to this embodiment.

[0141] 13A , ranging points 61 may be displayed (for example, superimposed) on the captured image P1. The processing unit 25 may cause the display unit to display the captured image P1 and the ranging points 61 used to generate the three-dimensional model M2 of the target area.

[0142] 13B, ranging points 61 may be displayed (e.g., superimposed) on the three-dimensional model M2. For convenience, only ranging points 61 on the ceiling surface are shown in Fig. 13A and Fig. 13B. The processing unit 25 may cause the display unit to display the three-dimensional model M2 and the ranging points 61 used to generate the three-dimensional model M2.

[0143] By displaying the ranging points on the display unit, the user can adjust at least one of the positions and the number of ranging points determined by the information processing device 20. Furthermore, the processing unit 25 may display the number of measurements for each ranging point, the time required for ranging, etc., in addition to the ranging points.

[0144] 4, if the processing unit 25 determines that measurement of all geometric regions has not been completed, that is, that there is a geometric model for which parameters have not been set (No in S200), it executes distance measurement of the next region (geometric region) and obtains the distance measurement result (S220).Then, the process proceeds to step S150, where it is determined whether the obtained distance measurement points are sufficient.

[0145] When there are a large number of geometric models, the measurement may be performed starting from an area with a higher priority (for example, an area with a larger area), and the number of areas for which distances are measured within a predetermined time limit may be limited.

[0146] (Variations of Embodiment 1) Below, each variation of Embodiment 1 will be described with reference to FIGS. 14A to 20. Note that the following description will focus on differences from Embodiment 1, and descriptions of content that is the same as or similar to Embodiment 1 will be omitted or simplified. The configuration of the information processing system in each variation may be the same as the configuration of the information processing system 1 according to Embodiment 1, and descriptions thereof will be omitted. Furthermore, each variation will be described using the reference numerals of the information processing system 1 according to Embodiment 1.

[0147] (Variation 1 of Embodiment 1) An information processing system 1 according to this variation will be described below with reference to FIGS. 14A and 14B. In this variation, an example will be described in which a depth estimation result of a captured image is used to determine a ranging point. FIG. 14A is a diagram showing a depth estimation result D1 of a captured image according to this variation. FIG. 14B is a diagram showing an example of setting a ranging point 61 based on a depth estimation result of a captured image P2a according to this variation. FIG. 14B shows an example in which a ranging point 61 is set on the wall on the left side of the page using the depth estimation result D1. Note that the image shown in FIG. 14B is the result of geometric region division of the captured image P2a, but for convenience, hatching for distinguishing the geometric regions has been omitted.

[0148] FIG. 14A shows a depth estimation result D1 of a captured image, with colors (shades in FIG. 14A) corresponding to depth. Depth estimation makes it possible to obtain relative positional relationships such as perspective. The depth estimation result D1 is the result of estimating the distance distribution within the target area. In this modification, the image capture unit 11 is a monocular camera, and depth estimation is performed using monocular depth estimation for images captured by the monocular camera. Monocular depth estimation from an image can be achieved using machine learning.

[0149] As shown in Fig. 14B, in the captured image P2a, six ranging points 61 are set on the wall on the left side of the page. Here, because distant areas appear smaller in the image, the ranging points 61 are set by increasing the number of ranging points 61 in distant areas compared to the number of ranging points 61 in closer areas. In other words, the ranging points 61 are set so that the spatial density of ranging points 61 increases as the area becomes more distant. This makes it possible to acquire distance information with a uniform density in three dimensions. Note that while the example in Fig. 14B shows an example in which distant ranging points 61 are increased, it is also possible to increase the number of measurements for one distant ranging point 61, for example.

[0150] In this way, the ranging point determination unit 22 determines at least one of the number of ranging points 61 and the number of measurements at each ranging point 61 by at least one of increasing the spatial density of the ranging points 61 and increasing the number of measurements at each ranging point 61 as the distance increases (is farther) based on the distance distribution. At least one of the number of ranging points 61 and the number of measurements at each ranging point 61 is an example of a measurement condition.

[0151] (Second Modification of First Embodiment) An information processing system 1 according to this modification will now be described with reference to Figs. 15A and 15B. In this modification, an example will be described in which an area in which no geometric model setting is performed is determined from a captured image. Fig. 15A is a diagram showing a first example of an area in which no geometric model setting is performed according to this modification. Fig. 15B is a diagram showing a second example of an area in which no geometric model setting is performed according to this modification.

[0152] The recognition unit 21 may determine that a specific area in the captured image is not a target area for distance measurement. Since no distance measurement points are set in such areas, the distance measurement time can be further reduced.

[0153] FIG. 15A shows the sky including clouds 81 (sky region) as an example of a region that is not the subject of distance measurement.

[0154] 15B shows a person 82 (person area) as an example of an area that is not subject to distance measurement. The person 82 is moving, but may also be stationary, for example. This makes it possible to prevent people from being included in the output 3D model even if they are present in the room. Furthermore, the person area is not subject to the geometric model, and in order to prevent excessive laser irradiation, it is possible to perform image recognition of the person's head in particular and not set distance measurement points near the head.

[0155] Furthermore, the area not subject to distance measurement may be a moving object (moving object area). The determination of a moving object can be performed using still image recognition, two or more frames of continuously captured images, or FMCW-LiDAR measurement information. For example, using FMCW-LiDAR makes it possible to quickly and accurately classify moving objects. In this way, the recognition unit 21 may exclude areas in the target area where moving objects exist from the geometric area of ​​the measurement target. Furthermore, the sky, people 82, etc. can be identified by image recognition.

[0156] Furthermore, if the purpose is to create a structural model of a structure, accessories (for example, desks, chairs, etc.) may be excluded from the range measurement target area.

[0157] (Variation 3 of Embodiment 1) Hereinafter, an information processing system 1 according to this variation will be described with reference to FIGS. 16A to 16C. In this variation, a process for complementing the texture of a concealed portion will be described. FIG. 16A is a diagram showing the presence of an accessory object 53 within a geometric area 51 according to this variation. FIG. 16B is a diagram showing the presence of a missing area R1, in which texture is missing, within a geometric area 51 according to this variation. FIG. 16C is a diagram showing the complemented texture of the missing area R1 according to this variation. FIG. 16D is a diagram showing the texture 53a complemented in the missing area R1 according to this variation. Note that the hatching shown in FIGS. 16A to 16D does not indicate a cross section, but indicates the texture. Different hatching patterns mean different textures.

[0158] 16A and 16B, if an area of ​​an accessory 53 exists within a geometric area 51, the processing unit 25 geometrically models only the geometric area 51. At this time, as shown in FIG. 16B, part of the texture of the geometric area 51 is missing. Specifically, the texture of a missing area R1 shown in FIG. 16B is missing. The missing area R1 is an area of ​​the same size and position as the area of ​​the accessory 53. Because the missing area R1 has not been photographed by the photographing unit 11, there is no texture in the missing area R1.

[0159] In this case, as shown in FIG. 16C , the texture of the region of the accessory 53 (missing region R1) may be generated using an image of the geometric region 51 around the accessory 53. The processing unit 25, for example, complements the texture of the region of the accessory 53 based on the texture of the geometric region 51 around the accessory 53. For example, the processing unit 25 may make the texture of the region of the accessory 53 the same as the texture of the geometric region 51 around the accessory 53. For example, the structure behind the region excluded from the measurement target may be complemented from a surrounding geometric model. At this time, the texture may also be complemented from a surrounding image. This can improve the appearance of the image.

[0160] In this case, the area of ​​texture 53a generated in FIG. 16C may be clearly indicated as shown in FIG. 16D. In FIG. 16D, the area of ​​texture 53a is indicated by a dashed line, but the area of ​​texture 53a corresponding to the area of ​​accessory 53 may be clearly indicated by changing the display color, for example. In this way, processing unit 25 displays the area where texture 53a is complemented and the area where it is not complemented (i.e., the area where there is no obscuration and no missing parts) in a distinguishable manner. Note that this texture 53a is the same area as the area of ​​accessory 53.

[0161] The processing unit 25 may display at least one of the model shown in Fig. 16C and the model shown in Fig. 16D on the display unit 210. Furthermore, the processing unit 25 may switch between the model shown in Fig. 16C and the model shown in Fig. 16D and display them based on an instruction from the user.

[0162] 17A to 19B are diagrams showing examples of setting distance measurement points according to this modification. The distance measurement points shown in Fig. 17A to 19B are determined by the distance measurement point determination unit 22.

[0163] 17A shows distance measurement points 61 set on a rectangular parallelepiped object 54 whose reflectance in a captured image (for example, reflectance in the visible light region) is higher than a predetermined value. For example, since the front surface of the object 54 is flat, three distance measurement points 61 are set.

[0164] The dot hatching in Figures 17B and 17C indicates that the objects are darker (have lower reflectance) than those in Figure 17A. In other words, it is possible to identify from the captured images that the objects 54 and 55 are rectangular parallelepipeds.

[0165] 17B shows an example in which the number of distance measurement points 61 set on the object 55 is increased when the brightness of the geometric region is low. Low brightness of the geometric region is synonymous with low reflectance of the geometric region.

[0166] 17C shows an example in which the number of measurements is increased for distance measurement points 61 set on the object 55 when the brightness of the geometric region is low. In FIG. 17C, three measurements are performed for each distance measurement point 61.

[0167] For example, if a region of the captured image is dark, the reflectance of the laser light in that region may be low. For example, assuming that if the reflectance of the region decreases, the reflectance of the laser light will also be low (i.e., accuracy will decrease when using laser light), the ranging points 61 and the like may be determined from the captured image as shown in Figures 17B and 17C.

[0168] In this way, the ranging point determination unit 22 determines at least one of the number of ranging points 61 and the number of measurements at each ranging point 61 by at least one of increasing the spatial density of the ranging points 61 and increasing the number of measurements at each ranging point 61 based on the intensity distribution within the target area. At least one of the number of ranging points 61 and the number of measurements at each ranging point 61 is an example of a measurement condition.

[0169] As shown in FIG. 18A, when the geometric model is a plane, three or more distance measurement points 61 are set.

[0170] 18B, when the geometric model is the side surface of a cylinder, two or more distance measurement points 61 are set. The radius of the cylinder can be estimated from the image information.

[0171] 18C, when the geometric model is a sphere, one or more distance measurement points 61 are set. The radius of the sphere can be estimated from the image information.

[0172] 18D to 18F, for a rod-shaped body such as a wire, distance measurement points 61 are set at a high density around the rod-shaped body. Note that for a three-dimensional parallel structure or lattice structure, distance measurement points 61 are similarly set at a high density around the rod-shaped body.

[0173] 19A shows distance measurement points 61 set on a three-dimensional geometric model such as a pillar or beam. If it is known from the captured image that the object is a rectangular parallelepiped, it is sufficient to measure, for example, at least one surface of the rectangular parallelepiped (two surfaces in the case of FIG. 19A ) by the number of points corresponding to the geometric model of that surface.

[0174] 19B shows distance measurement points 61 set on a geometric model of the entire room. In this case, one distance measurement point 61 may be set on each face constituting a rectangular parallelepiped. Note that the recognition unit 21 may detect the shape of the room that encompasses the space to be measured as a rectangular parallelepiped. In this case, for example, measurement is performed from inside the rectangular parallelepiped.

[0175] (Fifth Modification of First Embodiment) Figure 20 is a diagram showing an example of setting ranging points 61 and 63 according to this modification. This modification describes a case where, due to the characteristics of the laser scanner, it is not possible to measure only the necessary ranging point 61, that is, where the geometric area is uniformly scanned. The ranging point 61 (●) is a ranging point that has been determined by the ranging point determination unit 22 as a ranging point for the plane. The ranging point 63 (○) is a ranging point that has not been determined by the ranging point determination unit 22 as a ranging point for the plane.

[0176] 20 , due to the structure of the scanlidar 10, measurements are performed for each ranging point, including ranging points 61 and 63. In this case, the signal processing unit 15 extracts point cloud data corresponding to ranging point 61 from the data for each ranging point obtained from the LiDAR 14, and transmits only the extracted point cloud data to the information processing device 20. The data for ranging point 63 is not used to generate a three-dimensional model. This reduces the amount of point cloud data transmitted to the information processing device 20.

[0177] (Embodiment 2) Hereinafter, this embodiment will be described with reference to Figs. 21 and 22. Note that the following description will focus on differences from embodiment 1, and descriptions of content that is the same as or similar to embodiment 1 will be omitted or simplified. The configuration of the information processing system in embodiment 2 may be the same as the configuration of the information processing system 1 according to embodiment 1, and description thereof will be omitted. Furthermore, this embodiment will be described using the symbols of the information processing system 1 according to embodiment 1. Fig. 21 is a sequence diagram showing the operation (information processing method) of the information processing system 1 according to this embodiment. In this embodiment, measurement is performed by the LiDAR 14 when the imaging unit 11 captures an image.

[0178] As shown in FIG. 21 , the information processing device 20 first transmits a start signal to the control unit 12 of the scanlidar 10, and the control unit 12 receives the start signal (S11). The start signal here includes not only the start of image capture by the image capture unit 11 but also the start of ranging by the LiDAR 14. Note that ranging by the LiDAR 14 here means measuring the entire target area with sparse ranging points. For example, ranging is performed using pre-set ranging points, i.e., ranging points that are not based on a geometric area or geometric model based on the captured image. The number of sparse ranging points may be, for example, less than half, less than one-third, less than one-fifth, or less than one-tenth of the maximum number of ranging points that can be measured by scanning using the scanner 13 and the LiDAR 14. Furthermore, from the perspective of effectively utilizing the time required for the processing of steps S14 and S15, the number of sparse ranging points may be, for example, the number of ranging points that can be measured within that time. The sparse distance measurement points may be distance measurement points set at equal intervals, or distance measurement points set at random intervals, for example.

[0179] Next, the control unit 12 transmits an image capturing instruction to the image capturing unit 11 based on the start signal, and the image capturing unit 11 acquires the image capturing instruction (S12). Also, the control unit 12 transmits a ranging instruction to the LiDAR 14 based on the start signal, and the LiDAR 14 acquires the ranging instruction (S31). For example, the transmission of the image capturing instruction may be triggered by the transmission of the ranging instruction.

[0180] Next, for example, after capturing an image (S13), the LiDAR 14 performs scanning and ranging (S32) and transmits the measured point cloud data (sparse point cloud data) to the information processing device 20, and the information processing device 20 acquires the transmitted point cloud data (sparse point cloud data) (S33). The acquired point cloud data is sparse point cloud data with a lower point cloud density than the point cloud density measurable by scanning with the scanner 13 and the LiDAR 14.

[0181] Next, if the information processing device 20 determines that additional ranging is necessary based on the acquired point cloud data and the ranging points determined by the ranging point determination unit 22 (S34), for example, if there is insufficient data at the ranging points acquired in step S32, it creates a control plan for measuring the missing ranging points (S16) and executes processing from step S17 onwards.

[0182] In Figure 21, an example is described in which scanning measurement (S32) is performed after photographing (S13), but this is not limited to this, and photographing (S13) may be performed after scanning measurement (S32), or scanning measurement (S32) and photographing (S13) may be performed in parallel.

[0183] Next, the operation executed by the information processing device 20 will be described with reference to Fig. 22. Fig. 22 is a flowchart showing the operation (information processing method) of the information processing device 20 according to this embodiment.

[0184] 22 , first, the recognition unit 21 acquires an image and sparse point cloud data (S110a). The recognition unit 21 acquires a captured image obtained by the imaging unit 11 capturing an image of a target area, and sparse point cloud data obtained by measuring the distance to the target area. The recognition unit 21 stores the sparse point cloud data in a storage unit (not shown), for example. The storage unit may be realized by, for example, a semiconductor memory, a hard disk drive (HDD), or the like, but is not limited to these.

[0185] The timing at which the image and the sparse point cloud data are acquired is not particularly limited, and they may be acquired at different times or at the same time. For example, the sparse point cloud data is acquired after the image is acquired. The sparse point cloud data only needs to be acquired at least before the determination of step S150 is performed. For example, it may be acquired during the execution of step S120 or between steps S120 and S150. The recognition unit 21 functions as an acquisition unit that acquires the captured image and the sparse point cloud data. The sparse point cloud data is an example of third distance information. Acquiring sparse point cloud data is also referred to as advance measurement.

[0186] Next, the recognition unit 21 sets a geometric area and a geometric model included in the target area based on the acquired captured image (S120).

[0187] Next, the consistency determination unit 24 determines whether there are enough ranging points based on the acquired ranging results (sparse point cloud data) and the ranging points determined by the ranging point determination unit 22 (S150). It can also be said that the consistency determination unit 24 determines whether the sparse point cloud data and the ranging points determined by the ranging point determination unit 22 are consistent with each other. The consistency determination unit 24 compares a first number of ranging points required to determine the parameters of the geometric region determined by the ranging point determination unit 22 with a second number of point cloud data from the sparse point cloud data that have measured the distances within the geometric region, and performs the determination in step S150 based on the comparison result. For example, if the second number is equal to or greater than the first number, the consistency determination unit 24 determines that there are enough ranging points in the geometric region, and if the second number is less than the first number, the consistency determination unit 24 determines that there are insufficient ranging points in the geometric region. The consistency determination unit 24 performs the determination in step S150 for each geometric region.

[0188] Next, if the consistency determination unit 24 determines that the number of ranging points is insufficient (No in S150), the ranging point determination unit 22 determines additional ranging points (S160). The ranging point determination unit 22 adds the missing number of ranging points in each geometric region as additional ranging points.

[0189] Next, the control planning unit 23 updates an additional control plan for performing distance measurement at the additional distance measurement points (S170). That is, the control planning unit 23 creates a control plan for the missing distance measurement points.

[0190] If the consistency determination unit 24 determines that there are sufficient distance measurement points (Yes in S150), the process proceeds to step S190. In this case, in step S190, parameters of the geometric model are set using the sparse point cloud data (S190). For example, if the sparse point cloud data is point cloud data measured during execution of step S120, the time required from measurement to output of the 3D model can be further reduced.

[0191] Next, the processing unit 25 determines whether measurement of all geometric regions has been completed (S200). If measurement of all geometric regions has been completed (Yes in S200), the processing unit 25 outputs a 3D model based on the parameters of each geometric model (S210). If measurement of all geometric regions has not been completed, that is, if it is determined that there is a geometric model for which parameters have not been set (No in S200), the processing unit 25 moves on to the next geometric region (S220a) and performs the processes from step S150 onwards for the next geometric region. In this embodiment, the processing unit 25 may generate a 3D model based on, for example, at least sparse point cloud data.

[0192] Note that advance ranging (ranging for acquiring third distance information) may be performed in parallel with image recognition of the captured image. For example, advance ranging may be performed based on the results of simple recognition in image recognition. Simple recognition may simply determine whether or not a predetermined object (e.g., an object that does not need to be measured, such as the sky or a person) is captured in the captured image, and advance ranging may be performed in an area other than the area of ​​the object that does not need to be measured.

[0193] Furthermore, measurement points for advance measurement may be determined in advance. For example, a range where the absolute value of the elevation angle is 60 degrees or more, which is a direction that is likely to be the floor or ceiling, may be determined in advance as measurement points for advance ranging. Furthermore, advance ranging may be performed based on simple image analysis with a short processing time (for example, image analysis not based on a geometric model). In simple image analysis, the image resolution may be reduced and an image segmentation process may be performed to identify large areas such as walls and floors, and advance ranging may be performed by prioritizing these large areas. Alternatively, advance ranging may be performed by detecting a predetermined marker, or by detecting a specific color area.

[0194] Examples of the marker include one-dimensional codes such as barcodes and two-dimensional codes such as QR Codes (registered trademark), which are attached in advance to locations where measurement is desired. The specific color area indicates an area of ​​an object having a color that may reduce the accuracy of laser ranging (e.g., a black object). The presence or absence of a specific color area may be determined, and advance ranging may be performed preferentially for the specific color area. Furthermore, directions measured in the past may be stored, and advance ranging may be performed for directions that are frequently used.

[0195] Such processing relating to advance ranging is executed by the information processing device 20.

[0196] (Various other modified examples) Various other modified examples that can be applied to any of Embodiment 1, Modifications 1 to 5 of Embodiment 1, and Embodiment 2 will be described below with reference to Figures 23A to 34. Note that the following description will focus on differences from Embodiment 1, Modifications 1 to 5 of Embodiment 1, or Embodiment 2, and descriptions of content that is the same as or similar to Embodiment 1, Modifications 1 to 5 of Embodiment 1, or Embodiment 2 will be omitted or simplified.

[0197] As a first modified example, a process for generating a 3D model of a partial area of ​​space will be described using Figures 23A to 27. As a second modified example, an example of the configuration of LiDAR will be described using Figures 28A and 28B. As a third modified example, a process for correcting the edges of a generated model will be described using Figures 29A and 29B. As a fourth modified example, a process for correcting the edges of a generated model will be described using Figures 30A to 31B. As a fifth modified example, a process for converting a 3D model into BIM data will be described using Figures 32 to 34.

[0198] 23A and 23B are diagrams for explaining examples of three-dimensional modeling of a part of space according to other various modifications. The solid lines in Fig. 23A and Fig. 23B indicate areas that are three-dimensionally modeled, and the dashed lines indicate areas that are not three-dimensionally modeled.

[0199] When three-dimensionally modeling a space such as a long corridor as shown in FIG. 23A or a large room as shown in FIG. 23B, the image resolution and distance measurement accuracy decrease as the distance from the point where the scan lidar 10 is installed increases, so the three-dimensional area to be modeled may be limited.

[0200] As shown in FIG. 23A , when generating a 3D model of a space 600 such as a long passageway, for example, when generating a 3D model of a space such as a hallway, the processing unit 25 may 3D model only a space 601 (e.g., a rectangular parallelepiped model space) within a predetermined distance (10 m in the example of FIG. 23A ) along the passageway, based on the point where the ScanLIDAR 10 is installed. The predetermined distance may be set in advance by the user or may be set based on the performance of the LiDAR 14. Furthermore, if a door 602 is open, the distance to the door 602 may not be accurately measured, so 3D modeling of the area outside the passageway of the door 602 may not be performed. In other words, the open door 602 may not be included in the measurement target. Similarly, the area outside the window may not be included in the measurement target.

[0201] Whether the door 602 is open or not may be determined by performing image recognition on the captured image, or may be determined based on the continuity of the distance between the surface of the door 602 and the surface of the wall.

[0202] As shown in Figure 23B, when generating a three-dimensional model of a space 610 such as a large room, the processing unit 25 may create a three-dimensional model of a space 611 of a predetermined range based on the point where the scan lidar 10 is installed.

[0203] The processing unit 25 may accept settings of the target area to be three-dimensionally modeled from the user, may automatically set the target area to be three-dimensionally modeled, or may determine the target area to be three-dimensionally modeled by a combination of these.

[0204] The processing unit 25 may create a three-dimensional model of an area designated by the user from among areas included in the image captured by the imaging unit 11 .

[0205] 24 is a diagram illustrating a first example of a method for determining a portion of space to be three-dimensionally modeled according to various other modified examples. The dot-hatched area shown in FIG. 24 indicates an area that is not subject to three-dimensional modeling.

[0206] 24 , the user sets a target region for 3D modeling in the captured image P4 by moving a cursor 622. The processing unit 25 may receive input of the position of the cursor 622 as a boundary between a region for which 3D modeling is performed and a region for which 3D modeling is not performed, and generate a 3D model based on the point cloud data and image data of the target region set based on the position of the cursor 622.

[0207] Referring again to FIG. 23B, the processing unit 25 may automatically set the target area to be three-dimensionally modeled based on either or both of the basic shape of the target area, such as a rectangular parallelepiped or a polygonal pillar, and a distance limit of the range (e.g., a maximum of 20 m), or based on the boundaries of structural objects such as pillars and walls.

[0208] 25 is a diagram illustrating a second example of a method for determining a portion of a space to be three-dimensionally modeled according to various other modified examples. For convenience, FIG. 25 shows an overhead image P5 of the space in which the scanlidar 10 is installed. The location where the scanlidar 10 is installed is indicated by an x ​​mark.

[0209] As shown in Fig. 25 , the processing unit 25 may determine, as the target region, a region R2 whose boundaries are bounded by structural objects such as pillars and walls within a preset maximum distance (10 m in Fig. 25 ). For example, if the distance between the outer two pillars 621 of four pillars 621 lined up on the bird's-eye view is 10 m, as shown in Fig. 25 , the region R2 surrounded by the eight pillars 621 is determined to be the target region, and the other regions are excluded from the region for generating a 3D model as they are not the target region. The pillars 621 are an example of an object.

[0210] In addition, LiDAR14 may measure the distance of the entire space including the target area, extract point cloud data included in the target area from the distance measurement results, and generate a three-dimensional model using the extracted point cloud data.

[0211] The excluded area may also be excluded from the area for which additional distance measurement is performed in step S180 shown in FIG.

[0212] Fig. 26 shows examples of displaying a three-dimensional model when a spatial region is limited and three-dimensionally modeled according to other various modifications. Fig. 26 shows a three-dimensional model when a spatial region in the shape of a long passage is limited and three-dimensionally modeled. In Fig. 26, the region outside the scope of the three-dimensional modeling is indicated by a dashed line.

[0213] As shown in FIG. 26 , when a portion of the entire space is 3D modeled, the processing unit 25 may indicate the presence of the area excluded from the 3D modeling in the generated 3D model by changing the hue, brightness, saturation, color, texture, etc. In the example of FIG. 26 , the distant passage area and the area of ​​the open door are indicated by dashed lines as areas excluded from the 3D modeling. The area excluded from the 3D modeling (area not subject to 3D modeling) may be treated as a flat wall. Note that the indication of the area excluded from the 3D modeling may be switchable by a user instruction.

[0214] The distant passage area and the area of ​​the open door are areas that exist in the image captured by the image capturing unit 11 .

[0215] Fig. 27 is a diagram for explaining measurements when a direct measurement impossible area exists according to other various modified examples. For convenience, Fig. 27 shows an image P6 of an overhead view of the space in which the scan lidar 10 is installed. In Fig. 27, the position where the scan lidar 10 is installed is indicated by an x ​​mark.

[0216] As shown in FIG. 27 , when measurement is performed from the measurement position marked with an X, an area hidden by the pillar 621 (an area that cannot be directly measured in FIG. 27 ) occurs. In other words, due to the blind spot caused by the pillar 621, only a portion of the space can be measured from the measurement position. The processing unit 25 may visualize and display the area that cannot be directly measured in a captured image of the space or a three-dimensional model of the space. For example, in FIG. 27 , the space that cannot be directly measured is visualized using diagonal hatching. The area that cannot be directly measured within the space to be measured may be visualized, for example, by a two-dimensional floor plan from an upper viewpoint (e.g., a bird's-eye view).

[0217] Furthermore, the processing unit 25 may further superimpose on the bird's-eye view measurement position candidates, which are candidates for additional measurement positions for measuring areas that cannot be directly measured. In the example of Fig. 27, the measurement position candidates are indicated by circles. As a result, when the user moves the scanlidar 10 to the positions indicated by the circles and performs additional distance measurements, the processing unit 25 can generate a 3D model in which the areas that cannot be directly measured have also been measured.

[0218] Note that blind spots do not necessarily need to be measured. For example, if a blind spot (dead area) is an area within the camera's field of view that cannot be measured by a ranging device (e.g., LiDAR 14) due to its structure, the blind spot, which is not essential for the measurement purpose, does not need to be measured. This can shorten the measurement time. For example, when an area essential for the measurement purpose (e.g., an area on the overhead view) is acquired, the processing unit 25 determines whether the non-directly measurable area is included in the essential area. If at least a portion of the non-directly measurable area is included in the essential area (e.g., at least a portion of the non-directly measurable area overlaps on the overhead view), the processing unit 25 displays additional measurement position candidates for measuring the included area superimposed on the overhead view. If the non-directly measurable area is not included in the essential area (e.g., does not overlap on the overhead view), the processing unit 25 may not display the measurement position candidates, or may query the user as to whether or not to display the measurement position candidates.

[0219] In addition, due to constraints on the range of motion of the driving mechanism of the LiDAR 14 (for example, the scanner 13 can only move 300 degrees) and constraints on the scan lidar 10 itself (for example, blind spots caused by the relative positions of the imaging unit 11 and the LiDAR 14), blind spots that cannot be measured due to the structure may occur.

[0220] (Second Modification of Various Other Modifications) Fig. 28A is a diagram showing the configuration of the LiDAR 114 according to various other modifications. Fig. 28A is a diagram showing the configuration of the LiDAR 114 as viewed from above. The scan lidar 10 may be equipped with the LiDAR 114 instead of the LiDAR 14. Also, in Fig. 28A, for convenience, the laser light for distance measurement (distance measurement laser) is shown by diagonal hatching.

[0221] 28A , the LiDAR 114 includes a LiDAR sensor 114a, a half mirror 114b, and a finder camera 114c. The LiDAR sensor 114a, the half mirror 114b, and the finder camera 114c are housed, for example, inside a housing provided for the LiDAR 114. Furthermore, the positional relationship between the LiDAR sensor 114a, the half mirror 114b, and the finder camera 114c is fixed, and the LiDAR sensor 114a, the half mirror 114b, and the finder camera 114c are configured to move as a unit in conjunction with a drive mechanism provided for the LiDAR 114.

[0222] The LiDAR sensor 114a emits a ranging laser and measures the distance to an object based on information about the reflected light. The LiDAR sensor 114a has a laser light source that emits laser light, and an optical member and a light receiving element that receive the reflected light.

[0223] In the second modified example, the laser light includes light of a wavelength that can be captured by the finder camera 114c, that is, the laser light is visible in an image captured by the finder camera 114c.

[0224] The half mirror 114b is arranged on the optical path of the laser light from the LiDAR sensor 114a, transmits the laser light emitted from the LiDAR sensor 114a, and reflects a portion of the light incident on the LiDAR sensor 114a toward the viewfinder camera 114c.

[0225] The finder camera 114c is an imaging device that captures an image of the ranging point of the ranging laser of the LiDAR sensor 114a and its surroundings (i.e., an image near the ranging point). The LiDAR sensor 114a, the half mirror 114b, and the finder camera 114c are arranged so that the optical axis of the finder camera 114c and the optical axis of the LiDAR sensor 114a coincide with each other via the half mirror 114b. Note that the optical axis of the finder camera 114c and the optical axis of the LiDAR sensor 114a do not necessarily have to coincide with each other and may differ.

[0226] FIG. 28B is a diagram for explaining deviation of the distance measurement position according to other various modified examples.

[0227] As shown in Figure 28B, the distance measurement position on the image may differ from the calculated distance measurement position due to errors in the drive mechanism or the scan lidar 10. The distance measurement position on the image is the position where the laser light is actually irradiated, and can be obtained from the image captured by the finder camera 114c. The calculated distance measurement position is the position where the laser light is theoretically irradiated, and can be obtained by calculation in advance.

[0228] The processing unit 25 determines whether an error equal to or greater than a predetermined value exists between the ranging position on the image and the calculated ranging position, and automatically corrects the error if an error equal to or greater than the predetermined value exists. If an error equal to or greater than the predetermined value exists between the ranging position on the image and the calculated ranging position, the processing unit 25 at least corrects the error or notifies the user. The processing unit 25 may, for example, present to the user a correction value of the calculation parameter used in calculating the ranging position, which is estimated from the attitude parameters (pan, tilt, etc.) of the drive mechanism and the ranging value, or the necessity of correction.

[0229] Note that correction is a process in which the distance obtained by distance measurement is the distance to the position where the laser light is actually irradiated, and may involve, for example, replacing the distance measurement position (coordinate) of the obtained distance from the calculated distance measurement position (coordinate) with the distance measurement position (coordinate) on the image.

[0230] (Third Modification Among Other Various Modifications) FIG. 29A is a diagram showing an image in which the edges of a generated three-dimensional model have been converted onto image coordinates, according to other various modifications. FIG. 29A shows an image in which the edges of a three-dimensional model generated by measuring the distance to the space shown in the captured image P2 have been converted (projected) onto image coordinates. Images can be generated from three-dimensional models using differential rendering or the like. FIG. 29B is a diagram showing an edge image of the entire periphery when measured from inside the space, according to other various modifications. FIG. 29B shows an edge image in which the edges of the captured image P2 captured from inside the space have been emphasized. In FIGS. 29A and 29B, the boundaries of the room are indicated by white lines.

[0231] The processing unit 25 adjusts the geometric parameters of the three-dimensional model to match the captured image P2. The geometric parameters indicate information such as vertex coordinate values ​​that define the shape and the vertices that make up each face. In other words, the processing unit 25 corrects the geometric parameters so that the edges of the captured image and the edges of the generated model match (so that the position of boundary portion 631 in FIG. 29A matches or approaches the position of boundary portion 632 in FIG. 29B). This allows the position of boundary portion 631 of the room in the three-dimensional model to match the position of boundary portion 632 in the captured image P2, thereby generating a more accurate three-dimensional model.

[0232] (Fourth Modification Among Other Various Modifications) Fig. 30A is a diagram showing the relationship between the light ray direction for distance measurement and the normal direction of the measurement object when the light ray direction for distance measurement and the normal direction of the measurement object are nearly orthogonal, according to various other modifications. Fig. 30B is a diagram showing the spot shape of laser light when the light ray direction for distance measurement and the normal direction of the measurement object are nearly orthogonal, according to various other modifications. Fig. 30A is a diagram showing the position of the light source and the measurement object from above, and Fig. 30B is a diagram of the measurement object as viewed from the position of the light source.

[0233] As shown in Figure 30A, when the direction of the light beam for ranging and the normal direction of the object to be measured (the direction perpendicular to the surface of the object to be measured) are nearly perpendicular, the distances from the scan lidar 10 to adjacent ranging points (solid arrows and dashed arrows) can be significantly different.

[0234] As shown in Figure 30B, when the direction of the light beam for distance measurement and the normal direction of the object to be measured are nearly perpendicular to each other, the spot shape of the laser light becomes elongated. This means that the measured distance changes significantly when the distance measurement position is shifted. In other words, this means that the distance accuracy is likely to decrease when the distance measurement position is shifted.

[0235] If the ray direction of the ranging and the normal direction of the object to be measured are nearly perpendicular, the processing unit 25 may exclude at least one of them from the candidates for the ranging point, or may exclude the measured result (distance) from the data used to generate the three-dimensional model.

[0236] Fig. 31A is a diagram showing the relationship between the light ray direction for distance measurement and the normal direction of the measurement object when the light ray direction for distance measurement and the normal direction of the measurement object are nearly parallel, according to various other modified examples. Fig. 31B is a diagram showing the spot shape of laser light when the light ray direction for distance measurement and the normal direction of the measurement object are nearly parallel, according to various other modified examples. Fig. 31A is a diagram showing the position of the light source and the measurement object from above, and Fig. 31B is a diagram of the measurement object as viewed from the position of the light source.

[0237] As shown in Figure 31A, when the direction of the light beam for ranging and the normal direction of the object to be measured (the direction perpendicular to the surface of the object to be measured) are nearly parallel, the distances from the scan lidar 10 to adjacent ranging points (solid arrow and dashed arrow) will be close values.

[0238] As shown in Figure 31B, when the direction of the light beam for distance measurement is nearly parallel to the normal direction of the object to be measured, the spot shape of the laser light becomes circular. This means that even if the distance measurement position is shifted, the measured distance will be a similar value. In other words, this means that the distance accuracy is unlikely to decrease due to a shift in the distance measurement position.

[0239] For these reasons, it is desirable to determine a measurement point whose normal direction to the measurement object is nearly parallel to the direction of the light beam for ranging as the measurement point for generating a three-dimensional model. The processing unit 25 determines whether the direction of the light beam for ranging and the normal direction to the measurement object are nearly parallel, and if they are nearly parallel, determines the measurement point as the measurement point for generating a three-dimensional model.

[0240] The normal direction of the measurement object (i.e., the direction of the surface normal) may be determined from the spot shape of the laser light used for distance measurement, or from the spot shape of the laser light captured in the image captured simultaneously with distance measurement. The more the shape deviates from a circle and becomes more elliptical, the closer the distance measurement light direction and the normal direction of the measurement object are to being perpendicular. Furthermore, the normal direction of the measurement object (i.e., the direction of the surface normal) may be determined from the results of monocular depth estimation or monocular layout estimation.

[0241] (Fifth Modification Among Other Various Modifications) A process for converting a 3D model into BIM (Building Information Modeling) data will be described with reference to FIGS. 32 to 34. FIG. 32 is a flowchart showing the operation of converting to BIM data according to other various modifications. It is assumed that the processing unit 25 performs a process of identifying, from among a plurality of preset geometric models, a geometric model that best represents the geometric region recognized on the image by the recognition unit 21 and the point cloud data corresponding to the geometric region. The processing unit 25 determines, as the geometric model for the point cloud, a geometric model that minimizes the error (e.g., square error) between the shape of the geometric region and the point cloud corresponding to the geometric region and each of the plurality of geometric models. Clustering may be performed based on Euclidean distance, or another method may be used.

[0242] The processing unit 25 performs a conversion process on all generated geometric models into a format that can be described as a class or subclass of a structure in the BIM data. That is, the processing unit 25 converts the geometric models into the corresponding class or subclass of a structure in the BIM data. As will be described later with reference to FIG. 34 , the processing unit 25 converts, for example, a rectangular parallelepiped geometric model into a beam or column class, and a planar geometric model into a wall (or floor or ceiling) class. That is, the processing unit 25 determines that a rectangular parallelepiped geometric model corresponds to a beam or column in the real space, and that a planar geometric model corresponds to a wall (or floor or ceiling) in the real space. The processing unit 25 converts a horizontally elongated rectangular parallelepiped geometric model into a beam class, and converts a vertically elongated rectangular parallelepiped geometric model into a column class.

[0243] The structures of the BIM data may be described in, for example, an IFC (Industry Foundation Classes) format, and the classes or subclasses of the structures may correspond to the classes and subclasses of IfcBuiltElement. The IFC format is an example of a description format for CAD (Computer Aided Design) used in building information modeling.

[0244] 32, the processing unit 25 determines whether all geometric models have been converted into BIM data (S310). If all geometric models have been converted into BIM data (yes in S310), the processing unit 25 ends the processing. If there are any geometric models that have not been converted into BIM data (no in S310), the processing unit 25 selects any one geometric model from among the one or more geometric models that have not been subjected to conversion processing (S320).

[0245] Next, the processing unit 25 selects geometric models adjacent to the selected geometric model (S330). Adjacent may mean, for example, that two geometric models share a boundary with each other. Furthermore, the processing unit 25 may select, as adjacent geometric models, two geometric models that at least partially overlap or two geometric models that are within a predetermined distance from each other.

[0246] Next, the processing unit 25 determines whether the geometric model selected in step S320 and the geometric model selected in step S330 can be combined (S340). Combining means treating two or more geometric models as a single geometric model (i.e., a single structure). Combining may mean, for example, sharing a portion of two or more geometric models (e.g., sharing a boundary). Three-dimensional structures such as rectangular parallelepipeds or steel shapes are registered in a storage unit (not shown) in advance, and the processing unit 25 determines whether the shapes of two or more adjacent geometric models (shapes of the structures) match the registered shapes. If they match, the processing unit 25 determines that the models can be combined. Matching may mean that the shapes of the two or more geometric models completely match, or that the two shapes do not completely match but are within an acceptable error range.

[0247] For example, if there is a horizontally elongated geometric model and two vertically elongated geometric models adjacent to both ends of the horizontally elongated geometric model, and an H-beam is registered as a three-dimensional structure, the shape formed by the horizontally elongated geometric model and the two vertically elongated geometric models matches the H-beam, so the processing unit 25 determines to combine the horizontally elongated geometric model and the two vertically elongated geometric models.

[0248] The determination of whether to combine is automatically performed by the information processing device 20, but may also be made by, for example, a user. When the user makes the determination, the processing unit 25 causes the display unit 210 to display a determination screen that prompts the user to determine whether to combine, and accepts the user's input on the determination screen (input as to whether to combine). For example, the information processing device 20 may include a reception device that accepts input from the user. The reception device may be realized by, for example, a button, a keyboard, a sound collection device, a touch panel, or the like.

[0249] Next, if the processing unit 25 determines that the geometric model can be combined (yes in S340), that is, if it determines that the geometric model in question and the adjacent geometric model should be combined, it combines them and then returns to step S330 to continue processing. On the other hand, if the processing unit 25 determines that the combination is not possible (no in S340), that is, if it determines that the geometric model in question and the adjacent geometric model should not be combined, it applies the data units (S350).

[0250] In step S350, the processing unit 25 selects a description unit (such as a class) of the structure in the IFC according to the shape and posture of the structure generated by combining. For example, if the structure is a rectangular parallelepiped with its long axis in the vertical direction, the processing unit 25 selects the IFC class of "pillar." The selection of the description unit is performed automatically by the information processing device 20, but may also be selected by the user, for example. When the user is to select the description unit, the processing unit 25 displays a selection screen on the display unit 210 for selecting the description unit, and acquires the user's input on the selection screen (selection of which description unit to use) via the reception device. Note that description units will be described later with reference to FIG. 34.

[0251] Next, the processing unit 25 converts the geometric parameters, etc. in the geometric model into an expression in the data unit applied in step S350 (S360). The processing unit 25 converts the description of the position and shape according to the description unit of the structure using the coordinate equation and parameter range stored as the geometric model. For example, if a pillar is selected as the unit of the structure, the processing unit 25 converts the position information of the point cloud in the geometric model into the position, height, etc. of each vertex on the coordinate system. An example of the expression in the data unit is described in, for example, "Example of Geometric Shape Description of Structure Class" shown in Figure 34, which will be described later.

[0252] Fig. 33 is a diagram showing an example of BIM data according to other various modified examples. Fig. 33 shows an example of BIM data in the case where the structure (class of structure) is a "beam."

[0253] BIM data includes information such as the structure's ID, name, material, cross-sectional dimensions (width and height), length, strength (compressive strength and tensile strength), and installation location (number of floors and coordinates). In a BIM model, detailed information about each structure is managed, including not only its position and shape but also material information such as strength.

[0254] The structure is defined as belonging to the class of "beam," and an ID and name that can uniquely identify the structure are described. The beam attributes include the material, dimensions, and strength. Furthermore, the position of the structure within the building is described.

[0255] The width, height, and length are calculated from the X, Y, and Z coordinates included in the parameters of the geometric model. The X, Y, and Z coordinates are the coordinates of each axis in a three-dimensional Cartesian coordinate system.

[0256] In BIM data, IFC is used (recommended) as a data format for describing structures using specific numerical values. IFC is one of the file formats for CAD data models. IFC can define system specifications for all objects that make up a building, and can describe not only CAD data but also detailed data such as structure type, material, model number, etc. as attributes.

[0257] 34 is a diagram showing a table of correspondence between BIM classes (classes of BIM data) and geometric models according to other various modified examples. The table is set in advance and stored in a storage unit (not shown).

[0258] As shown in Fig. 34, the table summarizes the structure descriptions in BIM data or IFC format, their correspondence with geometric models, and measurement points that are often additionally measured to determine the geometric models. The expressions in the table, i.e., class names, geometric shape descriptions of the classes, etc., are examples. The table includes, as typical examples, the names of structure classes in BIM data or IFC, example geometric shape descriptions of the structure classes, corresponding geometric models, and typical additional measurement points.

[0259] The name of the structure class in the BIM data or IFC indicates the name of the structure class (BIM class) used in the BIM data or IFC. The name is set in advance.

[0260] The example of the geometric shape description of the structure class shows an example of the description format of the geometric shape of the structure identified as the structure class in question. Items of each size converted in step S360 are shown.

[0261] The corresponding geometric model indicates a geometric model corresponding to the name of the BIM data or IFC structure class.

[0262] Typical additional measurement points are examples of positions or regions of measurement points that are added to identify a geometric model. For example, in the case of a rectangular parallelepiped, in step S160 shown in FIG. 4, typically, the vicinity of the boundary line between a beam and a ceiling, a beam and a column, or a wall is determined as the measurement points for additional measurement.

[0263] In the table, the shape of the geometric model is set to correspond to the class of the BIM data. For example, the class of the BIM data and the shape of the geometric model have a one-to-one correspondence.

[0264] In this way, the information processing device 20 according to the fifth modification has information associating the class of BIM data with the shape of the geometric model, so that the class of BIM data can be identified from the geometric model. In addition, it can be said that the information processing device 20 determines, from the 3D point cloud information, which class of BIM data the shape corresponds to in the process of fitting the geometric model.

[0265] Figure 34 shows the correspondence between BIM classes, geometric models, and additional measurement points. For example, if at least one of the measurement points indicated by the additional measurement points for the target BIM class is measured in the additional measurement, it is considered that the geometric model or BIM class has been determined using the above method. Note that the table does not necessarily include typical additional measurement points.

[0266] (Other Embodiments) The information processing device 20 etc. according to one or more aspects has been described above based on Embodiments 1 and 2, Modifications 1 to 5 of Embodiment 1, and various other modifications (embodiments etc.), but the present disclosure is not limited to these embodiments etc. As long as it does not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art would conceive of to the present embodiment and embodiments constructed by combining components of different embodiments may also be included in the present disclosure.

[0267] For example, in the above embodiments, an example has been described in which the image capturing unit is a visible light camera, but the image capturing unit may be an infrared camera or a multispectral camera. When a far-infrared camera is used, it is possible to generate a three-dimensional model of a space even in complete darkness. Furthermore, when a multispectral camera is used, it is possible to generate a three-dimensional model that includes information such as the material properties of the surface of the measurement target. Furthermore, the image capturing unit may perform long-exposure photography. This makes it possible to create a three-dimensional model of a space even in a dark place.

[0268] Furthermore, the scan lidar according to the above-described embodiments may further include an acceleration sensor, and the vertical measurement results (e.g., the direction of gravity) of the acceleration sensor may be used as information for determining the vertical direction of the spatial model.

[0269] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0270] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0271] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0272] Furthermore, the information processing device 20 according to the above-described embodiments and the like may be realized as a single device or may be realized by multiple devices. When the information processing device 20 is realized by multiple devices, the components of the information processing device 20 may be distributed in any manner among the multiple devices. When the information processing device 20 is realized by multiple devices, the communication method between the multiple devices is not particularly limited, and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.

[0273] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Here, the term "LSI" is used, but depending on the level of integration, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. After LSI fabrication, a field programmable gate array (FPGA) that can be programmed or a reconfigurable processor that can reconfigure the connections or settings of circuit cells within the LSI may also be used. Furthermore, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.

[0274] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and is specifically a computer system comprising a microprocessor, ROM, RAM, etc. The ROM stores computer programs. The system LSI achieves its functions when the microprocessor operates in accordance with the computer programs.

[0275] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the information processing method shown in any of Figures 3, 4, 21, and 22.

[0276] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes.

[0277] The present disclosure is useful for devices that generate three-dimensional models.

[0278] 1 Information processing system 10, 100 Scan lidar 11 Photography unit 12 Control unit 13 Scanner 14, 114 LiDAR (ranging unit) 15 Signal processing unit 20 Information processing device 21 Recognition unit 22 Ranging point determination unit 23 Control planning unit 24 Consistency determination unit (determination unit) 25 Processing unit 50, 54, 55 Object 51, 52, 52a, 52b Geometric area 53 Accessory 53a Texture 61, 62, 63 Ranging point (measurement point) 70 3D mesh 114a LiDAR sensor 114b Half mirror 114c Finder camera 200 Mobile terminal 210 Display unit 300 Server 510 Ceiling surface (geometric area) 520 Wall surface (geometric area) 530 Desk surface (geometric region) 540 Floor surface (geometric region) 600, 601, 610, 611 Space 602 Door 621 Pillar (object) 622 Cursor 631, 632 Boundary portion D1 Depth estimation result M1, M2 3D model P1, P2, P2a, P3, P4 Captured image P5, P6 Image R1 Missing region R2 Region

Claims

1. An information processing device comprising: an acquisition unit that acquires a photographed image of a target area; a determination unit that determines measurement points corresponding to one or more geometric models that indicate the geometric shape of the target area based on the acquired photographed image; and a processing unit that acquires first distance information obtained by measuring the distance to the determined measurement points, and generates a three-dimensional model of the target area based on the acquired first distance information.

2. The information processing device described in claim 1, wherein when second distance information obtained by measuring additional measurement points necessary to determine the one or more geometric models is required in addition to the acquired first distance information, the processing unit acquires the second distance information and further generates the three-dimensional model based on the second distance information.

3. The information processing device according to claim 2, further comprising a determination unit that determines whether or not additional measurements are required for each of one or more geometric regions corresponding to one or more geometric models extracted from the captured image.

4. The information processing device according to claim 3, wherein the determination unit determines whether the additional measurement is necessary by determining whether at least one of the one or more geometric models is consistent with the first distance information for that geometric model.

5. An information processing device according to claim 4, wherein the determination unit determines measurement points for determining whether or not there is consistency as measurement points for the at least one geometric model in addition to measurement points corresponding to the geometric model.

6. An information processing device described in any one of claims 1 to 5, wherein the acquisition unit acquires third distance information obtained by measuring the distance to the target area, and the processing unit determines, based on the captured image and the third distance information, whether the measurement points included in the third distance information are sufficient for determining the one or more geometric models, and if it is determined that they are not sufficient, the first distance information is acquired.

7. The information processing device according to any one of claims 1 to 5, wherein the geometric model includes at least one of a plane, a curved surface, a rectangular parallelepiped, a straight line, and a curve.

8. An information processing device according to any one of claims 1 to 5, wherein the processing unit causes a display unit to display the three-dimensional model and the measurement points used to generate the three-dimensional model.

9. An information processing device according to any one of claims 1 to 5, wherein the processing unit estimates a distance distribution within the target area by estimating monocular depth, and the determination unit determines measurement conditions corresponding to each of the one or more geometric models based on the distance distribution.

10. The information processing device according to claim 9, wherein the determination unit determines the measurement conditions by at least one of increasing the spatial density of measurement points and increasing the number of measurements at the measurement points as the distance increases.

11. An information processing device according to any one of claims 1 to 5, wherein the determination unit determines measurement conditions corresponding to each of the one or more geometric models based on the intensity distribution within the target region.

12. An information processing device according to claim 11, wherein the determination unit determines the measurement conditions by at least one of increasing the spatial density of measurement points and increasing the number of measurements at the measurement points as the intensity decreases.

13. The information processing device according to claim 3, further comprising a recognition unit that extracts the one or more geometric regions for measurement based on image recognition of the image of the target region.

14. The information processing device according to claim 13, wherein the recognition unit excludes an area in the target area where a moving object exists from the one or more geometric areas.

15. An information processing device according to any one of claims 1 to 5, wherein the processing unit generates the three-dimensional model using the first distance information for a portion of the target area.

16. An information processing device according to any one of claims 1 to 5, wherein the processing unit outputs information indicating that an area of ​​the target area that is not a target area for generating the three-dimensional model is an area that is not a target area on the three-dimensional model.

17. An information processing device according to any one of claims 1 to 5, wherein, when there is an error of a predetermined value or more between the irradiation position of the laser light emitted by the distance measurement unit that measures the distance to the measurement point and the calculated irradiation position of the laser light emitted by the distance measurement unit, the processing unit performs at least one of correcting the error or notifying the user.

18. An information processing device according to any one of claims 1 to 5, wherein the processing unit, when a blind spot exists based on the positional relationship between the distance measurement unit that measures the distance to the measurement point and an object present in the target area, presents an additional measurement position for measuring the distance to the blind spot.

19. An information processing method comprising: acquiring a photographed image of a target area; determining measurement points corresponding to one or more geometric models that indicate the geometric shape of the target area based on the acquired photographed image; acquiring first distance information obtained by measuring the distances of the determined measurement points; and generating a three-dimensional model of the target area based on the acquired first distance information.

20. A program for causing a computer to execute the information processing method according to claim 19.

Images