Information processing method, information processing device, and program

Abstract

Provided are a device and a method for executing highly accurate alignment and integration processing of two three-dimensional models on different coordinate systems. A common identifiable structure in models is detected from each of two three-dimensional models on different coordinate systems, alignment of the two three-dimensional models is performed using the detected structure, and integration processing of the two three-dimensional models is executed using the alignment result. The common identifiable structure in models is, for example, a structure such as a window that can be identified from both sides of a joint surface of the two three-dimensional models, and the integration processing of the two three-dimensional models is executed using a coordinate transformation matrix calculated on the basis of the fact that the location of the structure in each of the coordinate systems is identical in real space.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program, and more particularly to an information processing method, an information processing device, and a program that generate a single three-dimensional model by performing high-precision alignment of multiple individually generated three-dimensional models.

[0002] In recent years, with the widespread use of high-precision cameras and sensors and advances in image analysis technology, three-dimensional models (3D models) of various spaces have been generated and used. For example, 3D models of outdoor spaces such as urban areas, as well as many 3D models of indoor spaces such as the interiors of various buildings and residential spaces, have been generated and used.

[0003] Many outdoor 3D models, such as 3D maps, are created by various private companies and public organizations and are freely available for use. For indoor spaces such as apartment buildings and offices, many services are also available that allow users to view 3D models created by housing developers and other entities, and check the detailed interior configuration.

[0004] Furthermore, it is expected that the use of robots that can move autonomously inside and outside buildings, such as parcel delivery robots, will become more common in the future. With this in mind, it is anticipated that it will become necessary to generate and use a single 3D model that integrates outdoor and indoor 3D models.

[0005] However, conventional general indoor 3D models are often generated as 3D models in their own individually defined local coordinate systems. That is, 3D models of specific areas, such as indoor spaces in apartment buildings and offices, are often 3D models that use different local coordinate systems, and in many cases, these use coordinate systems that are different from the coordinate systems (e.g., geocentric Cartesian coordinate systems) of 3D models, such as 3D maps created by private companies and public organizations.

[0006] As a method for aligning two pieces of three-dimensional shape data, for example, the Iterative Closest Point (ICP) method is known. This ICP method is an algorithm for stitching together three-dimensional models based on point clouds obtained from multiple measurements, and when two pieces of point cloud data corresponding to two three-dimensional models are given, this algorithm estimates rotation and translation that minimizes the distance between corresponding points of the two three-dimensional models.

[0007] However, this method is an algorithm that assumes that multiple 3D models have corresponding points, i.e., 3D data at the same position, and therefore cannot be applied to the alignment or integration of 3D models in which both models do not have corresponding data at the same points, such as a 3D model of the exterior wall of a building and a 3D model of the interior of each individual room within the building.

[0008] Furthermore, Patent Document 1 (JP 2016-527574 A) discloses a method for aligning multiple three-dimensional models. However, the method disclosed in this document requires manual specification of points to be used as references for alignment, which results in problems such as increased processing time and costs when a large amount of data needs to be processed.

[0009] Special table 2016-527574 publication

[0010] The present disclosure has been made in consideration of, for example, the above-mentioned problems, and aims to provide an information processing method, an information processing device, and a program that generate a single 3D model by performing high-precision alignment of multiple individually generated 3D models, such as a combination of an outdoor 3D model (3D model) and an indoor 3D model (3D model).

[0011] A first aspect of the present disclosure resides in an information processing method in which a data processing unit of an information processing device executes a model-commonly identifiable structure detection step of detecting a model-commonly identifiable structure from each of two three-dimensional models, an alignment step of aligning the two three-dimensional models using the model-commonly identifiable structures, and a three-dimensional model integration step of performing integration processing of the two three-dimensional models using the alignment results of the two three-dimensional models.

[0012] Furthermore, a second aspect of the present disclosure resides in an information processing device having: a model-commonly identifiable structure detection unit that detects a model-commonly identifiable structure from each of two three-dimensional models; a three-dimensional model alignment execution unit that aligns the two three-dimensional models using the model-commonly identifiable structure; and a three-dimensional model integration processing execution unit that executes integration processing of the two three-dimensional models using the alignment result of the two three-dimensional models.

[0013] Furthermore, a third aspect of the present disclosure is a program for executing information processing in an information processing device, the program causing a data processing unit of the information processing device to execute a model-commonly identifiable structure detection step for detecting a model-commonly identifiable structure from each of two three-dimensional models, an alignment step for aligning the two three-dimensional models using the model-commonly identifiable structures, and a three-dimensional model integration step for performing integration processing of the two three-dimensional models using the alignment results of the two three-dimensional models.

[0014] The program of the present disclosure is, for example, a program that can be provided in a computer-readable format via a storage medium or a communication medium to an information processing device or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.

[0015] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0016] According to an embodiment of the present disclosure, an apparatus and method are realized that perform highly accurate alignment and integration of two 3D models in different coordinate systems. Specifically, for example, a commonly identifiable structure is detected from each of two 3D models in different coordinate systems, the two 3D models are aligned using the detected structure, and the alignment results are used to integrate the two 3D models. The commonly identifiable structure is a structure, such as a window, that can be identified from both sides of the interface between the two 3D models. The two 3D models are integrated using a coordinate transformation matrix calculated based on the alignment of the coordinate systems of this structure in real space. This configuration realizes an apparatus and method that perform highly accurate alignment and integration of two 3D models in different coordinate systems. Note that the effects described in this specification are merely exemplary and not limiting, and additional effects may also be present.

[0017] 1 is a diagram illustrating problems with integration processing of three-dimensional models in different coordinate systems. FIG. 1 is a diagram illustrating problems with integration processing of three-dimensional models in different coordinate systems. FIG. 2 is a diagram illustrating an outline of processing executed by an information processing device of the present disclosure. FIG. 3 is a diagram illustrating an example in which a window, which is a model commonly identifiable structure, is extracted from an outdoor three-dimensional model A. FIG. 4 is a diagram illustrating an example in which a window, which is a model commonly identifiable structure, is extracted from an indoor three-dimensional model B. FIG. 5 is a diagram illustrating a specific example of a position adjustment processing of three-dimensional models using a window, which is a model commonly identifiable structure. FIG. 6 is a diagram illustrating a specific example of a position adjustment processing of three-dimensional models using a door, which is a model commonly identifiable structure. FIG. 7 is a flowchart illustrating an outline of a processing sequence executed by an information processing device of the present disclosure. FIG. 8 is a block diagram illustrating an example of a configuration of an information processing device of the present disclosure. FIG. 9 is a diagram illustrating a flowchart illustrating a detailed sequence of processing executed by a data processing unit of an information processing device of the present disclosure. FIG. 10 is a diagram illustrating an object DB. FIG. 11 is a diagram illustrating a detailed sequence of processing executed by a data processing unit of an information processing device of the present disclosure. FIG. 11 is a diagram illustrating a detailed sequence of processing for integrating two three-dimensional models executed by a data processing unit of an information processing device of the present disclosure. FIG. 12 is a diagram illustrating a specific example of a position coordinate acquisition processing executed in integration processing of two three-dimensional models. FIG. 1 is a diagram illustrating a specific example of a position coordinate acquisition process executed when integrating two three-dimensional models. FIG. 2 is a diagram illustrating a specific example of a coordinate transformation matrix calculation process executed when integrating two three-dimensional models. FIG. 3 is a diagram illustrating a specific example of a coordinate transformation matrix calculation process executed when integrating two three-dimensional models. FIG. 4 is a diagram illustrating a flowchart illustrating a detailed sequence of a learning model update process executed by a data processing unit of an information processing device of the present disclosure. FIG. 5 is a diagram illustrating a specific example of an error in semantic segmentation to which a learning model is applied. FIG. 6 is a diagram illustrating an example of a hardware configuration of an information processing device of the present disclosure.

[0018] The information processing method, information processing device, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items: 1. Problems with integration processing of 3D models in different coordinate systems 2. Overview of processing executed by the information processing device of the present disclosure 3. Overview of processing sequence executed by the information processing device of the present disclosure 4. Example configuration of the information processing device of the present disclosure 5. Details of processing executed by the data processing unit of the information processing device of the present disclosure 6. Example hardware configuration of the information processing device of the present disclosure 7. Summary of the configuration of the present disclosure

[0019] [1. Problems with Integration Processing of Three-Dimensional Models in Different Coordinate Systems] First, problems with integration processing of three-dimensional models in different coordinate systems will be described.

[0020] As explained above, in recent years, with the widespread use of high-precision cameras and sensors and advances in image analysis technology, three-dimensional models (3D models) of various spaces are being generated and used.

[0021] 1 shows two examples of three-dimensional models. Specifically, the following two three-dimensional models are shown: (A) a three-dimensional model A of an outdoor space such as an urban area, and (B) a three-dimensional model B of an indoor space such as an apartment building or office. The three-dimensional model B of the indoor space 20 shown on the right side of FIG. 1 is a three-dimensional model of one room inside one building 10 in the outdoor three-dimensional model A shown on the left side of FIG. 1.

[0022] For example, when a parcel delivery robot or the like is made to travel automatically, a process is executed in which a safe travel route is determined by referring to a three-dimensional map that accurately reflects the travel route, i.e., a three-dimensional model.

[0023] A robot that travels only in outdoor spaces such as urban areas can travel safely while detecting a safe outdoor route by referring to (A) outdoor three-dimensional model A shown in Fig. 1. On the other hand, a robot that travels only indoors 20 shown in Fig. 1 can travel safely while detecting a safe indoor route by referring to (B) indoor three-dimensional model B shown on the right side of Fig. 1.

[0024] However, a robot that moves autonomously inside and outside a building needs to switch between and refer to both outdoor 3D model A and indoor 3D model B. For example, if an autonomous mobile robot travels outdoors using outdoor 3D model A, and then enters the interior of building 10 of outdoor 3D model A shown in Fig. 1 and travels within one of the rooms therein, it needs to switch between the two 3D models shown in Fig. 1, i.e., (A) 3D model A of an outdoor space such as an urban area, and (B) 3D model B of an indoor space such as an apartment building or office, at the boundary between the outdoors and the indoors.

[0025] When switching between three-dimensional models (three-dimensional maps), it is necessary to newly execute self-position estimation processing to analyze where the robot's current position is in the newly used three-dimensional model (three-dimensional map), direction estimation processing to confirm which direction the robot is facing, etc. In this way, when the three-dimensional model (three-dimensional map) is switched, new processing is required to analyze the correspondence between the new three-dimensional model and the robot itself, and the processing time required for this significantly reduces the efficiency of package delivery and driving processing.

[0026] To improve efficiency, the two 3D models (A) and (B) shown in FIG. 1 should be unified into one 3D model.

[0027] However, many outdoor 3D models, such as an outdoor 3D map as shown in FIG. 1A, are often generated using a global coordinate system (coordinate system a shown in FIG. 1), such as a geocentric Cartesian coordinate system.

[0028] On the other hand, indoor 3D models such as those shown in Fig. 1B are often generated using unique, individually defined local coordinate systems. That is, 3D models of specific areas, such as indoor spaces in apartment buildings and offices, are often 3D models that use different, unique local coordinate systems (coordinate system b shown in Fig. 1).

[0029] In the example shown in Fig. 1, the local coordinate system (coordinate system b shown in Fig. 1) is a coordinate system in which the origin is set at one vertex of the rectangular floor of the indoor space 20, the constituent sides of the floor are set as the X-axis and Z-axis, and the vertically upward direction is set as the Y-axis. Note that the local coordinate system only needs to be able to identify the three-dimensional position of an object in a specific three-dimensional space, for example, a limited three-dimensional space such as the indoor space 20 shown in Fig. 1, and various settings are possible for the origin position and the X, Y, and Z axes.

[0030] The local coordinate system (coordinate system b shown in FIG. 1) that defines the three-dimensional space indoors 20 shown in FIG. 1 is a coordinate system that is completely different from the global coordinate system (coordinate system a shown in FIG. 1) that defines the three-dimensional space outdoors shown in FIG. 1.

[0031] Therefore, when integrating the indoor 3D model B shown in Fig. 1(B) with the outdoor 3D model A shown in Fig. 1(A), it is necessary to analyze the position and angle at which the indoor 3D model B shown in Fig. 1(B) is located relative to the outdoor 3D model A shown in Fig. 1(A). However, it is difficult to align the positions of two 3D models using different individual coordinate systems.

[0032] A specific example of the alignment process for two individually generated 3D models will be described with reference to Fig. 2. Fig. 2 shows an example of the alignment process for the two 3D models described with reference to Fig. 1, namely, (A) 3D model A of an outdoor space such as an urban area, and (B) 3D model B of an indoor space such as an apartment building or office.

[0033] The outdoor three-dimensional model A shown in Fig. 2(A) is a three-dimensional model generated using a global coordinate system (coordinate system a). On the other hand, the indoor three-dimensional model B shown in Fig. 2(B) is a three-dimensional model generated using a local coordinate system (coordinate system b shown in Fig. 2) that is different from the global coordinate system (coordinate system a).

[0034] Assume that it has been analyzed that the 3D model B of the indoor space is a part of the interior space of the building 10 shown in the 3D model A of the outdoor space. That is, it has been analyzed that the "area 21 corresponding to the indoor 3D model B" is located somewhere in the "wall area 11 of the building 10 of the outdoor 3D model A," as shown in the center of Figure 2.

[0035] However, it is difficult to analyze where the correct position of the "area 21 corresponding to indoor three-dimensional model B" is in the "wall area 11 of building 10 of outdoor three-dimensional model A."

[0036] As mentioned above, the ICP (Iterative Closest Point) method is known as a method for aligning two pieces of three-dimensional shape data. The ICP method is an algorithm for stitching together three-dimensional models based on point clouds obtained from multiple measurements, and when two sets of point cloud data corresponding to two three-dimensional models are given, the ICP method estimates rotation and translation that minimizes the distance between corresponding points in the two three-dimensional models.

[0037] However, this method is an algorithm that assumes that multiple 3D models have corresponding points, i.e., 3D data at the same position, and therefore cannot be applied to aligning 3D models in which both models do not have corresponding data at the same points, such as a 3D model of the exterior wall of a building and a 3D model of the interior of each individual room within the building.

[0038] 2. Overview of Processing Executed by Information Processing Device of the Present Disclosure Next, an overview of processing executed by the information processing device of the present disclosure will be described.

[0039] An overview of the processing executed by the information processing device of the present disclosure will be described with reference to Figure 3 and subsequent figures. Figure 3 shows two three-dimensional models, as described with reference to Figures 1 and 2. That is, the following two three-dimensional models: (A) a three-dimensional model A of an outdoor space such as an urban area, and (B) a three-dimensional model B of an indoor space such as an apartment building or office. The three-dimensional model B of the indoor space 20 shown on the right side of Figure 3 is a three-dimensional model of one room inside one building 10 in the outdoor three-dimensional model A shown on the left side of Figure 3.

[0040] The outdoor three-dimensional model A shown in Fig. 3(A) is a three-dimensional model generated using a global coordinate system (coordinate system a). On the other hand, the indoor three-dimensional model B shown in Fig. 3(B) is a three-dimensional model generated using a local coordinate system (coordinate system b shown in Fig. 3) that is different from the global coordinate system (coordinate system a).

[0041] The information processing device disclosed herein performs a process of integrating these two 3D models on different coordinate systems into a 3D model on a single coordinate system, and performs a highly accurate alignment process of the two 3D models for this integration process.

[0042] The information processing device of the present disclosure first extracts "model commonly identifiable structures" from the two 3D models to be integrated in order to perform highly accurate alignment processing of the two 3D models to be integrated. Figure 3 is a diagram showing a specific example of the extraction processing of "model commonly identifiable structures" executed by the information processing device of the present disclosure.

[0043] In the example shown in Fig. 3 , the model-common identifiable structure 50 is a window 52. The window 52 is an identifiable structure in the outdoor three-dimensional model A shown in Fig. 3(A) and also an identifiable structure in the indoor three-dimensional model B shown in Fig. 3(B). In other words, the window 52 is a structure that can be identified from the exterior wall of the building 10 in the outdoor three-dimensional model A shown in Fig. 3(A) and also from the interior of the indoor 20 in the indoor three-dimensional model B shown in Fig. 3(B).

[0044] In this way, the information processing device of the present disclosure first extracts from the two 3D models to be integrated identifiable structures that are common to these two 3D models, i.e., model common identification structures 50. The information processing device of the present disclosure aligns the two 3D models to be integrated using the extracted model common identification structures 50.

[0045] 4 shows an example in which a window 52, ​​which is a model commonly identifiable structure 50, is extracted from an outdoor three-dimensional model A. Note that the process of detecting a specific structure, which is a model commonly identifiable structure 50, from the three-dimensional model A can be performed using, for example, semantic segmentation processing or the like.

[0046] Semantic segmentation is a process that uses pre-generated learning data to determine the type of object to which each pixel belongs on a pixel-by-pixel basis that makes up an image, and divides pixel regions according to the object type.

[0047] For example, the types of various objects included in an image, such as buildings, roads, vehicles, people, trees, windows, doors, roofs, walls, etc., are determined for each pixel region of the image. Note that in the processing of the present disclosure, the image that is the target of semantic segmentation processing is a 3D model image, and by performing semantic segmentation on the 3D model image, the types of various objects that make up the 3D model are determined.

[0048] The process of detecting a specific structure that is a model commonly identifiable structure 50 from the three-dimensional model A is not limited to semantic segmentation processing, and other processing may be used.

[0049] For example, if the model commonly identifiable structure 50 is a window, the optical transparency of windows is significantly different from that of other structural components, and the detection values ​​of various sensors used in the generation stage of the three-dimensional model can be analyzed to detect the window area, and the detection information can be recorded as attribute information of the three-dimensional model and used.

[0050] For example, when creating a three-dimensional model using images captured by a drone camera, an ultrasonic sensor attached to the drone can detect the presence of a surface, and the detection information from sensors such as LiDAR and ToF sensors can be analyzed to determine whether it is an optically transparent area. Areas that are both surface areas and light-transmitting areas can be determined to be window areas, and this determination information can be recorded as attribute information corresponding to each component included in the three-dimensional model and used.

[0051] FIG. 5 shows an example in which a window 52, ​​which is a model commonly identifiable structure 50, is extracted from an indoor three-dimensional model B.

[0052] This process can also be performed using the semantic segmentation process described above. Furthermore, as described above, by utilizing the fact that the optical transmittance of windows is significantly different from that of other structural components, a window region may be detected from sensor information such as a LiDAR or ToF sensor during the generation of the 3D model, and the detected information may be recorded as attribute information of the 3D model and used.

[0053] In addition to the above-described process, the process of detecting a model-commonly identifiable structure 50 from a three-dimensional model may also be performed as a process in which, for example, a user displays two three-dimensional models on a display, observes and compares them, and detects them.

[0054] FIG. 6 is a diagram showing a specific example of a three-dimensional model alignment process using a window 52, ​​which is a model-commonly identifiable structure 50 extracted from both two three-dimensional models to be aligned and integrated, i.e., an outdoor three-dimensional model A and an indoor three-dimensional model B.

[0055] As shown in Figure 6, a window 52, ​​which is a model-commonly identifiable structure 50 extracted from outdoor 3D model A, and a window 52, ​​which is a model-commonly identifiable structure 50 extracted from indoor 3D model B, are the same structure, and by positioning the two 3D models so that the position of this window 52 matches, it is possible to align the 3D models in two different coordinate systems with high precision.

[0056] 6 shows the "wall area 11 of the building 10 of the outdoor three-dimensional model A" and the "indoor three-dimensional model B constituent area 21." The information processing device of the present disclosure arranges the two three-dimensional models so that the position of the window 52, ​​which is a model commonly identifiable structure 50 in the "wall area 11 of the building 10 of the outdoor three-dimensional model A," coincides with the position of the window 52, ​​which is a model commonly identifiable structure 50 in the "indoor three-dimensional model B constituent area 21."

[0057] The information processing device of the present disclosure thus aligns the two 3D models using the model-commonly identifiable structures 50 extracted from the 3D models in two different coordinate systems, and then uses the alignment results to integrate the two 3D models, i.e., integrate them into a 3D model in one of the coordinate systems. Details of this process will be described later.

[0058] The "model-commonly identifiable structure 50" to be detected by the information processing device of the present disclosure is, for example, a structure that can be identified from both sides of the joint surface (boundary surface) of two 3D models to be integrated. In the example shown in Fig. 6, the wall surface of the building 10 in the outdoor 3D model A and one wall surface of the indoor 3D model B correspond to the joint surface (boundary surface) of the two 3D models, and the window 52, ​​which is the model-commonly identifiable structure 50, is a structure that can be identified from both sides of the joint surface (boundary surface).

[0059] In this way, structures that can be identified from both sides of the joint surface (boundary surface) of the two 3D models to be integrated are detected as "model common identification structures 50," and the detected "model common identification structures 50" are applied to aligning the two 3D models. This type of processing makes it possible to align the two 3D models with high accuracy, even if they do not have overlapping areas.

[0060] The aforementioned ICP (Iterative Closest Point) method is also known as a method for aligning two 3D models, but ICP detects overlapping areas that have 3D data at the same position from the two 3D models and performs alignment using the detected overlapping areas. Therefore, if no overlapping areas are detected between the two 3D models to be integrated, ICP cannot be applied.

[0061] In contrast, the process of the present disclosure enables highly accurate alignment even when there are no overlapping areas between the two 3D models to be integrated. The process of the present disclosure detects structures that can be identified from both sides of the joint surface (boundary surface) of the two 3D models to be integrated as "model-common identification structures 50," and applies the detected "model-common identification structures 50" to the alignment of the two 3D models. Therefore, highly accurate alignment is possible even for two 3D models that do not have overlapping areas, and highly accurate integration of the two 3D models is possible based on the highly accurate alignment.

[0062] 3 to 6 are examples in which one of the 3D models to be integrated is a 3D model using a global coordinate system and the other is a 3D model using a local coordinate system. The 3D model integration process executed by the information processing device of the present disclosure is not limited to such a combination, and can also be applied to integration of two 3D models each using different local coordinates.

[0063] Furthermore, in the processing example described with reference to Figures 3 to 6, an example was described in which the model-commonly identifiable structure 50 was a window 52, ​​but the model-commonly identifiable structure 50 extracted from the two 3D models to be integrated is not limited to a window, and various structures can be used.

[0064] FIG. 7 is a diagram showing a specific example of the integration process of two three-dimensional models each using different local coordinates.

[0065] Figure 7 shows the following two taste reduction models: (A) Three-dimensional model A using local coordinate a (B) Three-dimensional model B using local coordinate b The local coordinate a and the local coordinate b are coordinates set differently.

[0066] As shown in the figure, (A) three-dimensional model A is, for example, a three-dimensional model of a living room 60, and (B) three-dimensional model B is a three-dimensional model of an entrance hall 70 adjacent to the living room 60. These two three-dimensional models are three-dimensional models generated separately, and are three-dimensional models on different local coordinate systems.

[0067] The process of the present disclosure can also be applied to the case where two 3D models on different local coordinate systems are integrated. In the example shown in Fig. 7, the model commonly identifiable structure 50 extracted from the two 3D models to be integrated is a door 54.

[0068] As shown in FIG. 7, the door 54 exists in the three-dimensional model A of the living room 60 and also in the three-dimensional model B of the entrance 70 .

[0069] This door 54 is extracted as a model commonly identifiable structure 50 from the 3D models in the two different coordinate systems to be integrated, and the extracted door 54 is used to align the two 3D models using the same process as previously described with reference to Fig. 6. Furthermore, this alignment result can be used to integrate the two 3D models, i.e., to integrate them into a 3D model in one of the coordinate systems.

[0070] In this way, the information processing device of the present disclosure detects structures that can be identified from both sides of the joint surface (boundary surface) of the two three-dimensional models to be integrated as ``model common identification structures 50,'' and uses the detected ``model common identification structures 50'' to align the two three-dimensional models.

[0071] [3. Overview of the processing sequence executed by the information processing device of the present disclosure] Next, an overview of the processing sequence executed by the information processing device of the present disclosure will be described with reference to the flowchart shown in Fig. 8. Note that the processing according to the flow shown in Fig. 8 is executed under the control of a control unit (data processing unit) composed of a CPU or the like having a program execution function, in accordance with a program stored in an internal memory of the information processing device of the present disclosure. Below, the processing of each step of the flow shown in Fig. 8 will be described in order.

[0072] (Step S101) First, in step S101, the data processing unit of the information processing device inputs two three-dimensional models to be aligned and integrated, namely, a three-dimensional model A (3D model A) and a three-dimensional model B (3D model B).

[0073] These three-dimensional models may be three-dimensional models generated in advance in an external device, or may be three-dimensional models generated by an information processing device that performs processing according to this flow.

[0074] The two 3D models to be integrated, 3D model A and 3D model B, are 3D models based on different coordinate systems. For example, 3D model A is a 3D model based on a global coordinate system, and 3D model B is a 3D model based on a local coordinate system. Alternatively, 3D model A is a 3D model based on local coordinate system a, and 3D model B is a 3D model based on local coordinate system b that is different from local coordinate system a.

[0075] (Step S102) Next, in step S102, the data processing unit of the information processing device detects model commonly identifiable structures (=model alignment structures) from the three-dimensional model A and three-dimensional model B input in step S101.

[0076] This process corresponds to the detection process of the window 52, ​​which is the model commonly identifiable structure 50, described above with reference to Figures 3 to 6. It also corresponds to the detection process of the door 54, which is the model commonly identifiable structure 50 in the example described with reference to Figure 7.

[0077] The model commonly identifiable structures detected from the two three-dimensional models to be integrated are structures used to align the two models, and correspond to model alignment structures.

[0078] (Step S103) Next, in step S103, the data processing unit of the information processing device performs alignment processing of the three-dimensional model A and the three-dimensional model B, taking advantage of the fact that the real-space positions of the model-commonly identifiable structures extracted from the three-dimensional model A and the three-dimensional model B in step S102 are almost identical.

[0079] This process corresponds to the process described above with reference to Fig. 6. That is, in the example shown in Fig. 6, by utilizing the fact that the real-space positions of the window 52, ​​which is a model-commonly identifiable structure extracted from the three-dimensional model A and the three-dimensional model B, are approximately the same, the two three-dimensional models are arranged so that the positions of the windows of the three-dimensional model A and the three-dimensional model B match, and the registration process of the three-dimensional models A and B is performed.

[0080] (Step S104) Next, in step S104, the data processing unit of the information processing device converts the coordinate system b of three-dimensional model B into the coordinate system a of three-dimensional model A based on the alignment result of three-dimensional model A and three-dimensional model B executed in step S103, and executes a process of combining three-dimensional model B with three-dimensional model A.

[0081] That is, one three-dimensional model A+B is generated by adding the three-dimensional configuration of the three-dimensional model B to the three-dimensional configuration of the three-dimensional model A.

[0082] In step S104, it is necessary to perform coordinate conversion processing to convert the coordinate system b of the three-dimensional model B into the coordinate system a of the three-dimensional model A. A specific example of this coordinate conversion processing will be described in detail later.

[0083] The above has described an overview of the sequence of the alignment and integration process of two different 3D models executed by the information processing device of the present disclosure. As described above, the information processing device of the present disclosure detects a commonly identifiable structure from two 3D models in different coordinate systems, aligns the two 3D models using the detected commonly identifiable structure, and then integrates the two 3D models using the alignment result.

[0084] The information processing device disclosed herein performs highly accurate alignment by aligning two 3D models using commonly identifiable structures such as windows and doors extracted from the two 3D models, thereby enabling more accurate integrated processing of the 3D models.

[0085] 4. Configuration Example of Information Processing Apparatus of the Present Disclosure Next, a configuration example of an information processing apparatus of the present disclosure will be described.

[0086] 9 is a block diagram showing an example configuration of an information processing device according to the present disclosure. As shown in FIG. 9, the information processing device 100 according to the present disclosure includes a data processing unit 101, a three-dimensional model (3D model) storage unit 131, and a learning model storage unit 132 for structure type identification processing.

[0087] 9 is a block diagram illustrating the main components used in the processing of the present disclosure. In addition to the components shown in FIG. 9, the information processing device 100 has general components that an information processing device such as a PC has, such as an input unit that allows user input, a display unit that displays images, etc., a speaker that outputs audio, etc., a storage unit that stores processing programs, etc., a memory, etc. An example hardware configuration of the information processing device 100 of the present disclosure will be described later.

[0088] As shown in Figure 9, the data processing unit 101 has a three-dimensional model acquisition unit (generation unit) 121, a structure type identification processing execution unit 122, a three-dimensional model joint area analysis unit 123, a model commonly identifiable structure detection unit 124, a three-dimensional model alignment execution unit 125, a three-dimensional model integration processing execution unit (coordinate transformation execution unit) 126, and a learning model update unit for structure type identification processing (learning processing execution unit) 127.

[0089] In practice, the data processing unit 101 can be configured by a processor such as a CPU having a program execution function, and each processing unit in the data processing unit 101 shown in Fig. 9 can be executed by an application program. In Fig. 9, to make it easier to understand the processing executed in the data processing unit 101, the processing executed by the application program is shown as blocks of each processing unit.

[0090] A description will be given of each of the components of the information processing device 100 shown in Fig. 9. The three-dimensional model (3D model) storage unit 131 is a storage unit that stores various three-dimensional models.

[0091] For example, various three-dimensional models are stored, such as (A) a three-dimensional model A of an outdoor space such as an urban area, and (B) a three-dimensional model B of an indoor space such as an apartment building or office, as previously described with reference to Figure 1 etc. Each three-dimensional model is a three-dimensional model on its own unique coordinate system.

[0092] The three-dimensional model stored in the three-dimensional model (3D model) storage unit 131 may be a three-dimensional model generated by the information processing device 100 or may be a three-dimensional model generated by an external device other than the information processing device 100 .

[0093] The structure type identification processing learning model storage unit 132 is a memory unit that stores a learning model for identifying the type of structure, such as a window or a door, from a 3D model. As described above, the process of identifying the type of structure, such as a window or a door, from a 3D model uses, for example, semantic segmentation processing.

[0094] As mentioned above, semantic segmentation is a process that uses pre-generated learning data to determine the type of object (window, door, etc.) to which each pixel belongs on a pixel-by-pixel basis that makes up an image, and divides pixel regions according to the object type.

[0095] The structure type identification processing learning model storage unit 132 stores the learning model used in this semantic segmentation processing.

[0096] Next, a description will be given of each component of the data processing unit 101. A three-dimensional model acquisition unit (generation unit) 121 acquires two three-dimensional models to be integrated from three-dimensional models stored in a three-dimensional model (3D model) storage unit 131.

[0097] For example, as described with reference to Figure 1 and other figures, two three-dimensional models are acquired: (A) a three-dimensional model A of an outdoor space such as an urban area, and (B) a three-dimensional model B of an indoor space such as an apartment building or office.

[0098] The three-dimensional model acquisition unit (generation unit) 121 may be configured to generate a three-dimensional model by itself using an image captured by a camera, etc. In other words, the three-dimensional model generated by the data processing unit 101 of the information processing device 100 may be set as the three-dimensional model to be integrated.

[0099] The structure type identification process execution unit 122 executes a process of identifying the type of structure in each of the two 3D models to be integrated that are acquired or generated by the 3D model acquisition unit (generation unit) 121 .

[0100] The process executed by the structure type identification process execution unit 122 is, for example, the semantic segmentation process described above. The structure type identification process execution unit 122 executes the semantic segmentation process using the structure type identification process learning model stored in the structure type identification process learning model storage unit 132.

[0101] As described above, the semantic segmentation process is a process that uses pre-generated learning data to determine the type of object (window, door, etc.) to which each pixel belongs for each pixel that constitutes an image, and then divides the pixel region according to the object type. The structure type identification process execution unit 122 executes this process.

[0102] The three-dimensional model joint region analysis unit 123 roughly aligns the two three-dimensional models to be integrated, that is, analyzes the rough joint position of the two three-dimensional models.

[0103] As described above, the information processing device 100 of the present disclosure detects structures that can be identified from both sides of the joint surface (boundary surface) of two 3D models to be integrated as "model-common identification structures," and applies the detected "model-common identification structures" to the alignment of the two 3D models. The 3D model joint region analysis unit 123 performs processing to identify the rough joint position of the two 3D models as a region for detecting this "model-common identification structure."

[0104] For example, in the example shown in Figure 6, the wall surface of building 10 in outdoor 3D model A and one wall surface in indoor 3D model B correspond to the junction surface (boundary surface) of the two 3D models, and the 3D model junction area analysis unit 123 analyzes the approximate junction position of outdoor 3D model A and indoor 3D model B at this junction surface.

[0105] The analysis process for the rough joining position of two 3D models to be integrated can utilize position information such as GPS position information associated with each pixel or pixel region of the two 3D model images to be integrated. The GPS position information for each pixel or pixel region of the 3D model images is recorded as attribute information when the 3D models are generated.

[0106] If there is no location data corresponding to the three-dimensional models, such as GPS location information, the information processing device 100 may be configured to display two three-dimensional models on the display unit, and the user may analyze the approximate joint position of the two displayed three-dimensional models to roughly align the two models.

[0107] The model-commonly identifiable structure detection unit 124 inputs the processing results of the structure type identification processing execution unit 122, i.e., the structure type identification processing results of the two 3D models to be integrated, and further inputs the rough joint positions of the two 3D models from the 3D model junction area analysis unit 123, and detects model-commonly identifiable structures in the vicinity of these joint positions.

[0108] For example, the windows described above with reference to FIGS. 3 to 6 and the doors described above with reference to FIG. 7 are detected as commonly identifiable structures.

[0109] The three-dimensional model alignment execution unit 125 executes high-precision alignment of the two three-dimensional models to be integrated. That is, the unit 125 executes high-precision alignment processing of the two three-dimensional models on different coordinate systems by using the model-commonly identifiable structures detected from each of the two three-dimensional models to be integrated by the model-commonly identifiable structure detection unit 124.

[0110] The processing executed by the three-dimensional model alignment execution unit 125 corresponds to, for example, the processing described above with reference to Fig. 6. That is, in the example shown in Fig. 6, by utilizing the fact that the real-space positions of the window 52, ​​which is a model-commonly identifiable structure extracted from the three-dimensional model A and the three-dimensional model B, match, the two three-dimensional models are arranged so that the positions of the windows of the three-dimensional model A and the three-dimensional model B match, and the alignment processing of the three-dimensional models A and B is executed.

[0111] The three-dimensional model alignment execution unit 125 uses the model commonly identifiable structures detected by the model commonly identifiable structure detection unit 124 in this manner to align two three-dimensional models on different coordinate systems.

[0112] The 3D model integration processing execution unit (coordinate transformation execution unit) 126 uses the alignment results of the two 3D models by the 3D model alignment execution unit 125 to generate a single 3D model by integrating the two 3D models.

[0113] In this 3D model integration process, two 3D models in different coordinate systems must be combined into one 3D model in a single coordinate system, and the coordinate system of one 3D model must be converted into the coordinate system of the other 3D model. This coordinate conversion process generates a single 3D model A+B, for example, by incorporating the 3D configuration of one 3D model A into the 3D configuration of the other 3D model B. Specific examples of the coordinate conversion process will be described in detail later.

[0114] The structure type identification processing learning model update unit (learning processing execution unit) 127 executes an update process for the structure type identification processing learning model stored in the structure type identification processing learning model storage unit 132 .

[0115] The structure type identification processing learning model update unit (learning processing execution unit) 127 inputs the processing results of the semantic segmentation processing performed by the structure type identification processing execution unit 122 by applying the structure type identification processing learning model stored in the structure type identification processing learning model storage unit 132, and performs re-learning processing using error information, etc. contained in the input processing results.

[0116] This re-learning process updates the learning model for structure type identification processing stored in the learning model storage unit 132 for structure type identification processing, generating a more accurate learning model and storing it in the learning model storage unit 132 for structure type identification processing.

[0117] The processes executed by the components of the information processing apparatus 100 of the present disclosure shown in FIG. 9 have been described above.

[0118] 5. Details of the Process Executed by the Data Processing Unit of the Information Processing Device of the Present Disclosure Next, details of the process executed by the data processing unit of the information processing device of the present disclosure will be described.

[0119] As described with reference to Figure 9, the data processing unit 101 of the information processing device 100 of the present disclosure has a 3D model acquisition unit (generation unit) 121, a structure type identification processing execution unit 122, a 3D model joint area analysis unit 123, a model commonly identifiable structure detection unit 124, a 3D model alignment execution unit 125, a 3D model integration processing execution unit (coordinate transformation execution unit) 126, and a learning model update unit for structure type identification processing (learning processing execution unit) 127.

[0120] Below, a detailed sequence of the processing executed by the data processing unit 101 of the information processing apparatus 100 of the present disclosure and specific examples of the processing will be described in order.

[0121] A detailed sequence of processing executed by the data processing unit 101 of the information processing device 100 of the present disclosure will be described with reference to the flowcharts in Fig. 10 and subsequent figures. Note that the flowcharts in Fig. 10 and subsequent figures correspond to detailed flowcharts of the processing in the flowchart shown in Fig. 8 described above.

[0122] The processing of each step in the flowchart shown in Fig. 10 and subsequent figures will be described below. (Step S211) First, in step S211, the data processing unit 101 of the information processing device 100 shown in Fig. 9 inputs (or generates) a three-dimensional model A (3D model A) using coordinate system a.

[0123] In the following description, the two 3D models to be integrated are referred to as 3D model A and 3D model B. 3D model A is a 3D model on coordinate system a, and 3D model B is a 3D model on coordinate system b.

[0124] Step S211 is processing executed by the 3D model acquisition unit (generation unit) 121 of the data processing unit 101 of the information processing device 100 shown in Fig. 9. The 3D model acquisition unit (generation unit) 121 acquires a 3D model A, which is one of two 3D models to be integrated, from the 3D models stored in the 3D model (3D model) storage unit 131. The 3D model acquisition unit (generation unit) 121 may be configured to generate the 3D model A by itself using an image captured by a camera, etc.

[0125] (Step S212) Next, in step S212, the data processing unit 101 of the information processing device 100 performs a process of identifying the type of structure of the 3D model A acquired (or generated) in step S211. For example, a semantic segmentation process is performed.

[0126] This process is executed by the structure type identification process execution unit 122 of the data processing unit 101 of the information processing device 100 shown in FIG.

[0127] The structure type identification process execution unit 122 executes the semantic segmentation process, for example, by using the structure type identification process learning model stored in the structure type identification process learning model storage unit 132.

[0128] As mentioned above, semantic segmentation is a process that uses pre-generated learning data to determine the type of object (window, door, etc.) to which each pixel belongs on a pixel-by-pixel basis that makes up an image, and divides pixel regions according to the object type.

[0129] A specific example of semantic segmentation processing for a three-dimensional model will be described with reference to Fig. 11. The example shown in Fig. 11 is an example in which semantic segmentation is performed on three-dimensional model A (A) shown on the left to detect a window region.

[0130] The window region detection result by semantic segmentation shown on the right side of Figure 11 (B) shows an example in which only the window region is set as a gray region by the semantic segmentation process. In this way, the semantic segmentation process makes it possible to extract image regions of specific structure types.

[0131] (Steps S221 to S222) Steps S221 to S222 are processing steps in which the same processing as steps S211 to S212 is executed on the other three-dimensional model B to be integrated.

[0132] That is, in step S221, a three-dimensional model B (3D model B) using coordinate system b is input (or generated). This process is executed by the three-dimensional model acquisition unit (generation unit) 121 of the data processing unit 101 of the information processing device 100 shown in FIG.

[0133] The three-dimensional model acquisition unit (generation unit) 121 acquires a three-dimensional model B, which is one of two three-dimensional models to be integrated, from the three-dimensional models stored in the three-dimensional model (3D model) storage unit 131. Alternatively, the three-dimensional model acquisition unit (generation unit) 121 generates the three-dimensional model B by itself using an image captured by a camera or the like.

[0134] Next, in step S222, a process for identifying the type of structure of the 3D model B acquired (or generated) in step S221 is performed. For example, a semantic segmentation process is performed.

[0135] 9 executes, for example, the semantic segmentation process described above. The structure type identification process execution unit 122 executes the semantic segmentation process using the structure type identification process learning model stored in the structure type identification process learning model storage unit 132.

[0136] (Step S225) Next, in step S225, the data processing unit 101 of the information processing device 100 analyzes the joining region between the 3D model A to be integrated and the 3D model B. For example, a rough alignment process is performed using GPS position data corresponding to each 3D model.

[0137] This process is executed by the three-dimensional model bonded region analysis unit 123 of the data processing unit 101 of the information processing device 100 shown in FIG.

[0138] As described above, the three-dimensional model junction region analysis unit 123 roughly aligns the two three-dimensional models to be integrated. That is, it analyzes the rough junction position of the two three-dimensional models. In other words, the three-dimensional model junction region analysis unit 123 performs processing to identify the rough junction position of the two three-dimensional models in order to limit the region for detecting a "model common identification structure" from the two three-dimensional models to be integrated.

[0139] As described above, the analysis process for the rough joining position of two 3D models to be integrated uses position information such as GPS position information associated with the two 3D model images to be integrated in pixel units or pixel region units. Alternatively, the two 3D models may be displayed on the display unit of the information processing device 100, and the user may analyze the rough joining position of the two displayed 3D models to perform rough alignment.

[0140] After the process of step S225 in the flow shown in Fig. 10, the data processing unit 101 of the information processing device 100 executes the process of step S231 shown in Fig. 12. The process of each step in the flow shown in Fig. 12 will be described below in order.

[0141] (Step S231) In step S231, the data processing unit 101 of the information processing device 100 executes a process of detecting model-commonly identifiable structures (=model alignment structures) from the three-dimensional model A and the three-dimensional model B. For example, the data processing unit 101 detects the windows described above with reference to FIGS. 3 to 6 and the doors described above with reference to FIG. 7 as model-commonly identifiable structures.

[0142] This process is executed by the model commonly identifiable structure detection unit 124 of the data processing unit 101 of the information processing device 100 shown in FIG.

[0143] The model-commonly identifiable structure detection unit 124 inputs the processing results of steps S212 and S222, i.e., the structure type identification processing results of the three-dimensional model A and three-dimensional model B to be integrated, and detects model-commonly identifiable structures from the two three-dimensional models.

[0144] For example, the windows described above with reference to FIGS. 3 to 6 and the doors described above with reference to FIG. 7 are detected as commonly identifiable structures.

[0145] The model-commonly identifiable structure detection unit 124 inputs rough joint position information of the two three-dimensional models to be integrated from the three-dimensional model joint area analysis unit 123, and detects model-commonly identifiable structures from the area near the rough joint positions of these two three-dimensional models.

[0146] (Step S232) Next, in step S232, the data processing unit 101 of the information processing device 100 determines whether the process of detecting model-common identifiable structures (=model alignment structures) from the three-dimensional model A and the three-dimensional model B has been successful.

[0147] This process is also executed by the model commonly identifiable structure detection unit 124 of the data processing unit 101 of the information processing device 100 shown in Fig. 9. In step S232, the model commonly identifiable structure detection unit 124 determines whether the model commonly identifiable structure detection process executed in step S231 was successful, and if successful, proceeds to step S233. If it is determined to have failed, the process ends. In other words, the integration process of the two 3D models is stopped.

[0148] (Step S233) If it is determined in step S232 that the model-common identifiable structure detection process executed in step S231 has been successful, the process of step S233 is executed.

[0149] In step S233, the data processing unit 101 of the information processing device 100 performs alignment processing of the three-dimensional model A and the three-dimensional model B using the model-common identifiable structure (=model alignment structure) detected in step S231.

[0150] This process is executed by the three-dimensional model alignment execution unit 125 of the data processing unit 101 of the information processing device 100 shown in FIG.

[0151] The three-dimensional model alignment execution unit 125 performs high-precision alignment processing of the two three-dimensional models A and B on different coordinate systems, using the model-commonly identifiable structures detected in step S231 from each of the two three-dimensional models A and B to be integrated.

[0152] This process corresponds to the process described above with reference to Fig. 6. That is, by utilizing the fact that the real-space positions of the window 52, ​​which is a model-commonly identifiable structure extracted from the three-dimensional model A and the three-dimensional model B, substantially coincide as shown in Fig. 6, the two three-dimensional models are arranged so that the positions of the windows of the three-dimensional model A and the three-dimensional model B coincide, and the process of aligning the three-dimensional models A and B is performed.

[0153] If there is an overlapping area between 3D models A and B to be integrated, an existing registration method such as ICP may be applied. The ICP algorithm is an algorithm that calculates the optimal position, i.e., the correct relative position of the two 3D models, by combining two pieces of point cloud data that make up the 3D model shape with the nearest points of the other 3D model as corresponding points for each point of one 3D model and moving the relative positions of the two models so as to minimize the squared error based on the correspondence relationship.

[0154] (Step S234) Next, in step S234, the data processing unit 101 of the information processing device 100 determines whether the process of step S233, that is, the process of aligning the three-dimensional model A and the three-dimensional model B, has been successful.

[0155] This process is also executed by the three-dimensional model alignment execution unit 125 of the data processing unit 101 of the information processing device 100 shown in FIG.

[0156] The three-dimensional model alignment execution unit 125 performs a determination process to determine whether the error between the position of a model-commonly identifiable structure extracted from, for example, three-dimensional model A and the position of a model-commonly identifiable structure extracted from three-dimensional model A is less than a predetermined threshold value.

[0157] For example, if the position error is less than a specified threshold value, it is determined that the alignment process has been successful, whereas if the position error is equal to or greater than the specified threshold value, it is determined that the alignment process has failed.

[0158] In step S234, the three-dimensional model alignment execution unit 125 determines whether the alignment process between three-dimensional model A and three-dimensional model B executed in step S233 was successful, and if it is determined that it was successful, it executes the processes of step S241 (FIG. 13) and steps S251 and S261 (FIG. 19). On the other hand, if it is determined that it was unsuccessful, it ends the process. In other words, it stops the integration process of the two three-dimensional models.

[0159] If it is determined in step S234 that the alignment process between 3D model A and 3D model B executed in step S233 was successful, the following two types of processes are executed: (Process 1) Execute the processes from step S241 onwards in the flow shown in Fig. 13 to execute integration process of 3D model A and 3D model B. (Process 2) Execute the processes from steps S251 and S261 onwards in the flow shown in Fig. 19 to execute update process (learning process) of the learning model for structure type identification process stored in the learning model storage unit 132 for structure type identification process.

[0160] The above (Process 1) will be described below with reference to Figures 13 to 18. The above (Process 2) will be described with reference to Figures 19 and 20.

[0161] First, the above (Process 1), that is, the integration process of the three-dimensional model A and the three-dimensional model B, will be described in detail with reference to FIGS.

[0162] The flowchart shown in Fig. 13 is a flowchart illustrating a detailed sequence of the integration process of 3D model A and 3D model B. The process according to this flow is executed by the 3D model integration process execution unit (coordinate conversion process execution unit) 126 of the data processing unit 101 of the information processing device 100 shown in Fig. 9. The process of each step of the flow shown in Fig. 13 will be described below.

[0163] (Step S241) In step S241, the 3D model integration processing execution unit (coordinate transformation processing execution unit) 126 of the data processing unit 101 of the information processing device 100 first acquires the position coordinates (position coordinates (Xa, Ya, Za) in the coordinate system a of the 3D model A) of the model commonly identifiable structure (= structure for model alignment) of the 3D model A.

[0164] (Step S242) In step S242, the three-dimensional model integration processing execution unit (coordinate transformation processing execution unit) 126 further acquires the position coordinates (position coordinates (Xb, Yb, Zb) in the coordinate system b of the three-dimensional model B) of the model-commonly identifiable structure (= structure for model alignment) of the three-dimensional model B.

[0165] A specific example of these processing steps, that is, steps S241 to S242, will be described with reference to FIGS.

[0166] Figure 14 shows two 3D models to be integrated: (A) 3D model A of an outdoor space such as an urban area, and (B) 3D model B of an indoor space such as an apartment building or office.

[0167] These two three-dimensional models include a window 52, ​​which is a "model common identifiable structure 50" used in the alignment in step S233, i.e., for high-precision alignment of the two three-dimensional models to be integrated.

[0168] In step S241, first, the position coordinates of the window 52, ​​which is the model commonly identifiable structure 50, are obtained from the three-dimensional model A (position coordinates (Xa, Ya, Za) in the coordinate system a of the three-dimensional model A).

[0169] Furthermore, in step S242, the position coordinates of the window 52, ​​which is the model commonly identifiable structure 50, are obtained from the three-dimensional model B (position coordinates (Xb, Yb, Zb) in the coordinate system b of the three-dimensional model B).

[0170] In steps S241 and S242, a plurality of position coordinates of positions that are approximately the same in real space are acquired from each of the two three-dimensional models to be integrated. A specific example will be described with reference to FIG.

[0171] Fig. 15 shows an example in which two position coordinates are acquired from each of two 3D models to be integrated. As shown in Fig. 15, the position coordinates of two different locations of a window 52, ​​which is a "model-common identifiable structure 50" used for highly accurate alignment of the two 3D models, are acquired.

[0172] 15, the position coordinates (Xa1, Ya1, Za1) of the upper left corner of the window 52 and the position coordinates (Xa2, Ya2, Za2) of the lower right corner of the window 52 are obtained. These are position coordinates according to the coordinate system a (e.g., global coordinates) of the three-dimensional model A.

[0173] The same positions are also obtained from the three-dimensional model B on the right side of Fig. 15, that is, the position coordinates (Xb1, Yb1, Zb1) of the upper left corner of the window 52 and the position coordinates (Xb2, Yb2, Zb2) of the lower right corner of the window 52. These are position coordinates according to the coordinate system b (for example, local coordinates) of the three-dimensional model B.

[0174] In this way, in steps S241 and S242, a plurality of position coordinates of positions that are approximately the same in real space are obtained from each of the two three-dimensional models to be integrated.

[0175] 15 is an example, and the position coordinates of other positions may be acquired. However, the acquired position coordinates are those of the points constituting the "model commonly identifiable structure 50" used for the high-precision alignment of the two 3D models. In other words, the acquired position coordinates are those of the points constituting the "model commonly identifiable structure 50" applied to the alignment of the two 3D models to be integrated.

[0176] (Step S243) Next, in step S243, the 3D model integration processing execution unit (coordinate transformation processing execution unit) 126 of the data processing unit 101 of the information processing device 100 executes the following processing.

[0177] That is, by taking advantage of the fact that the real-space positions of the model-commonly identifiable structures of the three-dimensional model A and three-dimensional model B to be integrated coincide, a coordinate transformation matrix (aTb) is calculated to convert the coordinate system b of three-dimensional model B into the coordinate system a of three-dimensional model A.

[0178] A specific example of the process in step S243 will be described with reference to Figures 16 and 17. Figure 16 is a diagram for explaining the relationship between the position coordinates acquired from each three-dimensional model in steps S241 and S242.

[0179] 16 shows two position coordinates obtained from the three-dimensional model A, namely, (Xa1, Ya1, Za1) and (Xa2, Ya2, Za2), which are the position coordinates of the upper left corner and the lower right corner of the window 52, ​​which is the "model-common identifiable structure 50." These are position coordinates according to the coordinate system a (e.g., global coordinates) of the three-dimensional model A.

[0180] Furthermore, two position coordinates obtained from the three-dimensional model B, namely (Xb1, Yb1, Zb1) and (Xb2, Yb2, Zb2), which are the position coordinates of the upper left and lower right ends of the window 52, ​​which is the "model commonly identifiable structure 50," are shown. These are position coordinates according to the coordinate system b (e.g., local coordinates) of the three-dimensional model B.

[0181] Here, the window 52 is a "model commonly identifiable structure 50" used for highly accurate alignment of the two three-dimensional models A and B, and since the window 52 has only one real-space position, the following holds: The real-space positions of (Xa1, Ya1, Za1) and (Xb1, Yb1, Zb1) are the same. Similarly, the real-space positions of (Xa2, Ya2, Za2) and (Xb2, Yb2, Zb2) are the same.

[0182] That is, the position coordinates (Xa1, Ya1, Za1) using the coordinate system a of three-dimensional model A and the position coordinates (Xb1, Yb1, Zb1) using the coordinate system b of three-dimensional model B are representations of the same point in real space using different coordinate systems. Similarly, the position coordinates (Xa2, Ya2, Za2) using the coordinate system a of three-dimensional model A and the position coordinates (Xb2, Yb2, Zb2) using the coordinate system b of three-dimensional model B are representations of the same point in real space using different coordinate systems.

[0183] In step S243, the three-dimensional model integration processing execution unit (coordinate transformation processing execution unit) 126 of the data processing unit 101 of the information processing device 100 utilizes the above-mentioned relationship, i.e., the fact that the real-space positions of the model-commonly identifiable structures of the three-dimensional models A and B to be integrated coincide with each other, to calculate a coordinate transformation matrix (aTb) that transforms the coordinate system b of the three-dimensional model B into the coordinate system a of the three-dimensional model A.

[0184] A specific example of the calculation process of the coordinate transformation matrix (aTb) will be described with reference to Fig. 17. Note that a matrix used to transform one coordinate system into another coordinate system is called a coordinate transformation matrix (T).

[0185] The coordinate transformation matrix (aTb) transforms a position (xb, yb, zb) defined in one coordinate system b into a position (xa, ya, za) in a different coordinate system a. The coordinate transformation matrix (aTb) not only transforms the position coordinates in coordinate system b into the position coordinates in coordinate system a, but also transforms the attitude of coordinate system b and the attitude of coordinate system a. The attitude of a coordinate system refers to the direction and inclination of the x, y, and z axes that define the coordinate system.

[0186] 17 is a diagram illustrating a coordinate transformation matrix (aTb) that transforms the position and orientation of coordinate system b into the position and orientation of coordinate system a. Specifically, for example, by applying this coordinate transformation matrix (aTb), it becomes possible to transform the position (bP) in coordinate system b to the position (aP) in coordinate system a for one point (P) in the three-dimensional space shown in the upper center of FIG.

[0187] Note that a point (P) (= (xa, ya, za)) in the three-dimensional space represented by coordinate system a is denoted as (aP), and a point (P) (= (xb, yb, zb)) in the three-dimensional space represented by coordinate system b is denoted as (bP).

[0188] The position of coordinate system a and the position of coordinate system b for one point (P) in the three-dimensional space shown in the upper center of Figure 17 are respectively shown as follows: Position of coordinate system a: aP Position of coordinate system b: bP

[0189] The coordinate transformation matrix that transforms the position (bP) in the coordinate system b of one point (P) in the above three-dimensional space into the position (aP) in the coordinate system a is defined as aTb.

[0190] As shown in the lower part of Fig. 17, the equation for converting the position (bP) of point (P) in the coordinate system b to the position (aP) in the coordinate system a can be expressed by the following (Equation 1): aP = aTb x bP (Equation 1)

[0191] In the above (Equation 1), aTb is a coordinate transformation matrix that transforms the position and orientation of the coordinate system b into the position and orientation of the coordinate system a.

[0192] This coordinate transformation matrix (aTb) can be expressed as the following matrix (Equation 2).

[0193]

[0194] In the above (Equation 2), aRb is a rotation matrix that converts the attitude defined in coordinate system b into the attitude defined in coordinate system a, and bPa is the origin position of coordinate system a in coordinate system b.

[0195] The coordinate transformation matrix (aTb) shown in the above (Equation 2) is a coordinate transformation matrix that transforms position coordinates (bP) in coordinate system b corresponding to one point (P) in three-dimensional space into position coordinates (aP) in coordinate system a.

[0196] By using this coordinate transformation matrix (aTb), it is possible to transform all position coordinates (bP) on coordinate system b into position coordinates (aP) on coordinate system a. That is, the equation for transforming the position (bP) on coordinate system b into the position (aP) on coordinate system a is the following (Equation 1) explained above: aP = aTb × bP (Equation 1)

[0197] By using the above formula (Formula 1), it is possible to convert all position coordinates (bP) on coordinate system b into position coordinates (aP) on coordinate system a, and using this result, it is possible to integrate the three-dimensional model b on coordinate system b with the three-dimensional model a on coordinate system a.

[0198] Here, as described above, the coordinate transformation matrix (aTb) can be expressed as the following matrix (Equation 2).

[0199]

[0200] The unknown parameters in the above (Equation 2) are the following two parameters: aRb: a rotation matrix that transforms the attitude defined in coordinate system b into the attitude defined in coordinate system a; and bPa: the position of the origin of coordinate system a in coordinate system b.

[0201] These parameters can be calculated by applying the following position coordinates acquired in steps S241 and S242: position coordinates (Xa1, Ya1, Za1), (Xa2, Ya2, Za2) using coordinate system a of three-dimensional model A, and position coordinates (Xb1, Yb1, Zb1), (Xb2, Yb2, Zb2) using coordinate system b of three-dimensional model B.

[0202] 18 , the position coordinates (Xa1, Ya1, Za1) using the coordinate system a of the three-dimensional model A and the position coordinates (Xb1, Yb1, Zb1) using the coordinate system b of the three-dimensional model B are expressed in different coordinate systems for the same point (P1) in real space. For the point (P1), the defined position of the position coordinates (Xa1, Ya1, Za1) using the coordinate system a of the three-dimensional model A is set to (aP1), and the defined position of the position coordinates (Xb1, Yb1, Zb1) using the coordinate system b of the three-dimensional model B is set to (bP1).

[0203] Similarly, position coordinates (Xa2, Ya2, Za2) using coordinate system a of three-dimensional model A and position coordinates (Xb2, Yb2, Zb2) using coordinate system b of three-dimensional model B are representations of the same point (P2) in real space using different coordinate systems. For point (P2), the defined position of position coordinates (Xa2, Ya2, Za2) using coordinate system a of three-dimensional model A is set to (aP2), and the defined position of position coordinates (Xb2, Yb2, Zb2) using coordinate system b of three-dimensional model B is set to (bP2).

[0204] According to the above (Equation 1), that is, aP=aTb×bP (Equation 1), the following two relational expressions are established: aP1=aTb×bP1 aP2=aTb×bP2.

[0205] By solving these two relational expressions as simultaneous equations, the unknown parameters aRb and bPa of the coordinate transformation matrix (aTb) can be calculated, and the coordinate transformation matrix (aTb) can be obtained.

[0206] In this way, the 3D model integration processing execution unit (coordinate transformation processing execution unit) 126 of the data processing unit 101 of the information processing device 100 calculates a coordinate transformation matrix (aTb) for transformation into the coordinate system a of the 3D model A in step S243.

[0207] That is, by taking advantage of the fact that the real-space positions of the model-commonly identifiable structures of the three-dimensional model A and three-dimensional model B to be integrated coincide, a coordinate transformation matrix (aTb) is calculated to convert the coordinate system b of three-dimensional model B into the coordinate system a of three-dimensional model A.

[0208] (Step S244) Next, in step S244, the 3D model integration processing execution unit (coordinate transformation processing execution unit) 126 of the data processing unit 101 of the information processing device 100 transforms the coordinate system b of the 3D model B into the coordinate system a of the 3D model A using the coordinate transformation matrix (aTb) calculated in step S243.

[0209] This is performed by applying the following (Equation 1) to convert the position (bP) in the coordinate system b described above into the position (aP) in the coordinate system a: aP = aTb × bP (Equation 1)

[0210] (Step S245) Next, in step S245, the three-dimensional model integration processing execution unit (coordinate conversion processing execution unit) 126 of the data processing unit 101 of the information processing device 100 executes processing to combine three-dimensional model B with three-dimensional model A using the processing result of step S244, i.e., the result of converting the coordinate system b of three-dimensional model B into the coordinate system a of three-dimensional model A. In other words, one three-dimensional model A+B is generated by including the three-dimensional configuration of three-dimensional model B in three-dimensional model A.

[0211] As described above, the information processing apparatus 100 of the present disclosure sequentially executes the following processes to generate one three-dimensional model A+B in which the three-dimensional configuration of the three-dimensional model B is included in the three-dimensional model A.

[0212] First, model-commonly identifiable structures are detected from each of the two three-dimensional models to be integrated, and the detected model-commonly identifiable structures are used to perform highly accurate alignment of the two three-dimensional models on different coordinate systems.

[0213] Furthermore, for multiple positions of the model-commonly identifiable structure that has been successfully aligned, position coordinates according to the coordinate systems of each of the two three-dimensional models are obtained, and the obtained position coordinates are used to calculate a coordinate transformation matrix (aTb), i.e., a coordinate transformation matrix (aTb) for transforming coordinate system b into coordinate system a.

[0214] Finally, the calculated coordinate transformation matrix (aTb) is applied to transform each position in the coordinate system b into a position in the coordinate system a, and the three-dimensional model B is integrated into the three-dimensional model A.

[0215] The information processing apparatus 100 of the present disclosure executes such processing to generate one three-dimensional model A+B in which the three-dimensional configuration of the three-dimensional model B is included in the three-dimensional model A.

[0216] The information processing device 100 of the present disclosure further executes processing according to the flowchart shown in FIG. 19 to perform processing to update the learning model stored in the structure type identification learning model storage unit 132.

[0217] As previously explained with reference to Figure 9, the structure type identification processing learning model storage unit 132 stores learning models for identifying the types of structures such as windows and doors from three-dimensional models.

[0218] This learning model is a learning model for performing the semantic segmentation process described above, and uses the learning model to analyze the features of the constituent images of the 3D model and identify the type of each structure in the 3D model, such as the type of structure, such as a window, door, etc.

[0219] However, if the accuracy of the learning model is low, an erroneous structure type identification process may be executed. An example of an error occurring when performing semantic segmentation processing using a learning model will be described with reference to Fig. 20 .

[0220] 20A shows the results of semantic segmentation processing performed on three-dimensional model A shown on the left, using a learning model stored in the structure type identification processing learning model storage unit 132. (B) on the right side of Fig. 20 shows the window region detected by semantic segmentation.

[0221] As shown in Figure 20, there are multiple image regions that are not identified as "windows" even though they are actually window regions. Such errors occur when the accuracy of the learning model applied to semantic segmentation is low.

[0222] In order to reduce such errors, the information processing device 100 of the present disclosure performs a re-learning process using the results of the semantic segmentation process that was actually performed, and performs an update process for the learning model stored in the structure type identification learning model storage unit 132.

[0223] This processing sequence will be described with reference to the flowchart shown in Fig. 19. The processing according to the flowchart shown in Fig. 19 is executed when a "Yes" determination is made in step S234 of the previously described flow in Fig. 12. In other words, this processing is executed when the 3D model alignment execution unit 125 determines that the alignment processing between 3D model A and 3D model B executed in step S233 has been successful.

[0224] The processing of each step in the flow shown in Fig. 19 will be described below in order. Note that the processing according to the flow shown in Fig. 19 is executed by the structure type identification processing learning model update unit (learning processing unit) 127 of the data processing unit 101 of the information processing device 100 shown in Fig. 9.

[0225] (Step S251) First, in step S251, the learning model update unit (learning processing unit) 127 for structure type identification processing of the data processing unit 101 of the information processing device 100 executes a process of detecting identification errors in the process of identifying the type of structure of the 3D model A (semantic segmentation, etc.).

[0226] For example, the occurrence of an error such as that described with reference to FIG. 20 is detected.

[0227] (Step S252) Next, in step S252, the learning model update unit (learning processing unit) 127 for structure type identification processing of the data processing unit 101 of the information processing device 100 uses the structure type identification error information analyzed in step S251 to perform an update process (relearning process) of the learning model for executing the structure type identification process of the 3D model A (such as a semantic segmentation execution learning model).

[0228] The updated learning model is stored in the structure type identification processing learning model storage unit 132.

[0229] (Steps S261 to S262) Steps S261 to S262 are steps for executing the same processes as steps S251 to S252 described above on three-dimensional model B, which is the other of the three-dimensional models to be integrated.

[0230] That is, in step S261, the learning model update unit (learning processing unit) 127 for structure type identification processing of the data processing unit 101 of the information processing device 100 performs a process of detecting identification errors in the process of identifying the type of structure of the three-dimensional model B (semantic segmentation, etc.).

[0231] Next, in step S262, the structure type identification error information analyzed in step S261 is used to perform an update process (relearning process) of the learning model (semantic segmentation execution learning model, etc.) for executing the structure type identification process of the 3D model B. The updated learning model is stored in the structure type identification process learning model storage unit 132.

[0232] By performing such a learning model update process, it is possible to improve the accuracy of the semantic segmentation process to which the learning model is applied.

[0233] 6. Hardware Configuration Example of Information Processing Device of the Present Disclosure Next, a hardware configuration example of the information processing device 100 of the present disclosure will be described with reference to FIG.

[0234] Each element of the hardware configuration shown in Fig. 21 will be described below. A CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to programs stored in a ROM (Read Only Memory) 302 or a storage unit 308. For example, it executes processes according to the sequences described in the above-mentioned embodiments. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0235] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various sensors, a camera, a switch, a keyboard, a mouse, a microphone, etc., and an output unit 307 consisting of a display, a speaker, etc.

[0236] The storage unit 308 connected to the input / output interface 305 is composed of, for example, a USB memory, an SD card, a hard disk, etc., and stores various data and programs executed by the CPU 301. The communication unit 309 functions as a transmission / reception unit for data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0237] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.

[0238] [7. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described above in detail with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0239] The technology disclosed in this specification can be configured as follows: (1) An information processing method in which a data processing unit of an information processing device executes: a model commonly identifiable structure detection step of detecting a model commonly identifiable structure from each of two three-dimensional models, a registration step of aligning the two three-dimensional models using the model commonly identifiable structures, and a three-dimensional model integration step of executing integration processing of the two three-dimensional models using the registration result of the two three-dimensional models.

[0240] (2) The information processing method according to (1), wherein the two three-dimensional models are three-dimensional models in different coordinate systems.

[0241] (3) The information processing method according to (1) or (2), wherein the model commonly identifiable structure detected in the model commonly identifiable structure detection step is a structure that can be identified from both sides of the joint surface of the two three-dimensional models.

[0242] (4) The data processing unit further executes a structure type identification processing step for identifying the type of structure in each of the two three-dimensional models, and the model-commonly identifiable structure detection step inputs a result of the structure type identification processing step and detects model-commonly identifiable structures from each of the two three-dimensional models. This is an information processing method described in any one of (1) to (3).

[0243] (5) The information processing method according to (4), wherein the structure type identification processing step performs semantic segmentation processing using a learning model to identify the type of structure in the three-dimensional model.

[0244] (6) The information processing method described in (5), wherein the data processing unit further has a learning model update processing execution step for updating a learning model used in the semantic segmentation processing in the structure type identification processing step, and the learning model update processing execution step includes a learning processing execution step using the processing results of the semantic segmentation processing in the structure type identification processing step.

[0245] (7) The data processing unit further executes a three-dimensional model joint area analysis step for analyzing the joint position of the two three-dimensional models, and the model commonly identifiable structure detection step inputs a result of the three-dimensional model joint area analysis step and detects a model commonly identifiable structure near the joint position of the two three-dimensional models. This is an information processing method described in any one of (1) to (6).

[0246] (8) The information processing method according to any one of (1) to (7), wherein the model commonly identifiable structure detection step detects a window or a door that is commonly identifiable from each of the two three-dimensional models as the model commonly identifiable structure.

[0247] (9) An information processing method according to any one of (1) to (8), wherein one of the two three-dimensional models is an outdoor three-dimensional model and the other is an indoor three-dimensional model, and the model commonly identifiable structure detection step detects, as the model commonly identifiable structure, a window that is commonly identifiable in each of the outdoor three-dimensional model and the indoor three-dimensional model.

[0248] (10) The information processing method according to any one of (1) to (9), wherein the alignment step aligns the two three-dimensional models by setting the model-commonly identifiable structures detected from each of the two three-dimensional models at the same position.

[0249] (11) An information processing method according to any one of (1) to (10), wherein the model-common identifiable structure is a structure that can be identified from both sides of the joint surface of the two three-dimensional models, and the alignment step aligns the two three-dimensional models by setting the model-common identifiable structure detected from each of the two three-dimensional models at the same position on the joint surface of the two three-dimensional models.

[0250] (12) The information processing method according to any one of (1) to (11), wherein the three-dimensional model integration step calculates a coordinate transformation matrix that can be used for transformation processing of the coordinate systems of the two three-dimensional models, and executes integration processing of the two three-dimensional models using the calculated coordinate transformation matrix.

[0251] (13) The information processing method according to (12), wherein the three-dimensional model integration step acquires position coordinates of the model commonly identifiable structure in the coordinate systems of the two three-dimensional models, and calculates the coordinate transformation matrix using the acquired position coordinates.

[0252] (14) An information processing device having: a model commonly identifiable structure detection unit that detects a model commonly identifiable structure from each of two three-dimensional models; a three-dimensional model alignment execution unit that aligns the two three-dimensional models using the model commonly identifiable structure; and a three-dimensional model integration processing execution unit that executes integration processing of the two three-dimensional models using the alignment result of the two three-dimensional models.

[0253] (15) A program for executing information processing in an information processing device, the program causing a data processing unit of the information processing device to execute: a model commonly identifiable structure detection step for detecting a model commonly identifiable structure from each of two three-dimensional models; an alignment step for aligning the two three-dimensional models using the model commonly identifiable structures; and a three-dimensional model integration step for performing integration processing of the two three-dimensional models using the alignment results of the two three-dimensional models.

[0254] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program from the recording medium to the computer, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0255] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are all located in the same housing.

[0256] As described above, an embodiment of the present disclosure provides an apparatus and method for performing highly accurate registration and integration of two 3D models in different coordinate systems. Specifically, for example, a commonly identifiable structure is detected from each of two 3D models in different coordinate systems, the two 3D models are aligned using the detected structure, and the alignment results are used to integrate the two 3D models. The commonly identifiable structure is a structure, such as a window, that can be identified from both sides of the interface between the two 3D models. The two 3D models are integrated using a coordinate transformation matrix calculated based on the alignment of the coordinate systems of the structure in real space. This configuration provides an apparatus and method for performing highly accurate registration and integration of two 3D models in different coordinate systems.

[0257] 10 Building 20 Indoors 50 Model commonly identifiable structure 52 Window 54 Door 100 Information processing device 101 Data processing unit 121 3D model acquisition unit (generation unit) 122 Structure type identification processing execution unit 123 3D model joint area analysis unit 124 Model commonly identifiable structure detection unit 125 3D model alignment execution unit 126 3D model integration processing execution unit (coordinate transformation execution unit) 127 Structure type identification processing learning model update unit (learning processing execution unit) 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / output interface 306 Input unit 307 Output unit 308 Storage unit 309 Communication unit 310 Drive 311 Removable media

Claims

1. An information processing method in which a data processing unit of an information processing device executes a model-commonly identifiable structure detection step of detecting model-commonly identifiable structures from each of two three-dimensional models, an alignment step of aligning the two three-dimensional models using the model-commonly identifiable structures, and a three-dimensional model integration step of performing integration processing of the two three-dimensional models using the alignment results of the two three-dimensional models.

2. The information processing method according to claim 1, wherein the two three-dimensional models are three-dimensional models on different coordinate systems.

3. The information processing method according to claim 1, wherein the model commonly identifiable structure detected in the model commonly identifiable structure detection step is a structure that can be identified from both sides of the joint surface of the two three-dimensional models.

4. The information processing method described in claim 1, wherein the data processing unit further executes a structure type identification processing step for identifying the type of structure within each of the two three-dimensional models, and the model-commonly identifiable structure detection step inputs the execution results of the structure type identification processing step and detects model-commonly identifiable structures from each of the two three-dimensional models.

5. An information processing method according to claim 4, wherein the structure type identification processing step performs semantic segmentation processing using a learning model to identify the type of structure within the three-dimensional model.

6. The information processing method described in claim 5, wherein the data processing unit further has a learning model update processing execution step for updating the learning model used for the semantic segmentation processing in the structure type identification processing step, and the learning model update processing execution step includes a learning processing execution step using the processing results of the semantic segmentation processing in the structure type identification processing step.

7. The information processing method according to claim 1, wherein the data processing unit further executes a three-dimensional model joint area analysis step for analyzing the joint position of the two three-dimensional models, and the model commonly identifiable structure detection step inputs the results of the three-dimensional model joint area analysis step and detects model commonly identifiable structures in the vicinity of the joint position of the two three-dimensional models.

8. An information processing method according to claim 1, wherein the model commonly identifiable structure detection step detects a window or door that is commonly identifiable from each of the two three-dimensional models as the model commonly identifiable structure.

9. An information processing method as described in claim 1, wherein one of the two three-dimensional models is an outdoor three-dimensional model and the other is an indoor three-dimensional model, and the model commonly identifiable structure detection step detects a window that is commonly identifiable in each of the outdoor three-dimensional model and the indoor three-dimensional model as the model commonly identifiable structure.

10. An information processing method according to claim 1, wherein the alignment step aligns the two three-dimensional models by setting the model-commonly identifiable structures detected from each of the two three-dimensional models at the same position.

11. The information processing method of claim 1, wherein the model-common identifiable structure is a structure that can be identified from both sides of the joint surface of the two three-dimensional models, and the alignment step aligns the two three-dimensional models by setting the model-common identifiable structure detected from each of the two three-dimensional models at the same position on the joint surface of the two three-dimensional models.

12. An information processing method according to claim 1, wherein the three-dimensional model integration step calculates a coordinate transformation matrix that can be used for transformation processing of the coordinate systems of each of the two three-dimensional models, and performs integration processing of the two three-dimensional models using the calculated coordinate transformation matrix.

13. The information processing method according to claim 12, wherein the three-dimensional model integration step acquires position coordinates of the model-commonly identifiable structure in the coordinate system of each of the two three-dimensional models, and calculates the coordinate transformation matrix using the acquired position coordinates.

14. An information processing device having: a model-commonly identifiable structure detection unit that detects model-commonly identifiable structures from each of two three-dimensional models; a three-dimensional model alignment execution unit that aligns the two three-dimensional models using the model-commonly identifiable structures; and a three-dimensional model integration processing execution unit that executes integration processing of the two three-dimensional models using the alignment results of the two three-dimensional models.

15. A program for executing information processing in an information processing device, which causes a data processing unit of the information processing device to execute a model-commonly identifiable structure detection step for detecting model-commonly identifiable structures from each of two three-dimensional models, an alignment step for aligning the two three-dimensional models using the model-commonly identifiable structures, and a three-dimensional model integration step for performing integration processing of the two three-dimensional models using the alignment results of the two three-dimensional models.

Images