3D model generation device and 3D model generation program

Abstract

To provide a 3D model generation device that can suppress the processing load. [Solution] The 3D model generation device 10 includes at least one CPU 10a and generates a 3D model from image data captured by a camera. The CPU 10a is configured to prepare multiple datasets DS1, DS2, and DS3, each containing multiple image data of regions A1, A2, and A3 that are at least partially different from each other, and to generate a wide-area 3D model in order to integrate the multiple 3D models that can be generated separately for each dataset DS1, DS2, and DS3. Generating a wide-area 3D model includes restricting the use of data from the inner circumference IS of each region A1, A2, and A3, and matching the feature points of the outer circumference OS to determine the relative positions of each region A1, A2, and A3.

Description

Technical Field

[0001] The disclosure according to this specification relates to a technique for generating a wide - area 3D model.

Background Art

[0002] Patent Document 1 discloses a technique for generating a 3D model of a forest. Specifically, trees are measured at a plurality of measurement positions, and matching is performed based on the positions of the trees (distance and angle between the trees).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Now, in the generation of such a 3D model, there is a need to generate a wide - area model. When generating a wide - area model, it is necessary to integrate data obtained at a large number of positions. However, if the technique of Patent Document 1 is directly applied to the generation of a wide - area model, data at a large number of measurement positions must be matched, resulting in an extremely high processing load.

[0005] One of the objects according to the disclosure of this specification is to provide a 3D model generation device and a 3D model generation program capable of suppressing the processing load.

Means for Solving the Problems

[0006] One of the aspects disclosed herein is a 3D model generation device including at least one processing unit (10a) that generates a 3D model from image data captured by a camera, The at least one processing unit, This involves preparing multiple datasets (DS1, DS2, DS3, DSa, DSb) containing multiple image data of regions (A1, A2, A3, Aa, Ab) that are at least partially different from each other, It is configured to generate a wide-area 3D model in order to integrate multiple 3D models that may be generated for each dataset, Generating a wide-area 3D model involves restricting the use of data from the inner region (IS) of each area, matching feature points from the outer region (OS), and determining the relative positions of each area.

[0007] Another aspect of the disclosed embodiment is a 3D model generation program that generates a 3D model from image data captured by a camera, At least one processing unit (10a) This involves preparing multiple datasets (DS1, DS2, DS3, DSa, DSb) containing multiple image data of regions (A1, A2, A3, Aa, Ab) that are at least partially different from each other, It is configured to generate a wide-area 3D model and to perform the following actions: integrating multiple 3D models that may be generated for each dataset. Generating a wide-area 3D model involves restricting the use of data from the inner region (IS) of each area, matching feature points from the outer region (OS), and determining the relative positions of each area.

[0008] According to these embodiments, a wide-area 3D model is generated by integrating multiple 3D models that can be generated for each dataset. Therefore, the amount of data handled and the processing load during generation are reduced compared to generating a wide-area 3D model all at once. In addition, the relative positions of each region are determined by matching feature points on the outer edges of the regions captured by the multiple image data included in the dataset. When regions that are at least partially different from each other are captured in each dataset, the outer edges have the highest probability of being matchable with the outer edges of other datasets. Furthermore, since the use of data from the inner edges of the regions captured by the multiple image data included in the dataset is restricted, the amount of data handled and the processing load during generation are also reduced in this respect. As a result, the processing load can be reduced in the generation of wide-area 3D models.

[0009] The symbols in parentheses included in the claims, etc., are illustrative examples illustrating the correspondence with the embodiments described later, and are not intended to limit the technical scope. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing the schematic configuration of a 3D model generation device. [Figure 2] A diagram illustrating the divided regions. [Figure 3] A diagram showing the functional configuration of a 3D model generation device. [Figure 4] A diagram to explain the matching process. [Figure 5] A diagram to explain the matching process. [Figure 6] A flowchart illustrating an example of processing using a 3D model generation device. [Figure 7] A diagram to explain the matching process. [Figure 8] A diagram illustrating a model using Gaussian splatting. [Figure 9] A flowchart illustrating an example of processing using a 3D model generation device. [Figure 10] A diagram for explaining an example of the positional relationship of regions corresponding to a dataset. [Figure 11] A diagram for explaining an example of the positional relationship of regions corresponding to a dataset. [Figure 12] A diagram for explaining an example of the positional relationship of regions corresponding to a dataset. [Figure 13] A diagram for explaining an example of the positional relationship of regions corresponding to a dataset. [Figure 14] A diagram for explaining an example of the positional relationship of regions corresponding to a dataset.

Mode for Carrying Out the Invention

[0011] In the present disclosure or the claims, the term "processor" is a processor as single or multiple hardware, which is configured to execute the processing defined by the computer program code (i.e., one or more instructions of the computer program) included in the computer program by reading the code each time. In other words, the "processor" is a hardware device that executes one or more programmed processes. Therefore, the computer program code can also be said to be software that can define the processing of the processor according to its content. For example, the "processor" can be a general-purpose or special-purpose processor, such as a CPU, a microprocessor, a GPU, and a DFP (Data Flow Processor), etc., but is not limited thereto.

[0012] In the present disclosure or the claims, the term "memory" refers to a non-transitory physical recording medium, which indicates one or more hardware memories configured to record computer program code and / or data in an accessible manner from the processor. The "memory" can be realized by memory technologies such as SRAM, SDRAM, non-volatile flash type memory, or other types of memory. The computer program code constituting the program is recorded on the memory and executed by the processor, thereby enabling the processor to realize various functions described above.

[0013] In the present disclosure or claims, the term "circuit" refers to one or more hardware logic circuits, which are configured to perform specific processing defined based on a pre-designed circuit configuration. In other words (and in contrast to a "processor"), the "circuit" in the present disclosure or claims does not refer to something whose processing is defined by software such as the above computer program code, but rather refers to a hardware device that executes specific processing based on a circuit configuration. For example, "circuit" may include custom ICs such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays) designed by a hardware description language (HDL: Hardware Description Language). That is, the "circuit" in the present disclosure or claims includes all hardware circuits except the above processor that executes processing by loading computer program code.

[0014] In the present disclosure or claims, the expression "at least one of a circuit and a processor" should be interpreted as disjunctive (logical OR), and should not be interpreted as at least one circuit and at least one processor.

[0015] In the present disclosure or claims, the term "processing unit" means a hardware device that executes processing by a "processor", a "circuit", or a combination thereof. If the function of the "processing unit" is interpreted as being impossible to be realized by a "circuit" and possible to be realized by a "processor", it may mean the "processor" itself.

[0016] Several embodiments will be described below with reference to the drawings. In each embodiment, the same reference numerals are used for corresponding components, and redundant explanations may be omitted. If only a part of the configuration is described in each embodiment, the configuration of other embodiments described earlier can be applied to the other parts of that configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations from multiple embodiments can be partially combined even if not explicitly stated, as long as there are no particular problems with the combination.

[0017] (First Embodiment) The 3D model generation device 10 of the first embodiment shown in Figure 1 is used, for example, in forest management services. Forest management services are services that visualize forest information to support the efficiency of forestry and the proper management of old-growth forests, etc. By properly managing forests, forests can demonstrate a high absorption capacity for greenhouse gases such as carbon dioxide. In addition, forest management services guarantee the quality of carbon credits through regular forest measurements.

[0018] Specifically, the forest management service captures image data of forests and provides a 3D model of a wide-area forest generated from the image data as visualized forest information. The 3D model generation device 10 performs the process of generating a 3D model of the forest.

[0019] Image data is captured using a camera. Alternatively, the image data may be captured by a drone or other aircraft equipped with a camera exploring the forest and capturing images at multiple locations within the forest. Instead of a drone, the image data may be captured by a robot capable of moving through the forest, or by a human photographer walking around the forest. The shooting route within the forest can be set based on a route plan.

[0020] The 3D model is a three-dimensional model of the forest's shape, and can be represented, for example, by a point cloud. Here, the forest's shape may include the shape of the ground. The forest's shape may include the position and shape of trees growing from the ground. In this case, the tree's shape only needs to include the shape of the trunk, which corresponds to the main body of the tree, and does not need to include the shapes of the branches and leaves. The trunk's shape includes its thickness. The trunk's shape may or may not include its height. The 3D model does not need to be constructed to be visualized in 3D space or through stereoscopic viewing; it is sufficient that it can be visualized on a 2D screen from the user's desired viewpoint.

[0021] The 3D model generation device 10 is configured primarily around, for example, at least one computer. The computer comprising the 3D model generation device 10 has at least one CPU (Central Processing Unit) 10a and at least one memory 10b. The memory 10b may be a non-transitory tangible storage medium that non-temporarily stores computer programs, data, artificial intelligence models, etc., that can be read by the CPU 10a. Furthermore, the computer may have a rewritable volatile storage medium such as RAM (Random Access Memory) 10c. The CPU 10a can execute various processes according to the computer programs stored in the memory 10b.

[0022] Furthermore, the computer may have an interface 10d for exchanging data with the outside world. Interface 10d may include a communication interface for connecting the computer to the internet or external devices. Interface 10d may include an operation interface such as a keyboard or mouse for accepting human input.

[0023] The computer may also have a display 10e. The display 10e may be a liquid crystal display or an OLED display that displays the results of computer processing as an image. The display 10e may also be a projection-type display such as a projector. The display 10e visualizes a 3D model of a forest so that the user can see it.

[0024] Furthermore, the 3D model generation device 10 or its computer may have databases 10f1, 10f2, and 10f3 for storing image data of forests and processed data obtained by processing this data. Databases 10f1, 10f2, and 10f3 are configured to include at least one type of non-transitional physical storage medium, such as semiconductor memory, magnetic media, and optical media. Multiple databases 10f1, 10f2, and 10f3 may be provided depending on the type of data. Multiple databases 10f1, 10f2, and 10f3 may be provided to divide and store a large amount of image data into multiple datasets DS1, DS2, and DS3.

[0025] Multiple datasets are created by dividing image data of a forest due to reasons such as the capacity of the storage medium, operational reasons, and the processing power of the 3D model generation device 10. Each dataset is configured to include multiple image data of a continuous area. Here, within the forest area, the area captured by the image data stored in one dataset will be referred to as a divided area below.

[0026] For example, as shown in Figure 2, divided regions A1, A2, and A3 are adjacent regions in the forest. In a large forest area, it is difficult to complete the imaging of all regions in one go on the same day. For this reason, imaging is carried out over multiple days. For example, imaging of divided region A1 is carried out on the first day, imaging of divided region A2 is carried out on the second day, and imaging of divided region A3 is carried out on the third day. In this example, the image data of divided region A1 is stored in dataset DS1, the image data of divided region A2 is stored in dataset DS2, and the image data of divided region A3 is stored in dataset DS3.

[0027] As shown in Figure 3, the 3D model generation device 10 has a preprocessing unit F1, a set-specific model generation unit F2, and a model integration unit F3 as functional units whose functions are realized when the CPU 10a executes a computer program (3D model generation program) stored in memory 10b.

[0028] The preprocessing unit F1 performs preprocessing on the image data of each dataset. Since the processing for each dataset is similar, the following description will focus on the processing for a single dataset.

[0029] The preprocessor unit F1 sequentially aligns the image data one by one. Alignment here includes image processing to prepare the image data for 3D model generation. For example, the preprocessor unit F1 excludes objects from the image data that are unsuitable for generating a 3D model of a forest, and selects appropriate objects.

[0030] The preprocessor unit F1 recognizes trees, ground, etc., reflected in the image data through image processing using artificial intelligence models, etc. The preprocessor unit F1 selects trees from among the objects reflected in the image data for which positional accuracy can be ensured, as targets for 3D model generation. The preprocessor unit F1 also selects ground from among the objects reflected in the image data for 3D model generation.

[0031] The set-specific model generation unit F2 generates 3D models for each dataset based on the data aligned by the preprocessing unit F1. The set-specific model generation unit F2 constructs a point cloud based on image data for the trees and ground that were recognized by the preprocessing unit F1 and selected as targets for 3D model generation. The points that make up the point cloud represent the 3D positions of the objects to be generated, and for example, one point is generated for each pixel in the image data. That is, the position of one point is the estimated position of the object reflected in the corresponding pixel.

[0032] Next, the set-specific model generation unit F2 constructs a 3D mesh based on the generated point cloud. The mesh represents the 3D shape as a collection of small polygons (e.g., triangles, quadrilaterals). By constructing these point clouds and meshes for all image data stored in the same dataset, the partitioned regions corresponding to that dataset are generated as 3D models.

[0033] The model integration unit F3 integrates multiple models generated for each dataset to create a wider-area 3D model. The model integration unit F3 matches feature points of the 3D models from each dataset and determines the relative positions between different divided regions. Once the relative positions between the divided regions are determined, it becomes possible to connect and integrate multiple 3D models from each dataset.

[0034] The model integration unit F3 restricts the use of data from the inner circumference IS of the divided region during matching and uses data from the outer circumference OS. Here, the outer circumference OS may refer to shooting points that are not enclosed within the interior of virtual lines when virtual lines are drawn to virtually connect all possible combinations of shooting points in a divided region recognized as containing multiple shooting points from multiple image data, and which constitute the outer edge. Conversely, the inner circumference IS may refer to shooting points enclosed within the interior of virtual lines.

[0035] Here, we will explain the overview of the matching process using Figure 4. The circles in Figure 4 schematically represent the locations where the image data was captured. Image data captured at the capture locations located in the outer circumference OS is the data used for matching. Image data captured at the capture locations located in the inner circumference IS is unused data that is not used for matching.

[0036] As shown in Figure 4, where the shooting locations of the outer perimeter OS are connected by straight lines, the model integration unit F3 compares image data of the outer perimeter OS belonging to different datasets DS on a one-to-one basis. The model integration unit F3 performs a matching process on the feature points of the image data to be compared. As shown in Figure 5, if there is an overlapping region OL within the range captured by the image data to be compared, objects captured in the overlapping region OL can be matched. If a match is found in the shape of objects (trees and ground) captured in the image data through feature matching, it is determined that there is a correspondence in the positions of those objects.

[0037] The model integration unit F3 may perform matching processing for all combinations of image data from the outer perimeter OS belonging to different datasets DS. On the other hand, if it can be determined that a combination clearly does not have a corresponding relationship based on its positional relationship with the image data targeted in the previously performed matching processing, the model integration unit F3 may skip the matching processing for that combination.

[0038] Next, an example of a method for generating a 3D model using the 3D model generation device 10 will be explained using the flowchart in Figure 6. The processes in steps S1 to S9 of the flowchart may be implemented by the CPU 10a executing a computer program stored in memory 10b.

[0039] In the initial S1, the first batch of image data is input to the program. That is, the preprocessing unit F1 acquires multiple image data stored in the dataset to be processed first. In S2, after processing in S1, the preprocessing unit F1 aligns the multiple image data to be processed. This completes the alignment of the dataset being processed.

[0040] After processing in S2, S3 determines whether there are any unaligned datasets remaining. If yes, proceed to S4. If no, proceed to S7.

[0041] In S4, the preprocessing unit F1 or the model integration unit F3 extracts the feature points and positions of each image data in the dataset currently being processed. In S5, after processing in S4, the preprocessing unit F1 or the model integration unit F3 deletes the data for the inner circumference IS of the divided region from the feature point and position data of the dataset currently being processed. As a result, only the data for the outer circumference OS is used for relative position matching.

[0042] In S6, following the processing in S5, the next batch of image data is input to the program. That is, the preprocessing unit F1 acquires multiple image data stored in the next dataset to be processed as the next target for processing. After processing in S6, the program proceeds to S2, where processing for the next dataset is executed.

[0043] On the other hand, if the answer to S3 is No, then from S7 onwards, the 3D model generation process is executed. In S7, the set-specific model generation unit F2 constructs a point cloud for each dataset. In S8, after the processing in S7, a mesh is constructed for each dataset. The processes in S7 and S8 are for generating a 3D model for each dataset. For this reason, the point cloud and mesh processing may be executed consecutively for one dataset, and then the point cloud and mesh processing may be executed consecutively for other datasets.

[0044] In S9, following the processing in S8, the model integration unit F3 integrates the dataset-specific models generated in S7 and S8. Specifically, the model integration unit F3 uses the data from the outer perimeter OS of each partitioned region that remained after processing in S4 to perform feature point matching between partitioned regions. Based on the matching results, the model integration unit F3 determines the relative position of each partitioned region and integrates the dataset-specific models. The series of processes ends in S9.

[0045] According to the first embodiment described above, the wide-area 3D model is generated by integrating multiple 3D models that can be generated separately for each dataset DS1, DS2, and DS3. Therefore, the amount of data and processing load handled during generation are reduced compared to generating the wide-area 3D model all at once. In addition, the relative positions of each region are determined by matching the feature points of the outer edges of the divided regions A1, A2, and A3 captured by multiple image data included in datasets DS1, DS2, and DS3. When divided regions A1, A2, and A3 are captured in each dataset DS1, DS2, and DS3, at least partially different from each other, the outer edge OS is most likely to be matchable with the outer edge OS of the other dataset. Furthermore, since the use of data from the inner edge IS is restricted among the regions captured by multiple image data included in the dataset, the amount of data and processing load handled during generation are also reduced in this respect. As a result, the processing load can be reduced in the generation of wide-area 3D models.

[0046] (Second Embodiment) As shown in Figure 7, the second embodiment is a modified version of the first embodiment. The second embodiment will be described focusing on the differences from the first embodiment.

[0047] In the second embodiment, in order to shorten the matching processing time or reduce the load, only a portion of the data from the peripheral OS is used for matching. Specifically, the model integration unit F3 preliminarily recognizes portions of the peripheral OS data that have commonalities with peripheral OS data from other datasets. The model integration unit F3 then deletes all data except for that filtered for use based on the pre-recognition.

[0048] The parts that share common features are the boundary edges (OSBs) that are adjacent to each other in the divided regions and form the boundaries of those regions. In other words, the outer perimeter of a divided region that is not adjacent to another divided region cannot have its feature points matched. Therefore, by excluding these data through prior recognition before performing the matching process, the time or load of the matching process can be improved. Prior recognition is performed by searching for image data with overlapping features using a simpler process that is less accurate than a precise matching process.

[0049] According to the second embodiment described above, in a specific dataset, the portion of the OS image data of the outer periphery adjacent to the partitioned region of another dataset is searched in advance. Then, the feature points of the OSBs that are adjacent portions of the outer periphery OS are matched to determine the relative position of the partitioned region. Since data unsuitable for the matching process is excluded from the data of the outer periphery OS in advance, the processing load in the generation of the 3D model can be further reduced.

[0050] (Third embodiment) As shown in Figures 8 and 9, the third embodiment is a modification of the first embodiment. The third embodiment will be described focusing on the differences from the first embodiment.

[0051] In the third embodiment, the set-specific model generation unit F2 generates a 3D model for each dataset using a neural network. Specifically, the 3D model can be generated using methods such as Gaussian splatting or Neural Radiance Fields (NeRF). These methods generally require a large amount of computation, and due to limitations of processors such as GPUs and RAM, it is difficult to generate wide-area 3D models. Therefore, these methods are adopted as methods for generating set-specific models.

[0052] A neural network is a mathematical model that mimics the nerve cells of the human brain. This neural network may be a deep neural network used in deep learning, a convolutional neural network, or a recurrent neural network.

[0053] The following describes an example using Gaussian splatting. The set-specific model generation unit F2 inputs the image data stored in the dataset, its shooting location, and shooting orientation (which can also be called the shooting direction) as an SFM arrangement (Structure from Motion arrangement) into a neural network-based model, trains the model, and generates a visualized view using the trained model.

[0054] Specifically, the set-based model generation unit F2 generates a point cloud consisting of sparse 3D feature points from the SFM configuration, and generates a set of 3D Gaussian functions from these point clouds. These 3D Gaussian functions may be defined by position, covariance matrix, and opacity. The set-based model generation unit F2 projects the 3D Gaussian functions onto 2D and compares the projection result with image data. Based on the difference between the projection result and the image, the set-based model generation unit F2 adjusts the parameters of the 3D Gaussian functions. The parameters are optimized by repeatedly projecting the adjusted 3D Gaussian functions onto 2D and comparing them with image data. In this way, a Gaussian splatting trained model is generated.

[0055] Once machine learning is complete, the set-specific model generation unit F2 inputs the image data, its shooting location, and shooting orientation into the trained model, and can obtain a view as output. Specifically, as shown in Figure 8, the trained model can render a new viewpoint different from the input image data and output a 3D Gaussian function. This output can be said to be a visualization of the 3D model.

[0056] The set-based model generation unit F2 then sequentially generates 3D models from the second and subsequent datasets. In this case, similar to the first embodiment, relative positions obtained using 2D feature points of the outer periphery OS in the previous dataset are used. That is, 2D feature point generation, machine learning, and visualization model generation are performed using relative positions. In other words, the relative positions between multiple neural network-based models are determined based on a matching process, and an appropriate model is selected from among the multiple models according to the shooting position and orientation of the input image data.

[0057] Next, an example of a method for generating a 3D model using the 3D model generation device 10 will be explained using the flowchart in Figure 9. The processes in steps S201 to S211 of the flowchart may be implemented by the CPU 10a executing a computer program stored in memory 10b.

[0058] In the initial S201, the preprocessing unit F1 acquires multiple image data stored in the dataset to be processed first. In S202, following the processing in S201, the preprocessing unit F1 or the model integration unit F3 generates two-dimensional feature points from the image data. In S203, following the processing in S202, the preprocessing unit F1 aligns the multiple image data to be processed.

[0059] In S204, following the processing in S203, the set-specific model generation unit F2 generates sparse 3D feature points based on the image data. In S205, following the processing in S204, the set-specific model generation unit F2 generates a 3D Gaussian function from the sparse 3D feature points. In S206, following the processing in S205, the set-specific model generation unit F2 performs machine learning on the model based on the 3D Gaussian function. In S207, following the processing in S206, the set-specific model generation unit F2 generates a 3D model for the currently processed dataset based on the results of the machine learning.

[0060] In S208, after processing in S207, it is determined whether or not there are any unaligned datasets remaining. If yes, proceed to S209. If no, proceed to S212.

[0061] In S209, the preprocessing unit F1 or the model integration unit F3 deletes the 2D feature points of the inner circumference IS in the currently processed dataset (also referred to as the previous dataset in contrast to the next dataset). As a result, only the data of the outer circumference OS is used for relative position matching. If the currently processed dataset is the first dataset, the 2D feature points to be deleted are those generated in S202. If the currently processed dataset is the second or later dataset, the 2D feature points to be deleted are those generated in the previous S211.

[0062] In S210, following the processing in S209, the preprocessing unit F1 acquires multiple image data stored in the dataset to be processed next. In S211, following the processing in S210, the set-specific model generation unit F2 performs the same processing as in S202 to S207 using the relative positions obtained using the two-dimensional feature points of the outer perimeter OS. After processing in S211, the process returns to S208.

[0063] On the other hand, in S212, if the answer to S208 is No, the model integration unit F3 integrates the dataset-specific models generated in the processing of S207 and S211. The integrated model here is configured in which multiple models are related to each other by relative positions defined by the matching process, and the model to be used for generating the view can be selected from among the multiple models that are related to each other based on the input data. The series of processes ends with S212.

[0064] According to the third embodiment described above, in generating a wide-area 3D model, multiple neural network-based pre-trained models are trained on multiple image data contained in each dataset, separately for each dataset. Then, the multiple pre-trained models are associated with each other based on the relative positions determined using the data of the outer OS. By having the 3D model composed of multiple pre-trained models that are associated with each other, a wide-area visualization environment can be constructed while reducing the processing load.

[0065] (Other embodiments) Although several embodiments have been described above, this disclosure is not limited to those embodiments and can be applied to various embodiments and combinations without departing from the spirit of this disclosure.

[0066] In another embodiment, the total number of datasets may be multiple, ranging from two to four or more.

[0067] In another embodiment, the multiple data sets DS1, DS2, and DS3 do not necessarily have to be stored on separate storage media, but may be stored on the same storage medium.

[0068] In another embodiment, the 3D model generation device 10 may not have to include at least a portion of the databases 10f1, 10f2, and 10d3 that store the datasets DS1, DS2, and DS3. The 3D model generation device 10 may prepare the datasets DS1, DS2, and DS3 by acquiring at least a portion of the datasets DS1, DS2, and DS3 from an external source.

[0069] In another embodiment, each imaging region in the image data stored in each dataset may be at least partially different from each other, and does not necessarily have to be a simple division of a wide area. Below are some examples of the relationship between two imaging regions Aa and Ab, which correspond individually to two datasets DSa and DSb.

[0070] As shown in Figure 10, the imaging regions Aa and Ab may have some boundary edges of the outer periphery OS touching each other. This relationship can be described as a relationship in which the region is effectively divided.

[0071] As shown in Figure 11, the imaging regions Aa and Ab may have some boundary edges separated from each other within the outer periphery OS, but they may be arranged within a predetermined separation distance DST. In other words, it is sufficient that the portion of the image data that captures the distant view corresponds to the overlapping region OL that can be matched. The predetermined separation distance DST may be, for example, 6m. Note that adjacency in the second embodiment may include relationships where the regions are arranged within a predetermined separation distance.

[0072] As shown in Figure 12, the imaging regions Aa and Ab may partially overlap. In this case, since there will be two or more points Pc1 and Pc2 that intersect the outer edge OS, matching processing can be performed using the data of points Pc1 and Pc2.

[0073] As shown in Figure 13, the imaging regions Aa and Ab may be in a relationship where their outer edges OS touch at a single point Pc3. In this case, matching processing can be performed using the data from point Pc3.

[0074] As shown in Figure 14, imaging region Aa encompasses imaging region Ab, and imaging regions Aa and Ab may have parts of their outer periphery OS touching or intersecting each other.

[0075] In another embodiment, the computer constituting the 3D model generation device 10 may include other types of processors (e.g., a GPU) in place of, or together with, the CPU 10a.

[0076] In another embodiment, the computer constituting the 3D model generation device 10 may incorporate circuits such as FPGAs, and a portion of the processing of the 3D model generation device 10 may be realized by the processor controlling the circuits based on a computer program, or by the circuits operating independently. [Explanation of Symbols]

[0077] 10: 3D model generation device, 10a: CPU (processing unit), A1, A2, A3, Aa, Ab: regions, DS1, DS2, DS3, DSa, DSb: datasets, IS: inner circumference, OS: outer circumference

Claims

1. A three-dimensional model generation apparatus comprising at least one processing unit (10a) that generates a three-dimensional model from image data captured by a camera, The at least one processing unit is, This involves preparing multiple datasets (DS1, DS2, DS3, DSa, DSb) containing multiple image data of regions (A1, A2, A3, Aa, Ab) that are at least partially different from each other, The system is configured to generate a wide-area 3D model in order to integrate multiple 3D models that can be generated for each dataset, A three-dimensional model generating apparatus that generates the aforementioned wide-area three-dimensional model, which includes restricting the use of data from the inner circumference (IS) of each region, matching feature points of the outer circumference (OS) to determine the relative position of each region.

2. Determining the aforementioned relative position is The process involves pre-searching the portion (OSB) of the image data in the outer periphery of a specific dataset that is adjacent to the region of another dataset, A three-dimensional model generation apparatus according to claim 1, comprising matching the feature points of adjacent portions of the outer periphery to determine the relative position.

3. Generating the aforementioned wide-area three-dimensional model is From the multiple image data contained in each of the aforementioned datasets, multiple neural network-based trained models are subjected to machine learning for each of the aforementioned datasets. A three-dimensional model generation apparatus according to claim 1, comprising associating a plurality of the trained models based on the determined relative positions.

4. A 3D model generation program that generates a 3D model from image data captured by a camera, At least one processing unit (10a) This involves preparing multiple datasets (DS1, DS2, DS3, DSa, DSb) containing multiple image data of regions (A1, A2, A3, Aa, Ab) that are at least partially different from each other, The system is configured to generate a wide-area 3D model in order to integrate multiple 3D models that can be generated for each of the aforementioned datasets, A three-dimensional model generation program that generates the aforementioned wide-area three-dimensional model includes restricting the use of data from the inner circumference (IS) of each region, matching feature points of the outer circumference (OS) to determine the relative position of each region.

Images