Three-dimensional space reconstruction device and method

Abstract

One embodiment of the present invention acquires first and second three-dimensional spatial models constructed on the basis of video data captured of a location including a target space under a first condition and a second condition different from the first condition, respectively. Then, first, a first target area that is not to be reconstructed and a second target area that is to be reconstructed are set respectively for a first image generated using the first three-dimensional spatial model and a second image generated using the second three-dimensional spatial model to generate first and second three-dimensional spatial maps. Next, a common three-dimensional spatial model representing the positional relationship in three-dimensional space between the first three-dimensional spatial model and the second three-dimensional spatial model and the correspondence between the first three-dimensional spatial map and the second three-dimensional spatial map is generated, and a third three-dimensional spatial model is generated by combining and joining the first three-dimensional spatial model and the second three-dimensional spatial model on the basis of the common three-dimensional spatial model so that the second target area is the target of reconstruction instead of the first target area.

Description

3D space reconstruction device and method

[0001] One aspect of the present invention relates to a device and method for reconstructing a three-dimensional space model used in a system for generating three-dimensional images of objects or scenes using three-dimensional space reconstruction technology.

[0002] In recent years, there has been a growing expectation for the realization of a service that reconstructs a three-dimensional space based on video captured outdoors, allowing users to experience the outdoors while indoors by viewing the scenery in the reconstructed three-dimensional outdoor space.

[0003] To realize this service, a technology for reconstructing a three-dimensional space from a two-dimensional image is required. One such three-dimensional space reconstruction technology is known as the Neural Radiance Field (NeRF) method. NeRF is a technology that creates a cubic grid as a representation domain for expressing a three-dimensional space, and reconstructs the three-dimensional shape and color of an object or landscape by using a deep learning model to learn the reflectance and transmittance of each grid in the representation domain from a two-dimensional image and the camera parameters of that image (see, for example, Non-Patent Document 1).

[0004] Now, consider the application of a 3D space reconstruction technology such as NeRF to reconstruct, for example, an outdoor location as a 3D space. In this case, when capturing outdoor video to obtain a 2D image, in places with heavy traffic or many structures, it is possible that the background or buildings necessary for reconstruction may not be captured because they are obstructed by cars, people, structures, etc. In such cases, it is difficult for existing 3D space reconstruction technologies to correctly reconstruct the obstructed 3D space.

[0005] Therefore, the inventor is studying a method of capturing images of the location to be reconstructed under different conditions to obtain two images, interpolating the area that is occluded in one image using the other image, and constructing an NeRF model using this interpolated image.

[0006] B. Mildenhall, et, al, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” ECCV 2020.

[0007] However, the above-described construction method under consideration performs interpolation processing on two-dimensional images required for constructing an NeRF model. Therefore, it becomes necessary to retrain the NeRF model according to the type of service to which it is applied. For example, there are service types in which a three-dimensional space needs to be reconstructed by interpolating all occluded areas, and in which a three-dimensional space needs to be reconstructed by interpolating only a specific area. In order to construct an NeRF model according to these different service types, it is necessary to re-interpolate the two-dimensional image each time and re-train the NeRF model using the re-interpolated binary image. Therefore, constructing a three-dimensional space model such as an NeRF model requires a large processing load and time, which causes an increase in costs.

[0008] This invention has been made in light of the above circumstances, and aims to provide a technology that reduces the processing load and time required to construct a three-dimensional space model that reduces the influence of occluded areas, thereby enabling cost reduction for constructing the three-dimensional space model.

[0009] In order to solve the above problem, one aspect of a 3D space reconstruction device or method according to the present invention acquires a first 3D space model constructed based on first video data obtained by capturing an image of a location including a target space under first conditions, and a second 3D space model constructed based on second video data obtained by capturing an image of the location under second conditions different from the first conditions. Then, for a first image generated using the first 3D space model and a second image generated using the second 3D space model, a first target region that is not to be reconstructed and a second target region that is to be reconstructed are set, respectively, and a first 3D space map and a second 3D space map that represent the set first target region and second target region are generated. Next, a common three-dimensional spatial model is generated that represents the positional relationship in three-dimensional space between the first three-dimensional spatial model and the second three-dimensional spatial model and the correspondence relationship between the first three-dimensional spatial map and the second three-dimensional spatial map, and a third three-dimensional spatial model is generated based on the common three-dimensional spatial model by combining the first three-dimensional spatial model and the second three-dimensional spatial model so that the second target area is the target of reconstruction instead of the first target area.

[0010] According to one aspect of the present invention, a combined third three-dimensional space model is constructed by combining a first three-dimensional space model constructed based on video data of a certain location captured under first conditions with a second three-dimensional space model constructed based on video data of the same location captured under second conditions. Therefore, even if the video data used to construct the first three-dimensional space model includes an occluded area, which makes it impossible to accurately reconstruct the three-dimensional space from the first three-dimensional space model, by using the third three-dimensional space model, it is possible to accurately reconstruct the three-dimensional space including the occluded area.

[0011] Furthermore, by combining the first and second three-dimensional space models that have already been constructed, a third three-dimensional space model that can reconstruct a three-dimensional space including the above-mentioned occluded area is constructed.Therefore, it is possible to construct a three-dimensional space model that can reconstruct a three-dimensional space including the occluded area with less processing load and in an extremely short time compared to acquiring new video data that includes the above-mentioned occluded area and reconstructing a three-dimensional space model based on this acquired video data, thereby making it possible to reduce the cost of constructing the three-dimensional space model.

[0012] In other words, according to one aspect of the present invention, the processing load and time required to construct a three-dimensional spatial model that eliminates the influence of occluded areas are reduced, thereby providing a technology that enables cost reduction for constructing a three-dimensional spatial model.

[0013] FIG. 1 is a block diagram showing an example of a hardware configuration of a three-dimensional space reconstruction device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a software configuration of the three-dimensional space reconstruction device according to an embodiment of the present invention. FIG. 3 is a flowchart showing an example of a three-dimensional space model construction process executed by a control unit of the three-dimensional space reconstruction device shown in FIG. 2. FIG. 4 is a flowchart showing an example of a processing procedure and processing content of a target area setting process in the three-dimensional space model construction process shown in FIG. 3. FIG. 5 is a flowchart showing an example of a processing procedure and processing content of a target area combining process in the three-dimensional space model construction process shown in FIG. 3. FIG. 6 is a flowchart showing an example of a processing procedure and processing content of a combined model generation process in the three-dimensional space model construction process shown in FIG. 3. FIG. 7 is a diagram for explaining an example of a target area setting process shown in FIG. 4. FIG. 8A shows a NeRF model M constructed based on video data captured under condition A. A FIG. 8B is a diagram showing an example of the NeRF model M constructed based on the video data captured under condition B. B FIG. 9 is a diagram showing an example of the NeRF model M A The three-dimensional space represented by the NeRF model M B6. FIG. 10 is a diagram showing an example of a three-dimensional space represented by a common three-dimensional space model expressed in voxels. FIG. 11A is a diagram for explaining an example of the target region combining process shown in FIG. 5, showing an example of a three-dimensional space map. FIG. 11B is a diagram for explaining an example of the target region combining process shown in FIG. 5, showing an example of a target region and a non-target region in the common three-dimensional space model. FIG. 12 is a diagram for explaining an example of the combined model generation process shown in FIG. 6.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] [One Embodiment] (Configuration Example) FIGS. 1 and 2 are block diagrams showing an example of the hardware configuration and software configuration, respectively, of a three-dimensional space reconstruction device according to one embodiment of the present invention.

[0016] The three-dimensional space reconstructing apparatus 1 according to an embodiment is configured as, for example, a personal computer. The three-dimensional space reconstructing apparatus 1 may be configured as a server computer installed on the web or on the cloud.

[0017] The three-dimensional space reconstruction device 1 includes a control unit 10 that uses a hardware processor such as a central processing unit (CPU), and a storage unit having a program storage unit 20 and a data storage unit 30, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 40 are connected to the control unit 10 via a bus 50.

[0018] An input device 2 and a display device 3 are connected to the input / output I / F unit 40. The input device 2 is composed of, for example, a keyboard and a mouse, and is used by a user or a system administrator to input information specifying a target area to be reconstructed, commands for instructing the 3D space reconstruction device 1 to perform various operations, and the like.

[0019] The display device 3 is used to display information indicating various operating states of the three-dimensional space reconstruction device 1 and information necessary for input operations, as well as to display three-dimensional space images reconstructed from the constructed three-dimensional space model.

[0020] An external storage medium (not shown) can be connected to the input / output I / F unit 40. A group of constructed NeRF models is stored in the external storage medium, and the 3D space reconstructing device 1 acquires the group of constructed NeRF models from the external storage medium.

[0021] A camera (not shown) may be connected to the input / output I / F unit 40. The camera captures images of a location including the target space to be reconstructed under a plurality of different conditions, and transmits the obtained video data to the 3D space reconstruction device 1. A plurality of cameras may be arranged at different positions.

[0022] The input / output I / F unit 40 may also include a communication interface. By including the communication interface, the 3D space reconstruction device 1 can acquire the constructed NeRF models from a server device or a terminal device on the Web or the cloud via a network. Furthermore, even if the camera is located far away, it can acquire video data transmitted from the camera via the network.

[0023] The program storage unit 20 is configured by combining, for example, a nonvolatile memory such as a solid-state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various controls according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0024] The data storage unit 30 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and the storage area thereof is provided with an NeRF model storage unit 31, a 3D space map storage unit 32, a common 3D space model storage unit 33, and a combined NeRF model storage unit 34 as main storage units related to one embodiment of the present invention.

[0025] The NeRF model storage unit 31 is used to store a group of constructed NeRF models acquired from an external storage medium, a server device, or the like.

[0026] The three-dimensional space map storage unit 32 is used to store a three-dimensional space map generated for each NeRF model by the control unit 10, which will be described later.

[0027] The common three-dimensional space model storage unit 33 is used to store a common three-dimensional space model generated by the control unit 10, which will be described later.

[0028] The coupled NeRF model storage unit 34 is used to store the coupled NeRF model generated by the control unit 10, which will be described later.

[0029] The control unit 10 includes, as processing functions necessary for implementing one embodiment, an NeRF model acquisition processing unit 11, a target area setting processing unit 12, a target area combining processing unit 13, a combined model generation processing unit 14, and an output processing unit 15.

[0030] Each of the processing units 11 to 15 is realized by causing a hardware processor of the control unit 10 to execute an application program stored in the program storage unit 20. Note that some or all of the processing units 11 to 16 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0031] The NeRF model acquisition processing unit 11 acquires a group of constructed NeRF models from, for example, an external storage medium or a server device via the input / output I / F unit 40 , and stores the acquired NeRF models in the NeRF model storage unit 31 .

[0032] The acquired NeRF model group includes a first NeRF model constructed based on video data captured under a first condition and a second NeRF model constructed based on video data captured under a second condition different from the first condition. The first and second conditions may be, for example, different time periods or imaging directions. Note that the number of acquired NeRF model groups is not limited to two, and may be three or more.

[0033] The target region setting processing unit 12 reads the first and second NeRF models from the NeRF model storage unit 31, and sets regions to be reconstructed or not to be reconstructed for each of the three-dimensional space images generated by rendering from the read first and second NeRF models. The target region setting processing unit 12 then generates three-dimensional space maps representing the set target regions, and stores the generated three-dimensional space maps in the three-dimensional space map storage unit 32. An example of the target region setting process and the three-dimensional space map generation process will be described in the operation example.

[0034] The target area merging processing unit 13 reads the first and second NeRF models and the three-dimensional space maps corresponding to these NeRF models, and generates a common three-dimensional space model that represents the positional relationship between the first and second NeRF models and the correspondence between the three-dimensional space maps.

[0035] Furthermore, the target region combination processing unit 13 derives a transformation matrix for transforming the three-dimensional space coordinates in the common three-dimensional space model into the three-dimensional space coordinates of the first and second NeRF models.

[0036] The target area combination processing unit 13 stores the generated common three-dimensional space model and transformation matrix in the common three-dimensional space model storage unit 33. An example of the process of generating the common three-dimensional space model and the process of deriving the transformation matrix will be described in the operation example.

[0037] The combined model generation processing unit 14 reads the first and second NeRF models, the common three-dimensional model, and the transformation matrix, and generates a combined NeRF model by combining the first NeRF model and the second NeRF model. An example of the generation process of this combined NeRF model will also be described in the operation example.

[0038] The output processing unit 15 generates a three-dimensional spatial image reconstructed using the coupled NeRF model, and outputs the generated three-dimensional spatial image from the input / output I / F unit 40 to the display device 3 for display.

[0039] (Example of Operation) Next, an example of operation of the three-dimensional space reconstruction device 1 configured as above will be described.

[0040] FIG. 3 is a flowchart showing an example of the processing procedure and processing content of the three-dimensional space model construction processing executed by the control unit 10 of the three-dimensional space reconstruction device 1.

[0041] (1) Acquisition of NeRF Model Group When a model construction request is detected in step S1, the control unit 10 of the three-dimensional space reconstruction device 1 first acquires an NeRF model group in step S2 under the control of the NeRF model acquisition processing unit 11, and performs processing to store the acquired NeRF group in the NeRF model storage unit 31.

[0042] For example, the NeRF model acquisition processing unit 11 acquires a group of constructed NeRF models from an external storage medium or a server device via the input / output I / F unit 40, and stores them in the NeRF model storage unit 31. The NeRF model group to be acquired includes the first NeRF model M A and the second NeRF model M B The first NeRF model M A is constructed by training a neural network based on image data captured in a location including the target area to be reconstructed at a time period or from a direction set in advance as the first condition A. On the other hand, the second NeRF model M Bwas constructed by training a neural network based on video data captured at the same location under second condition B, which is different from the first condition A in terms of time or direction of capture.

[0043] The NeRF model acquisition processing unit 11 repeatedly executes the above NeRF model acquisition processing for each model, and acquires all the necessary NeRF models, that is, the first NeRF model M A and the second NeRF model M B If it is determined that the NeRF model has been acquired, the NeRF model acquisition process ends.

[0044] It is also possible that the 3D space reconstruction device 1 has a function of constructing an NeRF model within the device itself based on video data captured by a camera. In this case, the NeRF model acquisition processing unit 11 acquires a group of NeRF models constructed using the 3D space model construction function and stores them in the NeRF model storage unit 31. As a means for constructing the 3D space model, a well-known NeRF model learning method, for example, "Instant NeRF," can be used.

[0045] (2) Setting of target region and generation of 3D space map Next, in step S4, the control unit 10 of the 3D space reconstruction device 1 sets the first and second NeRF models M A , M B The process of setting the target region required for reconstruction and the process of generating a three-dimensional space map representing this target region are executed as follows.

[0046] FIG. 4 is a flowchart showing an example of the processing procedure and processing content of the target area setting processing and the three-dimensional space map generating processing executed by the target area setting processing unit 12.

[0047] That is, first, in step S41, the target region setting processing unit 12 retrieves the first NeRF model M from the NeRF model storage unit 31. A Next, in step S42, the target region setting processing unit 12 reads the first NeRF model M AThen, in step S43, a target region is set for each of the generated three-dimensional spatial images.

[0048] 7 shows an example of the target region setting process. In this example, the target region setting processing unit 12 sets the first NeRF model M A Then, the generated three-dimensional space images are displayed on the display device 3 via the input / output I / F unit 40. In this state, the user or the system administrator can view the first NeRF model M A An operation is performed to set a region that is not to be reconstructed in the three-dimensional space image corresponding to the above.

[0049] For example, now, the NeRF Model M A 8A, the user or system administrator sets the area occluded by the group of people (the white area in the figure) as an area not to be reconstructed. As a result, target area images a1 to a3 as shown in FIG. 7 are generated.

[0050] The target region may be set by the target region setting processing unit 12 extracting an image of an obstructing object such as a person through image processing using an existing object detection model, etc. As the object detection model, for example, a segmentation model (Mask R-CNN, etc.) or a multi-view segmentation model can be used.

[0051] Subsequently, in step S44, the target region setting processing unit 12 generates a three-dimensional space map (3D mask) m indicating the position of the target region in the three-dimensional space represented by the NeRF model, based on each of the target region images and the NeRF model. A This three-dimensional space map m A To generate the three-dimensional space map m, for example, a model that performs three-dimensional segmentation from a plurality of two-dimensional images is used. Ais stored in the three-dimensional space map storage unit 32.

[0052] The target region setting processing unit 12 calculates the first and second NeRF models M A , M B In step S45, it is determined whether the setting of the target region for the NeRF model and the generation of the 3D space map have been completed. If there are any unprocessed NeRF models remaining, the process returns to step S41 and repeats the series of processes from step S41 to step S44.

[0053] (3) Generation of Common 3D Space Model and Deriving of Transformation Matrix Subsequently, in step S5, under the control of the target area combination processing unit 13, the control unit 10 of the 3D space reconstruction device 1 executes a process of generating a common 3D space model and a process of deriving a transformation matrix as follows.

[0054] FIG. 5 is a flowchart showing an example of the processing procedure and processing content of the processing executed by the target region merger 13.

[0055] That is, first, in step S51, the target region combining processing unit 13 retrieves the first and second NeRF models M from the NeRF model storage unit 31. A , M B At the same time, in step S52, the object region combining processing unit 13 reads the first and second NeRF models M A , M B The three-dimensional space map m A , m B Load.

[0056] The first and second NeRF models M A , M B are constructed based on video data captured under different conditions A and B, and therefore have different rendering ranges and positional relationships in real coordinates, as shown in FIG. 9. Therefore, in step S53, the target area integration processing unit 13 integrates the first and second NeRF models M A , M BA common three-dimensional space model M is expressed by dividing a common three-dimensional grid between the two regions by voxels of a predetermined size (for example, 256×256×256). AB The common three-dimensional space model M is created as shown in FIG. AB The image of the configuration is shown below.

[0057] Next, in step S54, the target region combining processing unit 13 combines the common three-dimensional space model M AB The common three-dimensional space model M is created so as to include the target area and the objects that are commonly drawn under conditions A and B. AB The three-dimensional spatial coordinates of the first and second NeRF models M A , M B Transformation matrix T for transforming each of the three-dimensional space coordinates A , T B Then, the target region combining unit 13 derives the derived transformation matrix T A , T B are stored in the common three-dimensional space model storage unit 33 together with the common three-dimensional space model.

[0058] Subsequently, in step S55, the target region combining processing unit 13 combines the common three-dimensional space model M AB In this case, a three-dimensional spatial map m A , m B Based on this, a tag is assigned to each voxel indicating whether the image data under condition A or condition B is used for reconstruction.

[0059] In this example, as shown in FIG. 11A, the three-dimensional space map m A 11B, the target region d1 represented by the following equation is not a target to be drawn, and only d2 is a target to be drawn. For this reason, in the common three-dimensional space model, tags are assigned to the voxels so that the target region e1 is reconstructed based on the video data captured under condition B, and the target region e2 is reconstructed based on the video data captured under condition A.

[0060] (4) Construction of a Combined Model Next, in step S6, the control unit 10 of the three-dimensional space reconstruction device 1 generates the first NeRF model M under the control of the combined model generation processing unit 14. A and the second NeRF model M B and combine them to build a combined NeRF model, which is carried out as follows:

[0061] FIG. 6 is a flowchart showing an example of the processing procedure and processing contents of the combined NeRF model construction processing executed by the combined model generation processing unit 14.

[0062] That is, the binding model generation processing unit 14 first obtains the first and second NeRF models M from the NeRF model storage unit 31 in step S61. A , M B At the same time, in step S62, the combined model generation processing unit 14 reads the common three-dimensional space model M from the common three-dimensional space model storage unit 33. AB and the transformation matrix T A , T B Load.

[0063] Next, in step S63, the combined model generation processing unit 14 generates the common three-dimensional space model M AB A virtual viewpoint is set in the target region e1, and a three-dimensional spatial image seen from the set virtual viewpoint is generated by rendering. Then, the combined model generation processing unit 14 determines, for each of a plurality of light rays on the rendering plane of the three-dimensional spatial image, whether or not the light ray passes through the target region e1.

[0064] FIG. 12 shows the common three-dimensional space model M AB 1 shows an example of an image rendered from the ray r and each ray on the rendering surface of the image. In this example, the ray r shown by the solid line B is a ray r shown by a dashed line passing through the target region e1. A does not pass through the target region e1. Note that CM in Fig. 12 indicates the position of the virtual viewpoint.

[0065] Subsequently, in step S64, the combined model generation processing unit 14 calculates the ray r determined to pass through the target region e1. BThe information representing the target region e1 is converted into a transformation matrix T B The second NeRF model M B Then, in step S65, the converted light ray r B Information representing the second NeRF model NN B By inputting the above ray r B Obtain color and density information.

[0066] At the same time, the combined model generation processing unit 14 also determines in step S66 that the ray r does not pass through the target region e1. A The information representing the transformation matrix T A The first NeRF model M A Then, in step S67, the converted light ray r A The information representing the first NeRF model NN A By inputting the above ray r A Obtain color and density information.

[0067] That is, by the above processing, the first NeRF model M A Rendered image C'(r B ) and the second NeRF model M B Rendered image C'(r B ) is obtained. A ), C'(r B ) is expressed by the following equation (1): where R in equation (1) represents a function that represents volume rendering.

[0068]

[0069] Subsequently, in step S68, the combined model generation processing unit 14 converts the rendering image C'(r A ), C'(r B) is subjected to color correction. For color correction, for example, a color chart is placed in the captured image data, and a color model corresponding to each of the image capturing conditions A and B is created in advance. Then, the rendering image C'(r A ), C'(r B ) to reduce the color difference between the C′(r A ), C'(r B ) is adjusted by adjusting the color gain of the rendering image C′(r A ), C'(r B ) This can make the color differences that occur between the two less noticeable.

[0070] In addition, color correction is performed by, for example, creating a deep learning model for color correction in advance, and applying the above C′(r A ), C'(r B ) can be input.

[0071] Finally, in step S69, the combined model generation processing unit 14 generates the color-corrected rendering image C'(r A ) and the rendering image C'(r B ) and a common three-dimensional space model M AB Then, the combined model generation processing unit 14 generates a rendering image C(r) on the common three-dimensional space model M AB The rendered image C(r) on the left is stored in the combined NeRF model storage unit 34. Thus, the first NeRF model M A The region not to be reconstructed is the second NeRF model M B The combined NeRF model is obtained by replacing the region of interest in the reconstruction of

[0072] (5) Output of Reconstructed Image of Three-Dimensional Space Under the control of the output processing unit 15, the control unit 10 of the three-dimensional space reconstruction device 1 generates a reconstructed image of three-dimensional space corresponding to a viewpoint specified by, for example, a user or a system administrator, using the coupled NeRF model stored in the coupled NeRF model storage unit 34. Then, the generated reconstructed image of three-dimensional space is output from the input / output I / F unit 40 to the display device 3 for display.

[0073] (Effects) As described above, in one embodiment, the first NeRF model M is constructed based on image data captured under the first condition A in a location including a target region to be reconstructed. A and a second NeRF model M constructed based on image data captured at the same location under second conditions B, which are different from the first conditions A in terms of time or direction of capture. B Then, the first and second NeRF models M A , M B , and generate a three-dimensional space map representing the target region. A , M B A common three-dimensional space model M is expressed by dividing the common three-dimensional grid between AB and the common three-dimensional space model M AB The three-dimensional spatial coordinates of the first and second NeRF models M A , M B Transformation matrix T for transforming each of the three-dimensional space coordinates A , T B Next, the common three-dimensional space model M AB and the above transformation matrix T A , T B and the first NeRF model M A and the second NeRF model M B and are combined so that their respective target regions are combined, thereby constructing a combined NeRF model.

[0074] Thus, according to one embodiment, the first NeRF model MA The image data used to construct the first NeRF model M A Even if the three-dimensional space cannot be accurately reconstructed from the first NeRF model M A The second NeRF model M B By using a combined NeRF model constructed by combining the above, it becomes possible to accurately reconstruct a three-dimensional space including the above-mentioned occluded region.

[0075] Moreover, the first and second NeRF models M A , M B By combining the above, a combined NeRF model capable of reconstructing a three-dimensional space including the occluded area is constructed. Therefore, compared to acquiring new video data including the occluded area and then reconstructing a three-dimensional space model based on this acquired video data, it is possible to construct a three-dimensional space model capable of reconstructing a three-dimensional space including the occluded area with a small processing load and in an extremely short time, thereby making it possible to reduce the cost for constructing a three-dimensional space model.

[0076] In one embodiment, the first and second NeRF models M A , M B When combining these NeRF models M A , M B Color correction is performed to reduce the color difference between the images, thereby reducing color variations in the reconstructed 3D spatial image and enabling the generation of a higher quality 3D spatial image.

[0077] [Other Embodiments] (1) In the above embodiment, two NeRF models M A , M B and the two NeRF models M A , M BHowever, three or more NeRF models with different imaging conditions may be acquired and the acquired three or more NeRF models may be selectively combined to construct a combined NeRF model. In this way, it is possible to construct a combined NeRF model that more reliably eliminates occluded areas, thereby enabling the reconstruction of a three-dimensional spatial image of even higher quality.

[0078] (2) The functional configuration of the three-dimensional space reconstruction device, its processing procedure and processing content, the type of neural network used in the learning model and its algorithm, the type and use of the image to be reconstructed, etc. can be modified and implemented in various ways without departing from the spirit of this invention.

[0079] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0080] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0081] REFERENCE SIGNS LIST 1... 3D space reconstruction device 2... input device 3... display device 10... control unit 20... program storage unit 30... data storage unit 40... input / output I / F unit 50... bus 11... NeRF model acquisition processing unit 12... target area setting processing unit 13... target area combining processing unit 14... combined model generation processing unit 15... output processing unit 31... NeRF model storage unit 32... 3D space map storage unit 33... common 3D space model storage unit 34... combined NeRF model storage unit

Claims

1. A first processing unit that acquires a first three-dimensional space model constructed based on first video data obtained by capturing an image of a location including a target space under first conditions, and a second three-dimensional space model constructed based on second video data obtained by capturing an image of the location under second conditions different from the first conditions; a second processing unit that sets a first target area that is not to be reconstructed and a second target area that is to be reconstructed for a first image generated using the first three-dimensional space model and a second image generated using the second three-dimensional space model, respectively, and generates a first three-dimensional space map and a second three-dimensional space map that respectively represent the set first target area and second target area; a third processing unit that generates a common three-dimensional space model that represents a positional relationship in three-dimensional space between the first three-dimensional space model and the second three-dimensional space model and a correspondence relationship between the first three-dimensional space map and the second three-dimensional space map; and a fourth processing unit that generates a third three-dimensional spatial model by combining and integrating the first three-dimensional spatial model and the second three-dimensional spatial model based on the common three-dimensional spatial model such that the second target area is a target for reconstruction instead of the first target area.

2. The 3D spatial reconstruction device of claim 1, wherein the third processing unit further performs a process of generating a transformation matrix for transforming each of the 3D spatial coordinates of the first target area and the second target area in the common 3D spatial model into the 3D spatial coordinates of the first 3D spatial model and the 3D spatial coordinates of the second 3D spatial model, respectively; and the fourth processing unit uses the transformation matrix to transform the 3D spatial coordinates of a 3D spatial image generated from the common 3D spatial model into the 3D spatial coordinates of the first 3D spatial model and the 3D spatial coordinates of the second 3D spatial model, respectively, and combines the transformed 3D spatial image of the first 3D spatial model and the 3D spatial image of the second 3D spatial model.

3. The three-dimensional space reconstruction device according to claim 1, wherein, when combining the first three-dimensional space model and the second three-dimensional space model, the fourth processing unit performs correction on an image generated from the first three-dimensional space model and an image generated from the second three-dimensional space model to reduce color difference between the two images.

4. A three-dimensional space reconstruction method executed by an information processing device, comprising the steps of: acquiring a first three-dimensional space model constructed based on first image data obtained by capturing an image of a location including a target space under a first condition; and acquiring a second three-dimensional space model constructed based on second image data obtained by capturing an image of the location under a second condition different from the first condition; and setting a first target area that is not to be reconstructed and a second target area that is to be reconstructed for a first image generated using the first three-dimensional space model and a second image generated using the second three-dimensional space model, respectively, and generating a first three-dimensional space map and a second three-dimensional space map that respectively represent the set first target area and second target area. a step of generating a common three-dimensional space model that represents a positional relationship in three-dimensional space between the first three-dimensional space model and the second three-dimensional space model and a correspondence relationship between the first three-dimensional space map and the second three-dimensional space map, and generating a transformation matrix for transforming each of the three-dimensional space coordinates of the first target area and the second target area in the common three-dimensional space model into the three-dimensional space coordinates of the first three-dimensional space model and the three-dimensional space coordinates of the second three-dimensional space model, respectively; and a step of generating a third three-dimensional space model by combining and joining the first three-dimensional space model and the second three-dimensional space model based on the common three-dimensional space model and the transformation matrix so that the second target area is a reconstruction target instead of the first target area.

Images