Information processing method, information processing system, and information processing program

Abstract

This three-dimensional reconstruction system includes a transparent device, a camera, and a reconstruction device. The transparent device switches between a non-display state in which a reference object is not displayed and a display state in which the reference object is displayed. The camera outputs subject image data obtained by capturing a subject placed on the transparent device in the non-display state, and outputs reference object image data obtained by capturing the reference object displayed on the transparent device in the display state. The reconstruction device acquires the subject image data and the reference object image data.

Description

Information processing method, information processing system, and information processing program

[0001] The present invention relates to an information processing method, an information processing system, and an information processing program.

[0002] There are known techniques for three-dimensionally reconstructing a subject by using images or videos of the subject captured from multiple viewpoints, such as photogrammetry, Neural Radiance Fields (NeRF), and 3D Gaussian Splatting (3DGS).

[0003] In 3D reconstruction technology, the more viewpoints there are, the better the quality of the output 3D model is expected to be, so it is necessary to capture images from as many viewpoints as possible. In addition, in 3D reconstruction technology, markers or charts are sometimes placed along with the subject and used for camera pose estimation, real scale acquisition, and image quality evaluation in preprocessing of 3D reconstruction.

[0004] JP 2002-232768 A International Publication No. 2023 / 223916

[0005] However, it is difficult to improve the accuracy of three-dimensional reconstruction of an object, for example, it is difficult to photograph the object from as wide a viewpoint as possible and to arrange markers or charts as widely as possible around the object.

[0006] The present invention has been made in view of the above, and has as its object to improve the accuracy of three-dimensional reconstruction of an object.

[0007] An information processing method according to one embodiment of the present disclosure is an information processing method executed by an information processing system including a display device, a photographing device, and an information processing device, wherein the display device switches between a first state in which a reference object is not displayed and a second state in which the reference object is displayed, the photographing device outputs first photographing data obtained by photographing a subject placed on the display device in the first state, and outputs second photographing data obtained by photographing the reference object displayed on the display device in the second state, and the information processing device acquires the first photographing data and the second photographing data.

[0008] 1 is a diagram illustrating a configuration example and a processing example of a 3D reconstruction system according to an embodiment; FIG. 2 is a diagram for explaining a specific example 1 of the 3D reconstruction technique according to the reference technique; FIG. 3 is a diagram for explaining a specific example 2 of the 3D reconstruction technique according to the reference technique; FIG. 4 is a diagram for explaining a specific example 3 of the 3D reconstruction technique according to the reference technique; FIG. 5 is a diagram for explaining a specific example 4 of the 3D reconstruction technique according to the reference technique; FIG. 6 is a diagram for explaining a specific example 5 of the 3D reconstruction technique according to the reference technique; FIG. 7 is a block diagram illustrating a configuration example of each device of the 3D reconstruction system according to an embodiment; FIG. 8 is a diagram illustrating an example of an imaging data storage unit of the reconstruction device according to an embodiment; FIG. 9 is a diagram illustrating an example of a 3D model data storage unit of the reconstruction device according to an embodiment; FIG. 10 is a diagram illustrating a specific example 1 of each process of the 3D reconstruction system according to an embodiment; FIG. 11 is a diagram illustrating a specific example 2 of each process of the 3D reconstruction system according to an embodiment; FIG. 12 is a diagram illustrating a specific example 3 of each process of the 3D reconstruction system according to an embodiment; FIG. 13 is a diagram illustrating a specific example 4 of each process of the 3D reconstruction system according to an embodiment; FIG. 14 is a diagram illustrating a specific example 5 of each process of the 3D reconstruction system according to an embodiment; FIG. 15 is a diagram illustrating a specific example 6 of each process of the 3D reconstruction system according to an embodiment; FIG. 16 is a diagram illustrating a specific example 7 of each process of the 3D reconstruction system according to an embodiment; FIG. 17 is a diagram illustrating a specific example 8 of each process of the 3D reconstruction system according to an embodiment; FIG. 18 is a flowchart illustrating a specific example 1 of the overall processing flow of the 3D reconstruction system according to an embodiment; Fig. 10 is a flowchart showing a specific example 2 of the overall processing flow of the 3D reconstruction system according to the embodiment. Fig. 11 is a flowchart showing a specific example 3 of the overall processing flow of the 3D reconstruction system according to the embodiment. Fig. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the reconstruction device according to the embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0010] The present disclosure will be described in the following order: 1. Embodiment 1-1. Configuration and processing of the 3D reconstruction system 100 according to the embodiment 1-2. Configuration and processing of each device of the 3D reconstruction system 100 according to the embodiment 1-3. Specific examples of each process of the 3D reconstruction system 100 according to the embodiment 1-4. Processing flow of the 3D reconstruction system 100 according to the embodiment 2. Effects of the embodiment 3. Hardware configuration

[0011] 1. Embodiment 1-1. Configuration and Processing of 3D Reconstruction System 100 The configuration and processing of a 3D reconstruction system 100, which is an information processing system according to an embodiment, will be described using Fig. 1. Below, an example of the overall configuration of the 3D reconstruction system 100, an example of the overall processing of the 3D reconstruction system 100, and the effects of the 3D reconstruction system 100 as a whole will be described.

[0012] (1-1-1. Example of the Overall Configuration of the 3D Reconstruction System 100) An example of the overall configuration of the 3D reconstruction system 100 will be described using FIG. 1. FIG. 1 is a diagram showing an example of the configuration and processing of the 3D reconstruction system 100 according to an embodiment. The 3D reconstruction system 100 is composed of a reconstruction apparatus 10, a transparent device 20, and a camera 30. Here, the reconstruction apparatus 10, the transparent device 20, and the camera 30 are connected to each other via a predetermined communication network (not shown) so as to be able to communicate with each other by wire or wirelessly. Note that the predetermined communication network can be any of various communication networks such as the Internet or a dedicated line.

[0013] (Reconstruction Device 10) The reconstruction device 10 is an information processing device that generates three-dimensional model data M1 by executing three-dimensional reconstruction processing. Note that the three-dimensional reconstruction system 100 shown in Fig. 1 may include multiple reconstruction devices 10. Also, while the example in Fig. 1 shows a case where the reconstruction device 10 is realized by a desktop PC (Personal Computer), it may also be realized by a notebook PC, a smartphone, a server device, a cloud system, etc.

[0014] (Transparent Device 20) The transparent device 20 is a display device that displays the reference object O1 (marker O1a, chart O1b, color checker O1c). The transparent device 20 is transparent and serves as a support base on which the subject P1 is placed.

[0015] Any display device can be used as long as it is configured to switch between displaying and hiding the reference object O1. In this example, however, a transparent device 20 is used so that the subject P1 can be photographed from as wide a viewpoint as possible. Furthermore, if the position of the subject P1 or the like moves when switching between displaying and hiding the reference object O1, this will affect the control of the estimation of the camera pose, which will be described later. Therefore, it is desirable that the display device be configured so that the user can switch between displaying and hiding the reference object O1 without directly touching it.

[0016] (Camera 30) The camera 30 (30A, 30B, . . . ) is an image capturing device that captures the subject P1 and the reference object O1 and outputs still image data and video data.

[0017] (1-1-2. Example of Processing of the Entire 3D Reconstruction System 100) An example of processing of the entire 3D reconstruction system 100 according to the embodiment will be described. Note that the processing of steps S1 to S6 below may be executed in a different order. Furthermore, some of the processing of steps S1 to S6 below may be omitted.

[0018] First, the camera 30 executes a calibration imaging process (step S1). For example, the camera 30 captures an image of the subject P1 through the transparent device 20 while the reference object O1 is not displayed, and outputs calibration imaging data D3.

[0019] Second, the transparent device 20 executes a reference object display switching process (step S2). For example, the transparent device 20 switches the reference object O1 between a display state and a non-display state.

[0020] Third, the reconstruction device 10 executes a photographing process for generating a 3D model (step S3). For example, the camera 30 photographs the subject P1 placed on the transparent device 20 when the reference object O1 is in a hidden state, and outputs the subject photographing data D1. Also, the camera 30 photographs the reference object O1 displayed on the transparent device 20 when the reference object O1 is in a displayed state, and outputs the reference object photographing data D2.

[0021] Fourth, the reconstruction device 10 executes a photographing data acquisition process (step S4). For example, the reconstruction device 10 acquires the subject photographing data D1, the reference object photographing data D2, and the calibration photographing data D3 as the photographing data output by the camera 30.

[0022] Fifth, the reconstruction device 10 performs 3D reconstruction preprocessing (step S5). For example, the reconstruction device 10 uses the calibration shooting data D3 to perform color and brightness correction of the subject shooting data D1. Also, the reconstruction device 10 uses the reference object shooting data D2 to perform camera pose estimation, real scale acquisition, shooting quality evaluation, etc. of the subject shooting data D1.

[0023] Sixth, the reconstruction device 10 performs 3D reconstruction processing (step S6). For example, the reconstruction device 10 applies photogrammetry, NeRF, 3DGS, or the like to the subject shooting data D1 that has been subjected to 3D reconstruction preprocessing, thereby generating 3D model data M1.

[0024] (1-1-3. Effects of the 3D reconstruction system 100) Using Figures 2 to 6, the basic principles of the 3D reconstruction technology related to the reference technology and the problems with the 3D reconstruction technology related to the reference technology will be explained, and then the effects of the 3D reconstruction system 100 will be explained.

[0025] (Basic Principles of 3D Reconstruction Technology According to Reference Technology) The basic principles of 3D reconstruction technology according to the reference technology will be described with reference to Figs. 2 to 4. Fig. 2 is a diagram for explaining a specific example 1 of the 3D reconstruction technology according to the reference technology. Fig. 3 is a diagram for explaining a specific example 2 of the 3D reconstruction technology according to the reference technology. Fig. 4 is a diagram for explaining a specific example 3 of the 3D reconstruction technology according to the reference technology.

[0026] The 3D reconstruction technology according to the reference technology applies photogrammetry, NeRF, 3DGS, etc., and uses photographic data (see FIG. 2(1)) of a subject photographed from multiple viewpoints to generate a 3D model (see FIG. 2(2)) that reconstructs the subject in 3D.

[0027] In the 3D reconstruction technology of the reference technology, the more viewpoints there are, the higher the quality of the output 3D model is expected to be, so it is necessary to capture images from more viewpoints. Furthermore, in the 3D reconstruction technology of the reference technology, information on the camera pose at the time of capture is essential. Here, the camera pose refers to the capture position and orientation relative to the subject of the image or video used for 3D reconstruction (see the pyramid shape in Figure 3 (2)). In the 3D reconstruction technology of the reference technology, more accurate camera pose estimation in the preprocessing of 3D reconstruction is expected to result in higher quality output 3D models.

[0028] In the 3D reconstruction technology according to the reference technology, markers (see FIG. 4(1)) or charts (see FIG. 4(2)) may be placed and photographed together with the subject for the purposes of use in the above-mentioned camera pose estimation, obtaining a real scale for converting the output 3D model to its actual size, and evaluating the quality of the photography, such as whether the subject is sufficiently in focus. In the 3D reconstruction technology according to the reference technology, it is desirable to be able to place the markers or charts in a wider area, i.e., so that they are widely visible in the angle of view of the captured image or video. In the 3D reconstruction technology according to the reference technology, for example, in camera pose estimation, by widely placing the markers or charts, it becomes possible to use more feature points that serve as clues, thereby enabling more robust and accurate estimation.

[0029] From the above, the 3D reconstruction technology related to the reference technology needs to simultaneously achieve shooting condition 1, which involves photographing the subject from as wide a viewpoint as possible, and shooting condition 2, which involves placing markers and charts as widely as possible around the subject.

[0030] (Problems with 3D reconstruction technology according to the reference technology) Problems with the 3D reconstruction technology according to the reference technology will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram for explaining a specific example 4 of the 3D reconstruction technology according to the reference technology. Fig. 6 is a diagram for explaining a specific example 5 of the 3D reconstruction technology according to the reference technology.

[0031] In the 3D reconstruction technology of the reference technology, the support table on which the subject is placed and the placement of the markers and charts pose problems when simultaneously achieving the above-mentioned imaging conditions 1 and 2. For example, with the 3D reconstruction technology of the reference technology, depending on the setting, the viewpoint from below the subject may be blocked by the markers or charts, so imaging condition 2 can be achieved but imaging condition 1 cannot be achieved (see Figure 5). On the other hand, with the 3D reconstruction technology of the reference technology, if a highly transmissive material is used for the support table, the viewpoint from below the subject can be captured, but it is difficult to recognize the "highly transmissive markers and charts" in the image, so imaging condition 1 can be achieved but imaging condition 2 cannot be achieved. As described above, it is difficult to simultaneously achieve imaging condition 1 and imaging condition 2 with the 3D reconstruction technology of the reference technology.

[0032] In the 3D reconstruction technique according to the reference technology, if the subject is turned upside down during shooting, this adversely affects camera pose estimation. This is because the 3D reconstruction technique according to the reference technology estimates the shooting direction relative to the subject using markers or charts as clues, and changing the relative positions of the subject and the markers or charts causes inconsistency before and after the subject is turned upside down.

[0033] Furthermore, in the 3D reconstruction technology related to the reference technology, when attempting to realize shooting condition 2, there is a problem that markers and charts are frequently reflected in the subject, which leads to a decrease in the quality of the output 3D model (see Figure 6).

[0034] There is also Reference Technology 1, which uses a transparent disc-shaped indicator plate to enable shooting of a subject from various viewpoints. However, Reference Technology 1 relies on the control mechanism of the turntable for camera pose estimation, which has the problem of significant constraints on the rotation mechanism, i.e., the rotation amount of the turntable control must be a strict value.

[0035] There is also Reference Technology 2, which uses transparent plates and three-dimensional markers to acquire camera poses when shooting from various viewpoints. However, Reference Technology 2 has the problem that it is susceptible to the influence of residual distortion from image calibration because it uses locally distributed features in the image to estimate the camera pose. Reference Technology 2 also has the problem that it does not take measures to prevent the markers used for camera pose from being reflected in the subject.

[0036] (Overview of 3D Reconstruction System 100) First, in the 3D reconstruction system 100, the camera 30 captures an image of the subject P1 through the transparent device 20 when the reference object O1 is in a hidden state, and outputs calibration imaging data D3. Second, in the 3D reconstruction system 100, the transparent device 20 switches between a visible state and a hidden state of the reference object O1. Third, in the 3D reconstruction system 100, the camera 30 captures an image of the subject P1 placed on the transparent device 20 when the reference object O1 is in a hidden state, and outputs subject imaging data D1. Furthermore, in the 3D reconstruction system 100, the camera 30 captures an image of the reference object O1 displayed on the transparent device 20 when the reference object O1 is in a visible state, and outputs reference object imaging data D2. Fourth, in the 3D reconstruction system 100, the reconstruction device 10 acquires the subject imaging data D1, the reference object imaging data D2, and the calibration imaging data D3 as imaging data output by the camera 30. Fifth, in the 3D reconstruction system 100, the reconstruction device 10 uses the calibration shooting data D3 to correct changes in color and brightness of the subject shooting data D1. Furthermore, the reconstruction device 10 uses the reference object shooting data D2 to estimate the camera pose of the subject shooting data D1, obtain the real scale, evaluate the shooting quality, etc. Sixth, in the 3D reconstruction system 100, the reconstruction device 10 applies photogrammetry, NeRF, 3DGS, etc. to the subject shooting data D1 that has been subjected to 3D reconstruction preprocessing, and generates 3D model data M1.

[0037] (Effects of the 3D reconstruction system 100) First, the 3D reconstruction system 100 improves the quality of the output 3D model by enabling imaging from a wide range of viewpoints and preprocessing using a wide range of reference objects O1 (markers O1a, charts O1b, color checkers O1c). Second, the 3D reconstruction system 100 can avoid degradation in the quality of the 3D model due to reflection of the reference object O1 in the subject P1. Third, the 3D reconstruction system 100 can suppress degradation in accuracy due to using images or videos captured through the transparent device 20 for 3D reconstruction.

[0038] As described above, the 3D reconstruction system 100 can improve the accuracy of the 3D reconstruction of the object P1.

[0039] 7 to 9, the configuration and processing of each device included in the 3D reconstruction system 100 shown in Fig. 1 will be described. Below, an example configuration of the entire 3D reconstruction system 100 according to the embodiment, an example configuration and processing of the reconstruction device 10, an example configuration and processing of the transparent device 20, and an example configuration and processing of the camera 30 will be described.

[0040] (1-2-1. Example of the Overall Configuration of the 3D Reconstruction System 100) An example of the overall configuration of the 3D reconstruction system 100 will be described using FIG. 7. FIG. 7 is a block diagram showing an example of the configuration of each device of the 3D reconstruction system 100 according to the embodiment. As shown in FIG. 7, the 3D reconstruction system 100 is composed of a reconstruction device 10 which is an information processing device, a transparent device 20 which is a display device, and a plurality of cameras 30 (30A, 30B, ...) which are imaging devices. Furthermore, the reconstruction device 10, the transparent device 20, and the cameras 30 are communicably connected by a communication network N1 which is realized by the Internet, a dedicated line, or the like.

[0041] (1-2-2. Configuration Example and Processing Example of Reconstruction Device 10) A description will be given of a configuration example and processing example of the reconstruction device 10. The reconstruction device 10 has an input unit 11, an output unit 12, a communication unit 13, a storage unit 14, and a control unit 15.

[0042] (Input Unit 11) The input unit 11 controls input of various information to the reconstructing device 10. For example, the input unit 11 is realized by a keyboard, a mouse, etc., and accepts input of various information to the reconstructing device 10.

[0043] (Output Unit 12) The output unit 12 controls the output of various information from the reconstruction device 10. For example, the output unit 12 is realized by a display or the like, and displays various information stored in the reconstruction device 10.

[0044] (Communication Unit 13) The communication unit 13 controls data communication with other devices. For example, the communication unit 13 performs data communication with each communication device via a router, etc. The communication unit 13 can also perform data communication with a terminal (not shown).

[0045] (Storage Unit 14) The storage unit 14 stores various information referenced by the control unit 15 when it operates and various information acquired when the control unit 15 operates. The storage unit 14 is composed of an imaging data storage unit 14a and a 3D model data storage unit 14b. Here, the storage unit 14 can be realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Note that in the example of FIG. 7 , the storage unit 14 is installed inside the reconstruction device 10, but it may also be installed outside the reconstruction device 10, or multiple storage units may be installed.

[0046] (Photography data storage unit 14a) The photography data storage unit 14a stores photography data. For example, the photography data storage unit 14a stores subject photography data D1, reference object photography data D2, and calibration photography data D3 acquired by an acquisition unit 15a of the control unit 15 (described later). An example of data stored in the photography data storage unit 14a will be described below with reference to FIG. 8. FIG. 8 is a diagram showing an example of the photography data storage unit 14a of the reconstruction device 10 according to the embodiment. In the example of FIG. 8, the photography data storage unit 14a has items such as "camera," "subject photography data," "reference object photography data," and "calibration photography data."

[0047] "Camera" indicates identification information for identifying the imaging device, such as the identification number or identification symbol of the camera 30. "Subject imaging data" is first imaging data obtained by imaging a subject P1 placed on a display device in a first state in which the imaging device does not display the reference object O1, such as still image data or video data of the subject P1 captured by the camera 30. "Reference object imaging data" is first imaging data obtained by imaging a reference object O1 displayed on a display device in a second state in which the imaging device displays the reference object O1, such as still image data or video data of the reference object O1 captured by the camera 30. "Calibration imaging data" is third imaging data obtained by imaging the subject P1 through a display device in the first state in which the imaging device does not display the reference object O1, such as still image data or video data of the subject P1 captured by the camera 30 through the transparent device 20.

[0048] That is, Figure 8 shows an example in which the following data is stored in the shooting data storage unit 14a as shooting data captured by the camera 30A identified by "camera A", such as {subject shooting data: "subject shooting data #A-1", reference object shooting data: "reference object shooting data #A-1", calibration shooting data: "calibration shooting data #A-1"}, {subject shooting data: "subject shooting data #A-2", reference object shooting data: "reference object shooting data #A-2", calibration shooting data: "calibration shooting data #A-2"}, {subject shooting data: "subject shooting data #A-3", reference object shooting data: "reference object shooting data #A-3", calibration shooting data: "calibration shooting data #A-3"}, etc.

[0049] (Three-dimensional model data storage unit 14b) The three-dimensional model data storage unit 14b stores three-dimensional model data M1. For example, the three-dimensional model data storage unit 14b stores three-dimensional model data M1 generated by a generation unit 15c of the control unit 15, which will be described later. Here, an example of data stored in the three-dimensional model data storage unit 14b will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of the three-dimensional model data storage unit 14b of the reconstruction device 10 according to the embodiment. In the example of FIG. 9, the three-dimensional model data storage unit 14b has items such as "subject" and "three-dimensional model data."

[0050] Here, the 3D model data M1 generated by the generation unit 15c and stored in the 3D model data storage unit 14b may be copied to another non-transitory storage device, such as another semiconductor memory element such as a flash memory, a hard disk, or an optical disk, via another storage medium or an online communication line. Furthermore, the 3D model data storage unit 14b or another non-transitory storage device may be connected to a display device, such as a head-mounted display, a display, or a large screen, to form an image playback device. By writing the 3D model data generated by the generation unit 15c to the storage device, an image playback device capable of playing back images based on the 3D model data M1 can be manufactured. This makes it possible to provide content, such as a game, to viewers, or to provide the data as background images when producing other video content, such as a movie.

[0051] "Subject" refers to identification information for identifying the subject P1 for which the three-dimensional model data M1 is generated, such as an identification number or identification symbol of the subject P1. "Three-dimensional model data" refers to three-dimensional model data generated by three-dimensional reconstruction processing from photographic data, such as three-dimensional model data M1 generated by photogrammetry, NeRF, 3DGS, etc.

[0052] That is, FIG. 9 shows an example in which data such as {subject: "subject X", three-dimensional model data: "three-dimensional model data X"}, {subject: "subject Y", three-dimensional model data: "three-dimensional model data Y"}, {subject: "subject Z", three-dimensional model data: "three-dimensional model data Z"}, etc. are stored in the three-dimensional model data storage unit 14b.

[0053] (Control Unit 15) The control unit 15 controls the entire reconstructing device 10. The control unit 15 is configured with an acquisition unit 15a, a display unit 15b, and a generation unit 15c. Here, the control unit 15 can be realized by, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0054] (Acquisition Unit 15a) The acquisition unit 15a acquires various types of information. The acquisition unit 15a may store the acquired various types of information in the storage unit 14. The following describes the shooting data acquisition process.

[0055] (Photography Data Acquisition Process) The acquisition unit 15a executes photography data acquisition process. For example, the acquisition unit 15a acquires subject photography data D1, which is photography data output by a camera 30 serving as a photography device, and is first photography data obtained by capturing an image of a subject P1 placed on the transparent device 20 serving as a display device in a first state (non-display state) in which the transparent device 20 does not display the reference object O1. The acquisition unit 15a also acquires a plurality of pieces of subject photography data D1 output by a plurality of cameras 30.

[0056] Specifically, the acquisition unit 15a acquires "subject image data #A-1," "subject image data #A-2," "subject image data #A-3," ... as subject image data D1 captured by the camera 30A identified by "camera A," and stores the acquired data in the image data storage unit 14a. The acquisition unit 15a also acquires "subject image data #B-1," "subject image data #B-2," "subject image data #B-3," ... as subject image data D1 captured by the camera 30B identified by "camera B," and stores the acquired data in the image data storage unit 14a.

[0057] The acquiring unit 15a also acquires reference object shooting data D2, which is second shooting data output by the camera 30 and is obtained by shooting the reference object O1 displayed on the transparent device 20 in a second state (display state) in which the transparent device 20 displays the reference object O1. The acquiring unit 15a also acquires multiple pieces of reference object shooting data D2 output by multiple cameras 30.

[0058] Specifically, the acquisition unit 15a acquires "reference object photography data #A-1," "reference object photography data #A-2," "reference object photography data #A-3," ... as reference object photography data D2 captured by the camera 30A identified by "camera A," and stores these in the photography data storage unit 14a. The acquisition unit 15a also acquires "reference object photography data #B-1," "reference object photography data #B-2," "reference object photography data #B-3," ... as reference object photography data D2 captured by the camera 30B identified by "camera B," and stores these in the photography data storage unit 14a.

[0059] The acquiring unit 15a also acquires calibration shooting data D3, which is third shooting data output by the camera 30 and is obtained by capturing an image of the subject P1 through the transparent device 20 in a first state (non-display state) in which the transparent device 20 does not display the reference object O1. The acquiring unit 15a also acquires multiple pieces of calibration shooting data D3 output by multiple cameras 30.

[0060] Specifically, the acquisition unit 15a acquires "calibration shooting data #A-1," "calibration shooting data #A-2," "calibration shooting data #A-3," ... as the calibration shooting data D3 captured by the camera 30A identified by "camera A," and stores the data in the shooting data storage unit 14a. The acquisition unit 15a also acquires "calibration shooting data #B-1," "calibration shooting data #B-2," "calibration shooting data #B-3," ... as the calibration shooting data D3 captured by the camera 30B identified by "camera B," and stores the data in the shooting data storage unit 14a.

[0061] (Display Unit 15b) The display unit 15b executes display control. The display unit 15b can also refer to various information stored in the storage unit 14. The display switching control process will be described below.

[0062] (Display Switching Control Process) The display unit 15b executes a display switching control process. For example, the display unit 15b executes display control for switching the transparent device 20 between a first state (non-display state) and a second state (display state). At this time, the display unit 15b executes display control for switching the transparent device 20 between the first state and the second state, and can also execute shooting control for the camera 30 to capture images in the first state and the second state. The display unit 15b can also execute display control for switching the transparent device 20 between the first state and the second state in accordance with the timing at which the camera 30 captures images at regular intervals. The display unit 15b can also execute display control for switching the transparent device 20 between the first state and the second state based on a user's input operation or a preset operating procedure.

[0063] Specifically, when the camera 30A identified by "camera A" captures the subject shooting data D1 or the calibration shooting data D3, the display unit 15b executes a non-display instruction to the transparent device 20 and switches to a first state in which the reference object O1 is not displayed. Also, when the camera 30A identified by "camera A" captures the reference object shooting data D2, the display unit 15b executes a display instruction to the transparent device 20 and switches to a second state in which the reference object O1 is displayed.

[0064] The display switching control for the transparent device 20 and the shooting control for the camera 30 in the first state and the second state may be configured so that the user can control the display switching and shooting operations one by one via an input interface such as a switch, touch panel, or mouse provided on the display unit 15b, etc. Alternatively, the display unit 15b, etc. may be configured to automatically control according to a preset procedure. Furthermore, in the case of automatic control, the user may be configured to be able to freely change the settings of the control procedure.

[0065] (Generation Unit 15c) The generation unit 15c generates various types of information. The generation unit 15c stores the generated various types of information in the storage unit 14. The 3D reconstruction pre-processing and the 3D model data generation process will be described below.

[0066] (3D Reconstruction Pre-Processing) The generating unit 15c executes 3D reconstruction pre-processing. For example, the generating unit 15c corrects changes in at least one of the color and brightness of the subject capture data D1 using the calibration capture data D3.

[0067] Specifically, first, the generation unit 15c refers to the "calibration shooting data #A-1" stored in the shooting data storage unit 14a and captured by the camera 30A identified by "camera A." Second, the generation unit 15c refers to the "subject shooting data #A-1" stored in the shooting data storage unit 14a and captured by the camera 30A identified by "camera A." Third, the generation unit 15c uses the "calibration shooting data #A-1" to correct changes in color and brightness of the "subject shooting data #A-1" and generate corrected "subject shooting data #A-1'." At this time, the generation unit 15c can correct changes in color and brightness of the "subject shooting data #A-1" captured through a transparent device 20 whose transmittance is not 1. Furthermore, the generation unit 15c may use the "calibration shooting data #A-1" to perform corrections to estimate the focal length of the "subject shooting data #A-1" and remove distortion.

[0068] Furthermore, the generation unit 15c uses the reference object shooting data D2 to perform at least one of camera pose estimation, real scale acquisition, and shooting quality evaluation of the subject shooting data D1.

[0069] Specifically, first, the generation unit 15c refers to the "reference object shooting data #A-1" stored in the shooting data storage unit 14a and captured by the camera 30A identified by "camera A." Second, the generation unit 15c identifies the reference object O1 in the "reference object shooting data #A-1" and outputs a "camera pose estimation result #A-1" indicating the shooting position and posture relative to the subject P1. Third, the generation unit 15c identifies the reference object O1 in the "reference object shooting data #A-1" and outputs a "real scale acquisition result #A-1" for converting the 3D model data M1 to an actual size. Fourth, the generation unit 15c identifies the reference object O1 in the "reference object shooting data #A-1" and outputs a "shooting quality evaluation result #A-1" indicating whether the subject is sufficiently in focus, etc. At this time, the generation unit 15c may use auxiliary acquired shooting data for camera pose estimation, real scale acquisition, shooting quality evaluation, etc.

[0070] (3D Model Data Generation Process) The generation unit 15c executes the 3D model data generation process. For example, the generation unit 15c generates 3D model data M1 of the subject P1 based on the plurality of subject shooting data D1 and the plurality of reference object shooting data D2. At this time, the generation unit 15c executes photogrammetry, NeRF, or 3DGS to generate the 3D model data M1.

[0071] Specifically, the generation unit 15c performs a three-dimensional reconstruction process using photogrammetry, NeRF, or 3DGS using the corrected "subject shooting data #A-1'," "camera pose estimation result #A-1," "actual scale acquisition result #A-1," and "shooting quality evaluation result #A-1," thereby generating "three-dimensional model data X," which is three-dimensional model data M1 of the subject P1 identified by "subject X," and stores it in the three-dimensional model data storage unit 14b.

[0072] Furthermore, the generation unit 15c can also manufacture a non-transitory storage medium by storing the generated three-dimensional model data M1 in the non-transitory storage medium.The generation unit 15c can also manufacture an image reproduction device (not shown) configured to reproduce the three-dimensional model data M1 by storing the generated three-dimensional model data M1 in a non-transitory storage medium connected to a display device (not shown).

[0073] (1-2-3. Configuration Examples and Processing Examples of the Transparent Device 20) Referring again to FIG. 7 , a configuration example and processing example of the transparent device 20 will be described. For example, the transparent device 20 is a display device having transparency, is formed of a material with high transmittance, and switches between a first state in which the reference object O1 is not displayed, and a second state in which the reference object O1 is displayed. The transparent device 20 is transparent at least in the first state in which the reference object O1 is not displayed, and is a device in which the display and non-display of the reference object O1 are switched by controlling electro-optical elements such as liquid crystal or organic electroluminescent (EL) that constitute the transparent device 20, or the display and non-display of the reference object O1 are switched by the presence or absence of light projected onto the transparent device 20.

[0074] Furthermore, the transparent device 20 may be a transparent display 20A having transparency, and may be configured with a well-known transparent liquid crystal display or organic EL display. If the transparent device 20 is configured with a liquid crystal display, the transmissive state can be switched by controlling the voltage applied to each liquid crystal element, thereby switching whether or not the reference object O1 is displayed. If the transparent device 20 is configured with an organic EL display, the display whether or not the reference object O1 is displayed can be switched by switching the light-emitting or non-light-emitting state of each organic EL element. Furthermore, the transparent device 20 may be a transparent screen 20B having transparency, and the reference object O1 projected from a projector 21B is displayed on the transparent screen 20B.

[0075] Specifically, when the transparent display 20A receives a non-display instruction from the reconstructing device 10, the transparent display 20A switches to a first state in which the reference object O1 is not displayed by setting the transparent display 20A to a non-display state. Furthermore, when the transparent display 20A receives a display instruction from the reconstructing device 10, the transparent display 20A switches to a second state in which the reference object O1 is displayed by setting the transparent display 20A to a display state.

[0076] Furthermore, when the transparent screen 20B receives a turn-off instruction from the reconstruction device 10, it switches to a first state in which the reference object O1 is not displayed by turning off the projector 21B that projects the reference object O1 onto the transparent screen 20B. Furthermore, when the transparent screen 20B receives a turn-on instruction from the reconstruction device 10, it switches to a second state in which the reference object O1 is displayed by turning on the projector 21B that projects the reference object O1 onto the transparent screen 20B.

[0077] The transparent device 20 displays a marker O1a representing a predetermined shape as the reference object O1. The transparent device 20 also displays a chart O1b representing a predetermined graphic pattern as the reference object O1. The transparent device 20 also displays a color checker O1c representing a predetermined color as the reference object O1.

[0078] Specifically, the transparent device 20 displays, for example, a marker O1a showing an asymmetrical white shape on a black background. The transparent device 20 also displays a chart O1b showing, for example, a chessboard-shaped pattern made up of black and white squares, or a pattern made up of black polka dots on a white background. The transparent device 20 also displays a color checker O1c made up of a plurality of colors, such as red, yellow, and blue. In this case, the transparent device 20 can display a reference object O1 that combines two or more of the marker O1a, the chart O1b, and the color checker O1c.

[0079] (1-2-4. Configuration and Processing Examples of Camera 30) Referring again to FIG. 7, a configuration and processing example of camera 30 will be described. For example, camera 30 may be realized as a photographing device that outputs still image data. Also, camera 30 may be realized as a photographing device that outputs video data.

[0080] The camera 30 outputs subject shooting data D1 obtained by shooting a subject P1 placed on the transparent device 20 in a first state in which the transparent device 20 does not display the reference object O1. The camera 30 also outputs reference object shooting data D2 obtained by shooting a reference object O1 displayed on the transparent device 20 in a second state in which the transparent device 20 displays the reference object O1. The camera 30 also outputs calibration shooting data D3 obtained by shooting a subject P1 through the transparent device 20 in the first state in which the transparent device 20 does not display the reference object O1. The camera 30 also outputs subject shooting data D1 or reference object shooting data D2 corresponding to each timing when the transparent device 20 is in the first state and the second state.

[0081] Specifically, the camera 30A identified by "camera A" outputs "subject shooting data #A-1," "subject shooting data #A-2," "subject shooting data #A-3," etc. as the subject shooting data D1. The camera 30A identified by "camera A" outputs "reference object shooting data #A-1," "reference object shooting data #A-2," "reference object shooting data #A-3," etc. as the reference object shooting data D2. The camera 30A identified by "camera A" outputs "calibration shooting data #A-1," "calibration shooting data #A-2," "calibration shooting data #A-3," etc. as the calibration shooting data D3.

[0082] Furthermore, the camera 30B identified by "camera B" outputs "subject shooting data #B-1," "subject shooting data #B-2," "subject shooting data #B-3," etc. as the subject shooting data D1. The camera 30B identified by "camera B" outputs "reference object shooting data #B-1," "reference object shooting data #B-2," "reference object shooting data #B-3," etc. as the reference object shooting data D2. The camera 30B identified by "camera B" outputs "calibration shooting data #B-1," "calibration shooting data #B-2," "calibration shooting data #B-3," etc. as the calibration shooting data D3.

[0083] The camera 30 is fixedly installed at a predetermined position around the transparent device 20, and by photographing the rotating transparent device 20 at predetermined intervals, outputs at least one of subject photographing data D1 from multiple viewpoints, reference object photographing data D2, and calibration photographing data D3.

[0084] Specifically, a camera 30A, which is installed above the transparent device 20 and identified as "camera A," photographs the rotating transparent device 20 every second, thereby outputting photographic data from multiple viewpoints. A camera 30B, which is installed directly beside the transparent device 20 and identified as "camera B," photographs the rotating transparent device 20 every second, thereby outputting photographic data from multiple viewpoints. A camera 30C, which is installed below the transparent device 20 and identified as "camera C," photographs the rotating transparent device 20 every second, thereby outputting photographic data from multiple viewpoints.

[0085] The camera 30 moves around the transparent device 20, photographing the fixed transparent device 20 at predetermined moving distances, and outputs at least one of subject photographing data D1 from multiple viewpoints, reference object photographing data D2, and calibration photographing data D3.

[0086] Specifically, the camera 30A, which is installed above the transparent device 20 and is identified as "camera A," outputs imaging data from multiple viewpoints by capturing an image every time the fixed transparent device 20 rotates 30 degrees. The camera 30B, which is installed directly beside the transparent device 20 and is identified as "camera B," outputs imaging data from multiple viewpoints by capturing an image every time the fixed transparent device 20 rotates 30 degrees. The camera 30C, which is installed below the transparent device 20 and is identified as "camera C," outputs imaging data from multiple viewpoints by capturing an image every time the fixed transparent device 20 rotates 30 degrees.

[0087] The cameras 30 are fixedly installed at predetermined intervals around the transparent device 20, and by photographing the fixed transparent device 20, output at least one of subject photographing data D1 from multiple viewpoints, reference object photographing data D2, and calibration photographing data D3.

[0088] Specifically, cameras 30A (20A-1, 20A-2, ..., 20A-12) installed above the transparent device 20 and identified by "camera A-1," "camera A-2," ..., "camera A-12" are installed at 30° intervals around the fixed transparent device 20 and capture images simultaneously to output image capture data from multiple viewpoints. Also, cameras 30B (20B-1, 20B-2, ..., 20B-12) installed directly to the side of the transparent device 20 and identified by "camera B-1," "camera B-2," ..., "camera B-12" are installed at 30° intervals around the fixed transparent device 20 and capture images simultaneously to output image capture data from multiple viewpoints. In addition, cameras 30C (20C-1, 20C-2, ..., 20C-12) installed below the transparent device 20 and identified by "camera C-1," "camera C-2," ..., "camera C-12" are installed every 30° around the fixed transparent device 20 and output shooting data from multiple viewpoints by simultaneously shooting.

[0089] 10 to 17, specific examples of the processes of the 3D reconstruction system 100 according to the embodiment will be described. Specific examples 1 to 8 of the processes of the 3D reconstruction system 100 according to the embodiment will be described below.

[0090] (1-3-1. Specific Example 1) Specific example 1 of each process of the 3D reconstruction system 100 according to the embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing specific example 1 of each process of the 3D reconstruction system 100 according to the embodiment. Below, as specific example 1, multi-viewpoint shooting using a fixed camera 30 and a rotatable transparent display 20A will be described.

[0091] As shown in FIG. 10, the first specific example is composed of a rotatable transparent display 20A, fixed cameras 30 (30A, 30B, 30C), and light sources L1 and L2.

[0092] The subject P1 is placed on the transparent display 20A. The transparent display 20A is turntable-shaped and rotates at a constant speed. The transparent display 20A switches between displaying and hiding a reference object O1 on the transparent display 20A.

[0093] The camera 30 is fixedly installed around the transparent display 20A, captures images of the transparent display 20A rotating at a constant speed, and outputs captured image data of the subject P1 and the reference object O1. In the example of FIG. 10 , multiple cameras 30 are used, but a single camera 30 may be used. The camera 30 may be fixed to the transparent display 20A to synchronize with the rotation of the transparent display 20A, or may be freely moved by a robot arm or the like. Here, when the camera 30 is synchronized with the rotation, it is necessary to capture images of the subject P1 from multiple viewpoints. However, since the camera 30 captures only the same surface of the subject P1, it cannot be used for capturing images from multiple viewpoints. However, when capturing images in a uniform lighting environment, it can acquire information that can be used for estimating the light source direction and for correction in image processing.

[0094] The light sources L1 and L2 emit light onto the subject P1 and the reference object O1 during image capture. In the example of FIG. 10 , multiple light sources L1 and L2 are used, but a single light source may be used. The light sources L1 and L2 may adjust the amount of light they emit depending on the brightness of the transparent display 20A. The light sources L1 and L2 may be fixed during image capture, or may be fixed to the transparent display 20A and synchronized with the rotation of the transparent display 20A, or may be freely movable by a robot arm or the like. When the light sources L1 and L2 are synchronized with the rotation or freely movable, a three-dimensional representation different from that obtained when the light sources L1 and L2 are fixed can be obtained in NeRF or 3DGS, which can also reproduce the lighting environment at the time of image capture.

[0095] (1-3-2. Specific Example 2) Specific example 2 of each process of the 3D reconstruction system 100 according to the embodiment will be described with reference to Fig. 11. Fig. 11 is a diagram showing specific example 2 of each process of the 3D reconstruction system 100 according to the embodiment. As specific example 2, multi-viewpoint shooting using a fixed camera 30 and a rotatable transparent screen 20B will be described below.

[0096] As shown in FIG. 11, the second specific example is composed of a rotatable transparent screen 20B, a projector 21B, fixed cameras 30 (30A, 30B, 30C), and light sources L1 and L2.

[0097] The subject P1 is placed on the transparent screen 20B. The projector 21B is also placed on the transparent screen 20B. The transparent screen 20B is turntable-shaped and rotates at a constant speed. The transparent screen 20B switches between displaying and hiding the reference object O1 projected by the projector 21B on the transparent screen 20B.

[0098] The cameras 30 are fixedly installed around the transparent screen 20B, capture images of the transparent screen 20B rotating at a constant speed, and output captured image data of the subject P1 and the reference object O1. In the example of FIG. 11 , multiple cameras 30 are used, but a single camera 30 may be used. The camera 30 may be fixed to the transparent screen 20B to synchronize with the rotation of the transparent screen 20B, or may be freely moved by a robot arm or the like. While the camera 30 is rotationally synchronized, it is necessary to capture images of the subject P1 from multiple viewpoints. However, since the camera 30 captures only the same surface of the subject P1, it cannot be used for capturing images from multiple viewpoints. However, when capturing images in a uniform lighting environment, it can acquire information that can be used for estimating the light source direction and for image processing correction.

[0099] Light sources L1 and L2 emit light onto the subject P1 and the reference object O1 during image capture. In the example of FIG. 11 , multiple light sources L1 and L2 are used, but a single light source may be used. Furthermore, the light sources L1 and L2 may adjust the amount of light they emit depending on the brightness of the transparent screen 20B. Furthermore, light sources L1 and L2 may be fixed during image capture, or may be fixed to the transparent screen 20B and synchronized with the rotation of the transparent screen 20B, or may be freely movable by a robot arm or the like. When light sources L1 and L2 are synchronized with the rotation or freely movable, a three-dimensional representation different from that obtained when the light sources are fixed can be obtained in NeRF or 3DGS, which can also reproduce the lighting environment at the time of image capture.

[0100] (1-3-3. Specific Example 3) Specific example 3 of each process of the 3D reconstruction system 100 according to the embodiment will be described with reference to Fig. 12. Fig. 12 is a diagram showing specific example 3 of each process of the 3D reconstruction system 100 according to the embodiment. Below, as specific example 3, multi-viewpoint shooting using a movable camera 30 and a fixed transparent display 20A will be described.

[0101] As shown in FIG. 12, the third specific example is configured with a fixed transparent display 20A, movable cameras 30 (30A, 30B, 30C), and light sources L1 and L2.

[0102] The subject P1 is placed on the transparent display 20A. The transparent display 20A is fixed. The transparent display 20A switches between displaying and hiding a reference object O1 on the transparent display 20A.

[0103] The camera 30 photographs the fixed transparent display 20A while freely moving using a robot arm, rails, or the like, and outputs photographic data of the subject P1 and the reference object O1. In the example of FIG. 12 , multiple cameras 30 are shown, but a single camera may also be used. The camera 30 may be fixed during photographing. Here, when the camera 30 is fixed, it is necessary to photograph the subject P1 from multiple viewpoints, but the camera 30 photographs only the same surface of the subject P1, so it cannot be used for photographing from multiple viewpoints. However, when photographing in a uniform lighting environment, it can acquire information that can be used for estimating the light source direction and for correction in image processing.

[0104] The light sources L1 and L2 irradiate the subject P1 and the reference object O1 with light during image capture. In the example of FIG. 12 , multiple light sources L1 and L2 are shown, but a single light source may be used. The light sources L1 and L2 may adjust the amount of light they irradiate depending on the brightness of the transparent display 20A. The light sources L1 and L2 may be fixed during image capture, or may be freely moved by a robot arm or the like. When the light sources L1 and L2 are freely movable, a three-dimensional representation different from that obtained when the light sources L1 and L2 are fixed can be obtained in NeRF or 3DGS, which can also reproduce the lighting environment at the time of image capture.

[0105] (1-3-4. Specific Example 4) Specific example 4 of each process of the 3D reconstruction system 100 according to the embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing specific example 4 of each process of the 3D reconstruction system 100 according to the embodiment. Below, as specific example 4, multi-viewpoint shooting using a movable camera 30 and a fixed transparent screen 20B will be described.

[0106] As shown in FIG. 13, the fourth specific example is composed of a fixed transparent screen 20B, a projector 21B, movable cameras 30 (30A, 30B, 30C), and light sources L1 and L2.

[0107] The subject P1 is placed on the transparent screen 20B. The projector 21B is also placed on the transparent screen 20B. The transparent screen 20B is fixed. The transparent screen 20B switches between displaying and hiding the reference object O1 projected by the projector 21B on the transparent screen 20B.

[0108] The camera 30 photographs the fixed transparent screen 20B while moving freely using a robot arm, rails, or the like, and outputs photographic data of the subject P1 and the reference object O1. In the example of FIG. 13 , multiple cameras 30 are shown, but a single camera may also be used. The camera 30 may also be fixed during photographing. Here, when the camera 30 is fixed, it is necessary to photograph the subject P1 from multiple viewpoints, but the camera 30 photographs only the same surface of the subject P1, so it cannot be used for photographing from multiple viewpoints. However, when photographing in a uniform lighting environment, it can acquire information that can be used for estimating the light source direction and for correction in image processing.

[0109] Light sources L1 and L2 irradiate light onto subject P1 and reference object O1 during image capture. In the example of FIG. 13 , multiple light sources L1 and L2 are used, but a single light source may be used. Furthermore, light sources L1 and L2 may adjust the amount of light they emit depending on the brightness of transparent screen 20B. Furthermore, light sources L1 and L2 may be fixed during image capture, or may be freely moved by a robot arm or the like. When light sources L1 and L2 are freely movable, a three-dimensional representation different from that obtained when light sources L1 and L2 are fixed can be obtained in NeRF or 3DGS, which can also reproduce the lighting environment at the time of image capture.

[0110] (1-3-5. Specific Example 5) Specific example 5 of each process of the 3D reconstruction system 100 according to the embodiment will be described using Fig. 14. Fig. 14 is a diagram showing specific example 5 of each process of the 3D reconstruction system 100 according to the embodiment. Below, as specific example 5, multi-viewpoint shooting using cameras 30 installed for the required number of viewpoints and fixed transparent displays 20A will be described.

[0111] As shown in FIG. 14, the fifth specific example is configured with a fixed transparent display 20A, cameras 30 (30A, 30B, 30C, 30D, 30E, 30F) installed for the required number of viewpoints, and light sources L1 and L2.

[0112] The subject P1 is placed on the transparent display 20A. The transparent display 20A is fixed. The transparent display 20A switches between displaying and hiding a reference object O1 on the transparent display 20A.

[0113] The cameras 30 are fixedly installed at the required number of viewpoints, capture images of the fixed transparent displays 20A, and output captured image data of the subject P1 and the reference object O1. The cameras 30 can output not only still image data but also video data.

[0114] The light sources L1 and L2 irradiate the subject P1 and the reference object O1 with light during image capture. In the example of FIG. 14 , multiple light sources L1 and L2 are used, but a single light source may be used. The light sources L1 and L2 may adjust the amount of light they emit depending on the brightness of the transparent display 20A. The light sources L1 and L2 may be fixed during image capture, or may be freely moved by a robot arm or the like. When the light sources L1 and L2 are freely movable, a three-dimensional representation different from that obtained when the light sources L1 and L2 are fixed can be obtained in NeRF or 3DGS, which can also reproduce the lighting environment at the time of image capture.

[0115] (1-3-6. Specific Example 6) Specific Example 6 of each process of the 3D reconstruction system 100 according to the embodiment will be described using Fig. 15. Fig. 15 is a diagram showing Specific Example 6 of each process of the 3D reconstruction system 100 according to the embodiment. Below, as Specific Example 6, multi-viewpoint shooting using cameras 30 installed for the required number of viewpoints and fixed transparent screens 20B will be described.

[0116] As shown in FIG. 15, the sixth specific example is configured with a fixed transparent screen 20B, a projector 21B, cameras 30 (30A, 30B, 30C, 30D, 30E, 30F) installed for the required number of viewpoints, and light sources L1 and L2.

[0117] The subject P1 is placed on the transparent screen 20B. The projector 21B is also placed on the transparent screen 20B. The transparent screen 20B is fixed. The transparent screen 20B switches between displaying and hiding the reference object O1 projected by the projector 21B on the transparent screen 20B.

[0118] The cameras 30 are fixedly installed at the required number of viewpoints, capture images of the fixed transparent screen 20B, and output captured image data of the subject P1 and the reference object O1. The cameras 30 can output not only still image data but also video data.

[0119] Light sources L1 and L2 irradiate light onto subject P1 and reference object O1 during photography. In the example of FIG. 15 , multiple light sources L1 and L2 are used, but a single light source may be used. Furthermore, light sources L1 and L2 may adjust the amount of light they emit depending on the brightness of transparent screen 20B. Furthermore, light sources L1 and L2 may be fixed during photography, or may be freely moved by a robot arm or the like. When light sources L1 and L2 are freely movable, a three-dimensional representation different from that obtained when they are fixed can be obtained in NeRF or 3DGS, which can also reproduce the lighting environment at the time of photography.

[0120] (1-3-7. Specific Example 7) Specific example 7 of each process of the 3D reconstruction system 100 according to the embodiment will be described with reference to Fig. 16. Fig. 16 is a diagram showing specific example 7 of each process of the 3D reconstruction system 100 according to the embodiment. Below, as specific example 7, a specific example of the reference object O1 displayed by the transparent device 20 will be described.

[0121] In the example of Figure 16 (1), the transparent device 20 displays a chart O1b showing a chessboard-like pattern (checker pattern) consisting of black squares and white squares as a reference object O1, and displays markers O1a showing asymmetrical shapes in white on a black background inside all the white squares.

[0122] In the example of Figure 16 (2), the transparent device 20 displays a chart O1b showing a chessboard-like pattern consisting of black squares and white squares as a reference object O1, and also displays a marker O1a showing an asymmetrical white shape on a black background inside some of the white squares.

[0123] In the example of Figure 16 (3), the transparent device 20 displays a chart O1b showing a chessboard-like pattern consisting of black squares and white squares as a reference object O1, and displays a marker O1a showing an asymmetrical white shape on a black background inside some of the white squares, and further replaces some of the white squares in the periphery with gray squares.

[0124] In the example of Figure 16 (4), the transparent device 20 displays a chart O1b showing a pattern (marble pattern) consisting of black polka dots on a white background as the reference object O1, and replaces the polka dots in the center with a marker O1a showing an asymmetrical white shape on a black background.

[0125] 16(1) to 16(4), the reference object O1 is a combination of a marker O1a and a chart O1b, but is not limited thereto. For example, the reference object O1 may be a single marker O1a, a single chart O1b, or a single color checker O1c, or may be a combination of two or more of these.

[0126] (1-3-8. Specific Example 8) Specific example 8 of each process of the 3D reconstruction system 100 according to the embodiment will be described with reference to Fig. 17. Fig. 17 is a diagram showing specific example 8 of each process of the 3D reconstruction system 100 according to the embodiment. Below, as specific example 8, a shooting data classification process executed after all viewpoint shooting will be described.

[0127] The reconstruction device 10 performs a captured image classification process after capturing images from all viewpoints, i.e., after acquiring the captured image data. For example, the reconstruction device 10 classifies the captured image data captured from below through the transparent device 20 and the captured image data captured directly from above as Class 1. At this time, the reconstruction device 10 may distinguish in advance the camera 30 for capturing images from below as Class 1, and may assign metadata for identification to the captured image data captured by the camera 30, or may determine whether the marker O1a is inverted.

[0128] At this time, the reconstruction device 10 can distinguish between the reference object O1 photographed from above the transparent device 20 (see Figure 17(1)) and the reference object O1 photographed from below the transparent device 20 (see Figure 17(2)), thereby classifying the photographed data even if the subject P1 is turned upside down during the photographing process.

[0129] The reconstruction device 10 also classifies the reference object shooting data D2 and the subject shooting data D1 into category 2. At this time, the reconstruction device 10 may classify the data into odd and even numbers using the shooting order, may add metadata for identification to the shooting data, or may classify the data based on whether the reference object O1 can be recognized.

[0130] By classifying the imaging data as described above, the reconstruction device 10 can improve the efficiency of subsequent preprocessing and 3D reconstruction processing. For example, by classifying the imaging data according to Category 1, the reconstruction device 10 can identify imaging data captured through the transparent device 20, i.e., calibration imaging data D3, and efficiently correct changes in color and brightness (luminance) only for the identified imaging data. Furthermore, by classifying the imaging data according to Category 2, the reconstruction device 10 can identify reference object imaging data D2 and subject imaging data D1, and efficiently perform camera pose estimation, real scale acquisition, imaging quality evaluation, etc.

[0131] 18 to 20, the processing flow of the 3D reconstruction system 100 according to the embodiment will be described. Specific examples 1 to 3 of the overall processing flow of the 3D reconstruction system 100 will be described below.

[0132] (1-4-1. Specific Example 1 of the Overall Processing Flow of the 3D Reconstruction System 100) Specific Example 1 of the overall processing flow of the 3D reconstruction system 100 according to the embodiment will be described using FIG. 18. FIG. 18 is a flowchart showing Specific Example 1 of the overall processing flow of the 3D reconstruction system 100 according to the embodiment. Below, as Specific Example 1, the overall processing flow of the 3D reconstruction system 100 executed in still image capture mode will be described. Note that the processes of steps S101 to S111 below can also be executed in a different order. Also, some of the processes of steps S101 to S111 below may be omitted.

[0133] First, the camera 30 executes a calibration imaging process (step S101). For example, the camera 30 receives an imaging instruction from the reconstruction device 10, captures an image of the subject P1 through the transparent device 20 while the reference object O1 is hidden, and outputs calibration imaging data D3, which is still image data. The reconstruction device 10 also acquires the calibration imaging data D3 from the camera 30.

[0134] Second, the transparent device 20 executes a reference object display process (step S102). For example, the transparent device 20 receives a display instruction from the reconstructing device 10 and sets the reference object O1 on the transparent device 20 to a display state.

[0135] Third, the camera 30 executes a reference object photographing process (step S103). For example, the camera 30 receives a photographing instruction from the reconstruction device 10, photographs the reference object O1 displayed on the transparent device 20, and outputs reference object photographing data D2, which is still image data. The reconstruction device 10 also acquires the reference object photographing data D2 from the camera 30.

[0136] Fourth, the transparent device 20 executes a reference object non-display process (step S104). For example, the transparent device 20 receives a non-display instruction from the reconstruction device 10 and makes the reference object O1 on the transparent device 20 non-display.

[0137] Fifth, the camera 30 executes a reference object photographing process (step S105). For example, the camera 30 receives a photographing instruction from the reconstruction device 10, photographs the subject P1 placed on the transparent device 20, and outputs subject photographing data D1, which is still image data. The reconstruction device 10 also acquires the subject photographing data D1 from the camera 30.

[0138] At this time, if the imaging of all viewpoints is completed (step S106: Yes), the reconstruction device 10 proceeds to the processing of step S108. On the other hand, if the imaging of all viewpoints is not completed (step S106: No), the reconstruction device 10 proceeds to the processing of step S107.

[0139] Sixth, the camera 30 executes a shooting viewpoint movement process (step S107). For example, the camera 30 receives a movement instruction from the reconstruction device 10, moves around the transparent device 20 using a robot arm, rails, or the like, and returns to the process of step S102.

[0140] Seventh, the reconstruction device 10 executes a photographic data classification process (step S108). For example, the reconstruction device 10 classifies the acquired subject photographic data D1, reference object photographic data D2, and calibration photographic data D3.

[0141] Eighth, the reconstruction device 10 executes a first pre-processing (step S109). For example, the reconstruction device 10 executes correction of changes in color and brightness of the subject shooting data D1 using the calibration shooting data D3.

[0142] Ninth, the reconstruction device 10 performs second pre-processing (step S110). For example, the reconstruction device 10 uses the reference object shooting data D2 to perform camera pose estimation, real scale acquisition, shooting quality evaluation, etc. of the subject shooting data D1.

[0143] Tenth, the reconstruction device 10 executes a 3D reconstruction process (step S111) and ends the process. For example, the reconstruction device 10 applies photogrammetry, NeRF, 3DGS, or the like to the subject shooting data D1 that has been subjected to the first preprocessing and the second preprocessing, thereby generating 3D model data M1.

[0144] (1-4-2. Specific Example 2 of the Overall Processing Flow of the 3D Reconstruction System 100) Specific Example 2 of the overall processing flow of the 3D reconstruction system 100 according to the embodiment will be described using FIG. 19. FIG. 19 is a flowchart showing Specific Example 2 of the overall processing flow of the 3D reconstruction system 100 according to the embodiment. Below, as Specific Example 2, the overall processing flow of the 3D reconstruction system 100 executed in video capture mode will be described. Note that the processes of steps S201 to S210 below can also be executed in a different order. Furthermore, some of the processes of steps S201 to S210 below may be omitted.

[0145] First, the camera 30 executes a calibration imaging process (step S201). For example, the camera 30 receives an imaging instruction from the reconstruction device 10, captures an image of the subject P1 through the transparent device 20 while the reference object O1 is hidden, and outputs calibration imaging data D3, which is video data. The reconstruction device 10 also acquires the calibration imaging data D3 from the camera 30.

[0146] Second, the camera 30 executes a moving image capturing start process (step S202). For example, the camera 30 receives a capturing start instruction from the reconstruction device 10 and starts capturing in a moving image capturing mode.

[0147] Third, the transparent device 20 executes a reference object display process (step S203). For example, the transparent device 20 receives a display instruction from the reconstructing device 10 and sets the reference object O1 on the transparent device 20 to a display state.

[0148] Fourth, the transparent device 20 executes a reference object non-display process (step S204). For example, the transparent device 20 receives a non-display instruction from the reconstruction device 10 and puts the reference object O1 on the transparent device 20 into a non-display state.

[0149] At this time, if the imaging of all viewpoints is completed (step S205: Yes), the reconstruction device 10 proceeds to the processing of step S207. On the other hand, if the imaging of all viewpoints is not completed (step S205: No), the reconstruction device 10 proceeds to the processing of step S206.

[0150] Fifth, the camera 30 executes a shooting viewpoint movement process (step S206). For example, the camera 30 receives a movement instruction from the reconstruction device 10, moves around the transparent device 20 using a robot arm, rails, or the like, and returns to the process of step S203.

[0151] Sixth, the camera 30 executes a video capture end process (step S207). For example, the camera 30 receives a capture end instruction from the reconstruction device 10 and ends capture in video capture mode. The reconstruction device 10 also acquires subject capture data D1, which is video data, from the camera 30. The reconstruction device 10 also acquires reference object capture data D2, which is video data, from the camera 30.

[0152] Seventh, the reconstruction device 10 performs a first pre-processing (step S208). For example, the reconstruction device 10 corrects changes in color and brightness of the subject shooting data D1 using the calibration shooting data D3.

[0153] Eighth, the reconstruction device 10 performs second pre-processing (step S209). For example, the reconstruction device 10 uses the reference object shooting data D2 to perform camera pose estimation, real scale acquisition, shooting quality evaluation, etc. of the subject shooting data D1.

[0154] Ninth, the reconstruction device 10 executes a 3D reconstruction process (step S210) and ends the process. For example, the reconstruction device 10 applies photogrammetry, NeRF, 3DGS, or the like to the subject shooting data D1 that has been subjected to the first preprocessing and the second preprocessing, thereby generating 3D model data M1.

[0155] (1-4-3. Specific Example 3 of the Overall Processing Flow of the 3D Reconstruction System 100) Specific Example 3 of the overall processing flow of the 3D reconstruction system 100 according to the embodiment will be described using FIG. 20. FIG. 20 is a flowchart showing Specific Example 3 of the overall processing flow of the 3D reconstruction system 100 according to the embodiment. Below, as Specific Example 3, a processing flow of the overall 3D reconstruction system 100 executed in video shooting mode on a time-varying subject P1 will be described. Note that the processes of steps S301 to S309 below can also be executed in a different order. Furthermore, some of the processes of steps S301 to S309 below may be omitted.

[0156] First, the camera 30 executes a calibration imaging process (step S301). For example, the camera 30 receives an imaging instruction from the reconstruction device 10, captures an image of the subject P1 through the transparent device 20 while the reference object O1 is hidden, and outputs calibration imaging data D3, which is video data. The reconstruction device 10 also acquires the calibration imaging data D3 from the camera 30.

[0157] Second, the camera 30 executes a moving image capturing start process (step S302). For example, the camera 30 receives a capturing start instruction from the reconstruction device 10 and starts capturing in a moving image capturing mode.

[0158] Third, the transparent device 20 executes a reference object display process (step S303). For example, the transparent device 20 receives a display instruction from the reconstructing device 10 and displays the reference object O1 on the transparent device 20.

[0159] Fourth, the transparent device 20 executes a reference object non-display process (step S304). For example, the transparent device 20 receives a non-display instruction from the reconstructing device 10 and makes the reference object O1 non-displayable on the transparent device 20.

[0160] At this time, if the movement of the subject P1 is completed or a certain time has elapsed (step S305: Yes), the reconstruction device 10 proceeds to the processing of step S306. On the other hand, if the movement of the subject P1 is not completed and the certain time has not elapsed (step S305: No), the reconstruction device 10 returns to the processing of step S303.

[0161] Fifth, the camera 30 executes a video capture end process (step S306). For example, the camera 30 receives a capture end instruction from the reconstruction device 10 and ends capture in video capture mode. The reconstruction device 10 also acquires subject capture data D1, which is video data, from the camera 30. The reconstruction device 10 also acquires reference object capture data D2, which is video data, from the camera 30.

[0162] Sixth, the reconstruction device 10 executes a first pre-processing (step S307). For example, the reconstruction device 10 executes correction of changes in color and brightness of the subject shooting data D1 using the calibration shooting data D3.

[0163] Seventh, the reconstruction device 10 performs second pre-processing (step S308). For example, the reconstruction device 10 uses the reference object shooting data D2 to perform camera pose estimation, real scale acquisition, shooting quality evaluation, etc. of the subject shooting data D1.

[0164] Eighth, the reconstruction device 10 executes a 3D reconstruction process (step S309) and ends the process. For example, the reconstruction device 10 applies photogrammetry, NeRF, 3DGS, or the like to the subject shooting data D1 that has been subjected to the first preprocessing and the second preprocessing, to generate 3D model data M1.

[0165] 2. Effects of the embodiment Finally, effects of the embodiment will be described below. Effects corresponding to the processing according to the embodiment will be described below.

[0166] In the processing according to the embodiment described above, the 3D reconstruction system 100 includes a transparent device 20, a camera 30, and a reconstruction device 10. The transparent device 20 switches between a non-display state in which the reference object O1 is not displayed and a display state in which the reference object O1 is displayed. The camera 30 outputs subject shooting data D1 obtained by shooting a subject P1 placed on the transparent device 20 in the non-display state, and outputs reference object shooting data D2 obtained by shooting the reference object O1 displayed on the transparent device 20 in the display state. The reconstruction device 10 acquires the subject shooting data D1 and the reference object shooting data D2. Therefore, in this processing, the accuracy of 3D reconstruction of the subject P1 can be improved.

[0167] In the processing according to the embodiment described above, the 3D reconstruction system 100 executes display control for switching the transparent device 20 between a non-display state and a display state, the reconstruction device 10 acquires a plurality of subject shooting data D1 and a plurality of reference object shooting data D2 output by a plurality of cameras 30, and generates 3D model data M1 of the subject P1 based on the plurality of subject shooting data D1 and the plurality of reference object shooting data D2. Therefore, this processing enables multi-viewpoint shooting and effective positioning of the reference object O1, thereby improving the accuracy of 3D reconstruction of the subject P1.

[0168] In the process according to the embodiment described above, the reconstruction device 10 executes display control for switching the transparent device 20 between a first state and a second state based on a user's input operation or a preset operation procedure, and the camera 30 outputs the subject shooting data D1 or the reference object shooting data D2 corresponding to each timing when the transparent device 20 is in the first state or the second state. Therefore, in this process, the accuracy of the 3D reconstruction of the subject P1 can be improved by acquiring shooting data based on a user's specification or setting.

[0169] In the processing according to the embodiment described above, the reconstruction device 10 acquires calibration imaging data D3 obtained by capturing an image of the subject P1 through the transparent device 20 in a non-display state, and corrects at least one change in color and brightness of the subject imaging data D1 using the calibration imaging data D3. Therefore, in this processing, the accuracy of the 3D reconstruction of the subject P1 can be improved by correcting the imaging data captured through the transparent device 20 whose transmittance is not 1.

[0170] In the process according to the embodiment described above, the reconstruction device 10 performs at least one of camera pose estimation, real scale acquisition, and photography quality evaluation of the subject photography data D1 using the reference object photography data D2. Therefore, in this process, the accuracy of 3D reconstruction of the subject P1 can be improved by obtaining the camera pose estimation result, the real scale acquisition result, and the photography quality evaluation result.

[0171] In the process according to the embodiment described above, the reconstruction device 10 generates the 3D model data M1 by performing photogrammetry, NeRF, or 3DGS. Therefore, in this process, the accuracy of the 3D reconstruction of the subject P1 can be improved by performing 3D reconstruction using photogrammetry, NeRF, or 3DGS.

[0172] In the process according to the embodiment described above, the reconstruction device 10 stores the three-dimensional model data M1 in a non-transitory storage medium, thereby manufacturing the non-transitory storage medium. Therefore, in this process, the accuracy of the three-dimensional reconstruction of the subject P1 can be improved, and a non-transitory storage medium that stores the high-accuracy three-dimensional model data M1 can be manufactured.

[0173] In the process according to the embodiment described above, the reconstruction device 10 stores the three-dimensional model data M1 in a non-transitory storage medium connected to a display device, thereby manufacturing an image reproduction device configured to reproduce the three-dimensional model data M1. Therefore, in this process, the accuracy of the three-dimensional reconstruction of the subject P1 can be improved, and an image reproduction device capable of reproducing the three-dimensional model data M1 with high accuracy can be manufactured.

[0174] In the process according to the embodiment described above, the transparent device 20 is a transparent display 20A having transparency, and a reference object O1 is displayed on the transparent display 20A. Therefore, in this process, by acquiring reference object shooting data D2 of the reference object O1 displayed on the transparent display 20A, it is possible to improve the accuracy of 3D reconstruction of the subject P1.

[0175] In the process according to the embodiment described above, the transparent device 20 is a transparent screen 20B having transparency, and a reference object O1 projected from a projector 21B is displayed on the transparent screen 20B. Therefore, in this process, by acquiring reference object shooting data D2 of the reference object O1 displayed on the transparent screen 20B, it is possible to improve the accuracy of the 3D reconstruction of the subject P1.

[0176] In the process according to the embodiment described above, the reference object O1 is a marker O1a having a predetermined shape. Therefore, in this process, the accuracy of the 3D reconstruction of the subject P1 can be improved by acquiring reference object image data D2 of the marker O1a displayed on the transparent device 20.

[0177] In the process according to the embodiment described above, the reference object O1 is a chart O1b showing a predetermined graphic pattern. Therefore, in this process, the accuracy of the 3D reconstruction of the subject P1 can be improved by acquiring reference object image data D2 of the chart O1b displayed on the transparent device 20.

[0178] In the process according to the embodiment described above, the reference object O1 is a color checker O1c that indicates a predetermined color. Therefore, in this process, by acquiring reference object image data D2 of the color checker O1c displayed on the transparent device 20, the accuracy of the 3D reconstruction of the subject P1 can be improved.

[0179] In the process according to the embodiment described above, the camera 30 is fixedly installed at a predetermined position around the transparent device 20, and captures images of the rotating transparent device 20 at predetermined time intervals to output subject image data D1 and reference object image data D2 from multiple viewpoints. Therefore, in this process, by enabling multi-viewpoint image capture using the rotatable transparent device 20, the accuracy of 3D reconstruction of the subject P1 can be improved.

[0180] In the process according to the embodiment described above, the camera 30 moves around the transparent device 20, capturing images of the fixed transparent device 20 at predetermined moving distances, thereby outputting subject image data D1 and reference object image data D2 from multiple viewpoints. Therefore, in this process, the accuracy of three-dimensional reconstruction of the subject P1 can be improved by enabling multi-viewpoint image capturing using the movable camera 30.

[0181] In the process according to the embodiment described above, the cameras 30 are fixedly installed at predetermined intervals around the transparent device 20, and output subject shooting data D1 and reference object shooting data D2 from multiple viewpoints by photographing the fixed transparent device 20. Therefore, in this process, by enabling multi-viewpoint shooting using the cameras 30 with the required number of viewpoints, the accuracy of 3D reconstruction of the subject P1 can be improved.

[0182] In the process according to the embodiment described above, the camera 30 outputs still image data, so that the accuracy of the three-dimensional reconstruction of the subject P1 using the still image data can be improved.

[0183] In the process according to the embodiment described above, the camera 30 outputs video data, which makes it possible to improve the accuracy of three-dimensional reconstruction of the subject P1 using the video data.

[0184] 3. Hardware Configuration An information processing device such as the reconfiguration device 10 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in FIG. 21 , for example. The reconfiguration device 10 according to the embodiment will be described below as an example. FIG. 21 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the reconfiguration device 10 according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0185] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0186] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0187] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0188] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0189] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a touch panel, keyboard, mouse, microphone, and camera via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, and printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

[0190] For example, when the computer 1000 functions as the reconstruction device 10 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 15, etc. Furthermore, the HDD 1400 stores the information processing program according to the present disclosure and data in the storage unit 14, etc. Note that the CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0191] Note that the present technology can also be configured as follows. (1) An information processing method executed by an information processing system including a display device, a photographing device, and an information processing device, wherein the display device switches between a first state in which a reference object is not displayed and a second state in which the reference object is displayed, the photographing device outputs first photographing data obtained by photographing a subject placed on the display device in the first state, and outputs second photographing data obtained by photographing the reference object displayed on the display device in the second state, and the information processing device acquires the first photographing data and the second photographing data. (2) The information processing method described in (1), wherein the display device has transparency, and the information processing device executes display control for switching the display device between the first state and the second state, acquires the plurality of first photographing data and the plurality of second photographing data output by the plurality of photographing devices, and generates three-dimensional model data of the subject based on the plurality of first photographing data and the plurality of second photographing data. (3) The information processing method according to (1) or (2), wherein the information processing device executes display control for switching the display device between the first state and the second state based on a user's input operation or a preset operation procedure, and the imaging device outputs first imaging data or second imaging data corresponding to each timing when the display device is in the first state or the second state. (4) The information processing method according to (2) or (3), wherein the information processing device acquires third imaging data obtained by imaging the subject through the display device in the first state, and corrects changes in at least one of color and brightness of the first imaging data using the third imaging data. (5) The information processing method according to any one of (2) to (4), wherein the information processing device executes at least one of camera pose estimation, real scale acquisition, and imaging quality evaluation of the first imaging data using the second imaging data.(6) The information processing method according to any one of (2) to (5), wherein the information processing device generates the three-dimensional model data by executing photogrammetry, Neural Radiance Fields (NeRF), or 3D Gaussian Splatting (3DGS). (7) The information processing method according to any one of (2) to (6), wherein the information processing device manufactures a non-transitory storage medium by storing the three-dimensional model data in the non-transitory storage medium. (8) The information processing method according to any one of (2) to (7), wherein the information processing device manufactures an image reproduction device configured to reproduce the three-dimensional model data by storing the three-dimensional model data in a non-transitory storage medium connected to a display device. (9) The information processing method according to any one of (1) to (8), wherein the display device is a transparent display, and displays the reference object on the display. (10) The information processing method according to any one of (1) to (9), wherein the display device is a transparent screen, and displays the reference object projected from a projector on the screen. (11) The information processing method according to any one of (1) to (10), wherein the reference object is a marker showing a predetermined shape. (12) The information processing method according to any one of (1) to (11), wherein the reference object is a chart showing a predetermined graphic pattern. (13) The information processing method according to any one of (1) to (12), wherein the reference object is a color checker showing a predetermined color. (14) The information processing method according to any one of (1) to (13), wherein the plurality of image capturing devices are fixedly installed at predetermined positions around the display device, and capture images of the rotating display device at predetermined time intervals, thereby outputting the first image capturing data and the second image capturing data from multiple viewpoints.(15) The information processing method according to any one of (1) to (14), wherein the plurality of image capturing devices output the first image capturing data and the second image capturing data from multiple viewpoints by capturing an image of the fixed display device at predetermined movement distances while moving around the display device. (16) The information processing method according to any one of (1) to (15), wherein the plurality of image capturing devices are fixedly installed at predetermined intervals around the display device and output the first image capturing data and the second image capturing data from multiple viewpoints by capturing an image of the fixed display device. (17) The information processing method according to any one of (1) to (16), wherein the image capturing device outputs still image data. (18) The information processing method according to any one of (1) to (17), wherein the image capturing device outputs video data. (19) An information processing system including a display device, a photographing device, and an information processing device, wherein the display device switches between a first state in which a reference object is not displayed and a second state in which the reference object is displayed, the photographing device outputs first photographing data obtained by photographing a subject placed on the display device in the first state, and outputs second photographing data obtained by photographing the reference object displayed on the display device in the second state, and the information processing device includes a control unit that acquires the first photographing data and the second photographing data. (20) An information processing program that causes an information processing system including a display device, an imaging device, and an information processing device to execute the following process: switching the display device between a first state in which a reference object is not displayed and a second state in which the reference object is displayed; outputting, to the imaging device, first imaging data obtained by imaging a subject placed on the display device in the first state; outputting, to the information processing device, second imaging data obtained by imaging the reference object displayed on the display device in the second state; and acquiring, to the information processing device, the first imaging data and the second imaging data.

[0192] REFERENCE SIGNS LIST 10 Reconstruction device 11 Input unit 12 Output unit 13 Communication unit 14 Storage unit 14a Shooting data storage unit 14b 3D model data storage unit 15 Control unit 15a Acquisition unit 15b Display unit 15c Generation unit 20 Transparent device 30 Camera 100 3D reconstruction system

Claims

1. An information processing method executed by an information processing system including a display device, a photographing device, and an information processing device, wherein the display device switches between a first state in which a reference object is not displayed and a second state in which the reference object is displayed, the photographing device outputs first photographing data obtained by photographing a subject placed on the display device in the first state, and outputs second photographing data obtained by photographing the reference object displayed on the display device in the second state, and the information processing device acquires the first photographing data and the second photographing data.

2. The information processing method according to claim 1, wherein the display device has transparency, and the information processing device executes display control for switching the display device between the first state and the second state, acquires a plurality of the first shooting data and a plurality of the second shooting data output by a plurality of the imaging devices, and generates three-dimensional model data of the subject based on the plurality of the first shooting data and the plurality of the second shooting data.

3. The information processing method according to claim 1, wherein the information processing device executes display control for switching the display device between the first state and the second state based on a user's input operation or a preset operating procedure, and the photographing device outputs first photographing data or second photographing data corresponding to each of the times when the display device is in the first state and the second state.

4. The information processing method of claim 2, wherein the information processing device acquires third shooting data obtained by photographing the subject through the display device in the first state, and uses the third shooting data to correct changes in at least one of the color and brightness of the first shooting data.

5. The information processing method according to claim 2, wherein the information processing device uses the second shooting data to perform at least one of camera pose estimation, real scale acquisition, and shooting quality evaluation of the first shooting data.

6. The information processing method according to claim 2, wherein the information processing device generates the three-dimensional model data by executing photogrammetry, Neural Radiance Fields (NeRF), or 3DGS (3D Gaussian Splatting).

7. The information processing method according to claim 2, wherein the information processing device manufactures the non-transitory storage medium by storing the three-dimensional model data in the non-transitory storage medium.

8. The information processing method according to claim 2, wherein the information processing device manufactures an image reproduction device configured to reproduce the three-dimensional model data by storing the three-dimensional model data in a non-transitory storage medium connected to a display device.

9. The information processing method according to claim 1, wherein the display device is a transparent display, and the reference object is displayed on the display.

10. The information processing method according to claim 1, wherein the display device is a transparent screen, and the reference object projected from a projector is displayed on the screen.

11. The information processing method according to claim 1, wherein the reference object is a marker showing a predetermined shape.

12. The information processing method according to claim 1, wherein the reference object is a chart showing a predetermined graphic pattern.

13. The information processing method according to claim 1, wherein the reference object is a color checker that displays a predetermined color.

14. The information processing method described in claim 1, wherein the plurality of imaging devices are fixedly installed at predetermined positions around the display device and output the first imaging data and the second imaging data from multiple viewpoints by imaging the rotating display device at predetermined time intervals.

15. The information processing method described in claim 1, wherein the plurality of imaging devices output the first imaging data and the second imaging data from multiple viewpoints by moving around the display device and imaging the fixed display device at predetermined movement distances.

16. The information processing method described in claim 1, wherein the plurality of imaging devices are fixedly installed at predetermined intervals around the display device, and output the first imaging data and the second imaging data from multiple viewpoints by imaging the fixed display device.

17. The information processing method according to claim 1, wherein the imaging device outputs still image data.

18. The information processing method according to claim 1, wherein the imaging device outputs video data.

19. An information processing system comprising a display device, an imaging device, and an information processing device, wherein the display device switches between a first state in which a reference object is not displayed and a second state in which the reference object is displayed, the imaging device outputs first imaging data obtained by imaging a subject placed on the display device in the first state, and outputs second imaging data obtained by imaging the reference object displayed on the display device in the second state, and the information processing device comprises a control unit that acquires the first imaging data and the second imaging data.

20. An information processing program that causes the information processing system, which includes a display device, an imaging device, and an information processing device, to execute the following process: switching the display device between a first state in which a reference object is not displayed and a second state in which the reference object is displayed; outputting to the imaging device, in the first state, first imaging data obtained by imaging a subject placed on the display device; and outputting, in the second state, second imaging data obtained by imaging the reference object displayed on the display device; and acquiring, in the information processing device, the first imaging data and the second imaging data.

Images