3D model generation apparatus, 3D model generation method, image display device, image display method, and program

Abstract

In 3D models, the goal is to reduce the amount of data while suppressing a decrease in accuracy. [Solution] The 3D model generation device 10 includes a virtual camera placement unit 11 that arranges multiple virtual cameras in a 3D space to capture a first 3D model, a rendering processing unit 12 that renders the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images, and a 3D model generation unit 13 that generates a second 3D model using the positions and orientations of the multiple virtual cameras and each of the multiple rendered images.

Description

Technical Field

[0006] , ,

[0005] , , , ,

[0001] The present disclosure relates to a technique for handling a high-precision 3D model.

Background Art

[0002] In recent years, with the development of 3D laser scanners, it has become easier to generate a 3D model of an object to be measured. In addition, the generated 3D model is very high-precision, and the error between the distance measured on the 3D model and the distance measured on the actual object is very small. Therefore, the generated 3D model is utilized in various fields such as architecture, surveying, and inspection of infrastructure.

[0003] Also, in the state where the 3D model is generated by a 3D laser scanner, some point clouds may be missing, and in this case, there is a deviation from the actual object. For this reason, a technique for complementing the missing point clouds using a machine learning model has been disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0007] To achieve the above objective, the 3D model generation apparatus in one aspect of this disclosure is: A virtual camera placement unit arranges multiple virtual cameras in a three-dimensional space to capture a first three-dimensional model, A rendering processing unit that uses the position and orientation of each of the multiple virtual cameras to render the first 3D model and generate multiple rendered images, A 3D model generation unit generates a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. It is characterized by having the following features.

[0008] To achieve the above objective, an image display device in one aspect of this disclosure is A data selection unit selects one of two 3D models based on user input: a first 3D model and a second 3D model created for the same object as the first 3D model, but with less data than the first 3D model. A processing execution unit that performs processing corresponding to the specified instructions on the selected three-dimensional model, A display unit that presents the results of the above processing, It is characterized by having the following features.

[0009] Furthermore, in order to achieve the above objectives, the method for generating a 3D model in one aspect of this disclosure is: A virtual camera placement step involves positioning multiple virtual cameras in a 3D space to capture a first 3D model, and A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step involves generating a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. It is characterized by having the following:

[0010] Furthermore, in order to achieve the above objectives, the image display method in one aspect of this disclosure is A data selection step in which, in response to user input, one of the following is selected: a first 3D model and a second 3D model created for the same object as the first 3D model, but with less data than the first 3D model. A processing execution step, which performs a process corresponding to the specified instruction on the selected three-dimensional model, A presentation step, which presents the results of the above processing, It is characterized by having the following:

[0011] Furthermore, in order to achieve the above objectives, the first program in one aspect of this disclosure is: On the computer, A virtual camera placement step involves positioning multiple virtual cameras in a 3D space to capture a first 3D model, and A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step involves generating a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. It is characterized by causing the execution of [the specified action].

[0012] Furthermore, in order to achieve the above objectives, the second program in one aspect of this disclosure is: On the computer, A data selection step of selecting one of a first 3D model and a second 3D model that is created for the same object as the first 3D model and has a smaller data amount than the first 3D model according to an input user operation; A process execution step of executing a process corresponding to the specified instruction on the selected 3D model; A presentation step of presenting the result of the process; Characterized by causing the above to be executed.

Effect of the Invention

[0013] According to the present disclosure, in a 3D model, it is possible to reduce the data amount while suppressing a decrease in accuracy.

Brief Description of the Drawings

[0014] [Figure 1] FIG. 1 is a configuration diagram showing a schematic configuration of an example of a 3D model generation device. [Figure 2] FIG. 2 is a configuration diagram showing the configuration of an example of a 3D model generation device more specifically. [Figure 3] FIG. 3 is a schematic diagram showing an example of the arrangement of a virtual camera. [Figure 4] FIG. 4 is a schematic diagram showing another example of the arrangement of a virtual camera. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of a 3D model generation device. [Figure 6] FIG. 6 is a configuration diagram showing the configuration of an example of an image display device. [Figure 7] FIG. 7 is a configuration diagram showing the configuration of an example of an image display device more specifically. [Figure 8] FIG. 8 is a flowchart showing an example of the operation of an image display device. [Figure 9] FIG. 9 is a diagram showing the configuration of another example of an image display device. [Figure 10] FIG. 1 is a block diagram showing an example of a computer that realizes a 3D model generation device and an image display device. [Modes for carrying out the invention]

[0015] (Embodiment 1) In the following, an example of a 3D model generation apparatus, a 3D model generation method, and a program in Embodiment 1 will be described with reference to Figures 1 to 5.

[0016] [Device configuration] First, we will explain the schematic configuration of an example of a 3D model generation device using Figure 1. Figure 1 is a schematic diagram showing the schematic configuration of an example of a 3D model generation device.

[0017] As shown in Figure 1, the 3D model generation device 10 is a device for generating another 3D model with a smaller data volume from a high-precision 3D model. As shown in Figure 1, the 3D model generation device 10 comprises a virtual camera placement unit 11, a rendering processing unit 12, and a 3D model generation unit 13.

[0018] The virtual camera placement unit 11 places multiple virtual cameras in a three-dimensional space to capture the first three-dimensional model. The rendering processing unit 12 uses the position and orientation of each of the multiple virtual cameras to render the first three-dimensional model and generate multiple rendered images. The three-dimensional model generation unit 13 uses the position and orientation of each of the multiple virtual cameras and each of the multiple rendered images to generate a second three-dimensional model.

[0019] In this way, the 3D model generation device 10 generates another 3D model from a rendering image of the original 3D model. The second 3D model generated in this way is created for the same object as the first 3D model, but it is a 3D model with less data than the first 3D model. Therefore, the 3D model generation device 10 makes it possible to reduce the amount of data in the 3D model while suppressing a decrease in accuracy.

[0020] Next, the configuration and functions of the 3D model generation device 10 will be explained in more detail using Figures 2 to 4. Figure 2 is a configuration diagram showing a more specific example of the configuration of the 3D model generation device. Figure 3 is a schematic diagram showing an example of the arrangement of virtual cameras. Figure 4 is a schematic diagram showing another example of the arrangement of virtual cameras.

[0021] As shown in Figure 2, the 3D model generation device 10 includes, in addition to the virtual camera placement unit 11, rendering processing unit 12, and 3D model generation unit 13 shown in Figure 1, a data acquisition unit 14 and an output unit 15.

[0022] The data acquisition unit 14 acquires a first 3D model from an external file server, storage, etc. The first 3D model is, for example, 3D point cloud data generated using a 3D laser scanner or the like.

[0023] A 3D laser scanner is equipped with a depth sensor such as LiDAR and an image camera. This configuration allows the 3D laser scanner to perform scanning using the depth sensor and image capture using the image camera, generating 3D point cloud data of the object and image data of the captured images. Furthermore, the 3D laser scanner overlays the image data onto the generated 3D point cloud data to produce the final 3D point cloud data.

[0024] The 3D point cloud data generated in this way is a highly accurate 3D model, and as mentioned above, the amount of data is very large. In other words, the first 3D model is a highly accurate 3D model with a very large amount of data.

[0025] In this embodiment, the virtual camera placement unit 11 arranges multiple virtual cameras so that the first 3D model fits within the rendering field of view. Specifically, as shown in Figure 3, the virtual camera placement unit 11, for example, uses a predetermined position in the 3D space 20 as a reference point, and determines the position of the virtual cameras so that the distance from this reference point, the horizontal angle, and the elevation angle are each in predetermined increments. Furthermore, the virtual camera placement unit 11 sets the orientation of each virtual camera so that its shooting direction is the direction of the reference point.

[0026] In another example, as shown in Figure 4, the virtual camera placement unit 11 determines the position of each virtual camera so as to cover the XYZ coordinates in a grid pattern in the three-dimensional space 20. Furthermore, in this case, the virtual camera placement unit 11 sets the shooting direction of each virtual camera to a predetermined direction (one of the six directions: front, back, left, right, up, and down) for each virtual camera. Note that when determining the position of the virtual cameras, the spacing between virtual cameras in the X, Y, and Z directions may be the same or different.

[0027] Furthermore, the virtual camera placement unit 11 can exclude positions where the shortest distance from the virtual camera to the first 3D model is less than or equal to a predetermined value. This is to prevent the rendered image from becoming a localized image of the first 3D model.

[0028] Furthermore, the virtual camera placement unit 11 can also arrange some or all of the multiple virtual cameras such that some or all of the multiple rendered images include parts of the 3D space other than the first 3D model, that is, parts other than the first 3D model are also captured.

[0029] For example, suppose the first 3D model is a 3D model that shows only the inside of a room within a building. In this case, if the virtual camera is positioned to photograph only the inside of the room, the 3D model generation unit 13 will not be able to determine whether or not the first 3D model showing the outside of the room exists. However, if a virtual camera is also positioned outside the room to photograph parts other than the first 3D model (the outside of the room), information indicating that the first 3D model does not exist outside the room will be added. As a result, the region in which the first 3D model exists becomes clear, and the above determination becomes possible in the processing of the 3D model generation unit 13.

[0030] In this embodiment, the rendering processing unit 12 performs rendering on the 3D point cloud data, which is the first 3D model, using the position and orientation of each of the multiple virtual cameras, as described above, and generates a rendered image for each virtual camera. Existing methods can be used as specific rendering techniques.

[0031] Furthermore, the rendering processing unit 12 can perform rendering by assuming that each point constituting the 3D point cloud data is a solid. Specifically, if the first 3D model is 3D point cloud data, the rendering processing unit 12 assumes that each point included in the point cloud data is a sphere or ellipsoid with size, and performs rendering under that assumption.

[0032] In this case, blank areas in the 3D point cloud data are pseudo-filled in the rendered image. As a result, the 3D model generation unit 13 can stably generate a second 3D model. Furthermore, the spatial continuity of the second 3D model is increased, and the data size is reduced.

[0033] Furthermore, the rendering processing unit 12 determines the size of the sphere or ellipsoid described above based on the point spacing in the 3D point cloud data. For example, the rendering processing unit 12 sets the radius of the sphere to the square root of 3 times the point spacing. In this case, rendering becomes possible so that adjacent spheres touch each other. In addition, the rendering processing unit 12 can deform the sphere into an ellipsoid along the normal vector of the sphere, in which case the surface of the object is represented more accurately in the rendered image.

[0034] Furthermore, in this embodiment, the rendering processing unit 12 determines the intrinsic parameters of the camera in rendering and passes these determined intrinsic parameters to the 3D model generation unit 13. Examples of intrinsic parameters include, but are not limited to, image size and focal length. Other examples of intrinsic parameters include those of an actual camera, such as a camera built into a smartphone. Moreover, the intrinsic parameters may also be those of the camera used to photograph the object (real object) of the first 3D model.

[0035] Furthermore, the rendering processing unit 12 can also generate a mask image or a depth image in addition to the rendered image. The mask image is information indicating whether or not each pixel constituting the rendered image represents empty space. The depth image is information representing the depth from the camera position for each pixel constituting the rendered image. Since the depth image takes a specific value when a pixel represents empty space, it also includes information as a mask image.

[0036] Such mask images and depth images are used, for example, when the 3D model generation unit 13 generates a second 3D model to determine the region in which the second 3D model exists within the entire 3D space.

[0037] In this embodiment, the 3D model generation unit 13 generates a second 3D model by inputting the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images into a machine learning model capable of generating 3D models.

[0038] Specifically, the 3D model generation unit 13 utilizes a method for constructing a 3D model from multi-view photographic images using machine learning models such as NeRF (Neural Radiance Fields) and Gaussian Splatting. In this case, the 3D model generation unit 13 applies multi-view rendering images to the machine learning model instead of multi-view photographic images. Furthermore, in such a method, in addition to the rendering image, the aforementioned mask image and depth image may also be used as auxiliary information.

[0039] Furthermore, in the method described above, the 3D model generation unit 13 first performs SfM (Structure from Motion) using multi-view rendering images to generate initial 3D point cloud data, and then generates a second 3D model based on the generated 3D point cloud data. In this case, instead of performing SfM, the 3D model generation unit 13 may generate initial 3D point cloud data by downsampling the data of the first 3D model.

[0040] Furthermore, the 3D model generation unit 13 can also generate a second 3D model without using a machine learning model. In this case, SfM (Structure from Model) can be used as the generation method.

[0041] The output unit 15 outputs the second 3D point cloud model generated by the 3D model generation unit 13 to the outside.

[0042] [Device operation] Next, an example of the operation of the 3D model generation device 10 will be explained using Figure 5. Figure 5 is a flowchart showing an example of the operation of the 3D model generation device. In the following explanation, Figures 1 to 4 will be referred to as appropriate. In Embodiment 1, the 3D model generation method is carried out by operating the 3D model generation device 10. Therefore, the explanation of the 3D model generation method in Embodiment 1 will be replaced by the following explanation of the operation of the 3D model generation device 10.

[0043] As shown in Figure 5, first, the data acquisition unit 14 acquires a first 3D model from an external file server, storage, etc. (Step A1). The data acquisition unit 14 then passes the acquired first 3D model to the virtual camera 11.

[0044] Next, the virtual camera placement unit 11 places multiple virtual cameras in the 3D space to capture the first 3D model acquired in step A1 (step A2). Specifically, in step A2, the virtual camera placement unit 11 places multiple virtual cameras so that the first 3D model is within the rendering field of view.

[0045] Next, the rendering processing unit 12 uses the positions and orientations of each virtual camera positioned in step A2 to render the first 3D model and generate multiple rendered images (step A3).

[0046] Next, the 3D model generation unit 13 generates a second 3D model using the positions and orientations of each of the multiple virtual cameras and each of the multiple rendering images (step A4). Specifically, in step A4, the 3D model generation unit 13 generates a second 3D model by inputting the positions and orientations of each of the multiple virtual cameras and each of the multiple rendering images into a machine learning model capable of generating 3D models.

[0047] Next, the output unit 15 outputs the second 3D point cloud model generated in step A4 to the outside (step A5).

[0048] Thus, in this embodiment, a second 3D model is generated from a first 3D model with high accuracy. As described above, the second 3D model is created for the same object as the first 3D model, but it is a 3D model with less data than the first 3D model. According to this embodiment, it is possible to reduce the amount of data while suppressing a decrease in accuracy in the 3D model. As a result, the load on data processing using the 3D model can be reduced (see Embodiment 2).

[0049] [program] In Embodiment 1, the program is one that causes a computer to execute steps A1 to A5 shown in Figure 5. By installing and executing this program on a computer, a 3D model generation device 10 and a 3D model generation method can be realized. In this case, the computer's processor functions as a virtual camera placement unit 11, a rendering processing unit 12, a 3D model generation unit 13, a data acquisition unit 14, and an output unit 15, and performs processing. In addition to general-purpose PCs and server computers, the computer can also be a smartphone, a tablet terminal device, etc.

[0050] Furthermore, in Embodiment 1, the program may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: a virtual camera placement unit 11, a rendering processing unit 12, a 3D model generation unit 13, a data acquisition unit 14, and an output unit 15.

[0051] (Embodiment 2) Hereinafter, an example of an image display device, an image display method, and a program in Embodiment 2 will be described with reference to Figures 6 to 8.

[0052] [Device configuration] First, we will explain the schematic configuration of an example image display device using Figure 6. Figure 6 is a configuration diagram showing the configuration of an example image display device.

[0053] The image display device 20 shown in Figure 6 is a device for displaying a three-dimensional model as an image on a screen. As shown in Figure 6, the image display device 20 comprises a data selection unit 21, a processing execution unit 22, and a presentation unit 23.

[0054] The data selection unit 21 selects one of the first 3D model and the second 3D model in response to the user's input. The first 3D model is a high-precision 3D model, as described in Embodiment 1. The second 3D model is created for the same object as the first 3D model, as described in Embodiment 1, but with less data than the first 3D model.

[0055] The processing execution unit 22 executes processing on the selected 3D model that corresponds to the user's input. The presentation unit 23 presents the results of the executed processing.

[0056] In this way, the image display device 20 can select a 3D model in response to an operation input by the user, execute processing, and display the result.

[0057] Next, we will explain the configuration and functions of the image display device 20 in more detail using Figure 7. Figure 7 is a configuration diagram that shows in more detail the configuration of an example of an image display device.

[0058] As shown in Figure 7, in this embodiment, the image display device 20 is connected to the database 30 for data communication. The database 30 may be built into the image display device 20. Database 30 stores the first 3D model and the second 3D model. The second 3D model is a 3D model generated by the 3D model generation device 10 shown in Embodiment 1. In Figure 7, 40 is a display device such as a liquid crystal display device.

[0059] Furthermore, as shown in Figure 7, the image display device 20 includes, in addition to the data selection unit 21, processing execution unit 22, and presentation unit 23 shown in Figure 6, an operation information analysis unit 24 and a data reading unit 25.

[0060] The operation information analysis unit 24 acquires operation information entered by the user and analyzes the acquired operation information. Specifically, the user inputs operation information that identifies the user's operation via input devices such as a touchpad, mouse, and keyboard on the user interface displayed on the screen of the display device 40.

[0061] User operations include operations performed on a 3D model, such as rotation, translation, scaling, selection of parts, and switching between models. Other user operations include selecting areas in 3D space. Therefore, when the operation information analysis unit 24 acquires operation information, it first identifies the operation entered by the user from the operation information and inputs the identified operation into the data selection unit 21.

[0062] Furthermore, the operation information may include specific numerical values, such as the central axis and amount of rotation during rotation, and the coordinate values ​​of the pixels of the selected part. In this case, the operation information analysis unit 24 extracts numerical values ​​from the operation information and inputs the extracted numerical values ​​into the data selection unit 21.

[0063] Furthermore, if the presentation unit 23 is displaying the 3D model on the screen of the display device 40, the operation information analysis unit 24 may also retain information regarding the viewpoint set in the display of the 3D model.

[0064] The data reading unit 25 reads the first 3D model and the second 3D model from the database 30. The data reading unit 25 may have already read both the first and second 3D models, or it may read only the 3D model selected by the data selection unit 21.

[0065] Based on the operation identified by the operation information analysis unit 24, the data selection unit 21 selects either the first 3D model or the second 3D model as the target of the operation.

[0066] Specifically, the data selection unit 21 selects the first 3D model if the specified operation is one in which accuracy is more important than processing speed, or if the specified operation is an operation to display the first 3D model. Examples of operations in which accuracy is more important than processing speed include operations to select a point or part of the 3D model, and operations in which the distance from the viewpoint to the 3D model in the display of the 3D model is below a predetermined threshold.

[0067] Furthermore, if the data selection unit 21 selects a first 3D model, it determines the coordinates of the area to be operated on in the first 3D model. The coordinates are set, for example, by dividing the XYZ coordinates in the 3D space containing the first 3D model into a grid at predetermined intervals.

[0068] The data selection unit 21 selects a second 3D model if the identified operation prioritizes processing speed over accuracy. Examples of operations that prioritize processing speed over accuracy include rotating, scaling, and reducing a 3D model on the screen.

[0069] Furthermore, if the data selection unit 21 selects a second 3D model, it determines the coordinates of the point or part to be operated on and determines a region that includes the coordinates of the determined estimated point. In addition, the region is determined, for example, based on the position and orientation of the viewpoint in the display of the 3D model, so that it is included in the field of view during display.

[0070] The processing execution unit 22 executes processing corresponding to the operation identified by the operation information analysis unit 24 on the 3D model selected by the data selection unit 21 in a virtual 3D space. For example, if the operation is to select a point or part of the 3D model, and the first 3D model is selected, the processing execution unit 22 selects the selected point or part of the first 3D model. Also, if the operation is to rotate the 3D model, and the second 3D model is selected, the processing execution unit 22 rotates the second 3D model according to the operation.

[0071] The display unit 23 displays the processing result on the screen of the display device 40. Specifically, the display unit 23 creates image data for screen display and outputs the created image data to the display device 40 in order to display the 3D model processed in a virtual 3D space by the processing execution unit 22 on the screen. As a result, the processed 3D model is displayed on the screen of the display device 40.

[0072] [Device operation] Next, an example of the operation of the image display device 20 will be explained using Figure 8. Figure 8 is a flowchart showing an example of the operation of the image display device. In the following explanation, Figures 6 and 7 will be referred to as appropriate. In Embodiment 2, the image display method is implemented by operating the image display device 20. Therefore, the explanation of the image display method in Embodiment 2 will be replaced by the following explanation of the operation of the image display device 20.

[0073] As shown in Figure 8, the data reading unit 25 first reads the first 3D model and the second 3D model from the database 30 (step B1).

[0074] Next, the operation information analysis unit 24 determines whether or not the user has entered operation information (step B2). Specifically, the operation information analysis unit 24 determines whether or not the user has entered operation information via an input device.

[0075] If the result of the determination in step B2 indicates that no user input has been made, the operation information analysis unit 24 enters a standby state.

[0076] On the other hand, if the determination in step B2 indicates that the user has entered operation information, the operation information analysis unit 24 acquires the operation information entered by the user and identifies the user's operation by analyzing the acquired operation information (step B3).

[0077] Next, the data selection unit 21 selects either the first 3D model or the second 3D model as the 3D model to be operated on, based on the operation identified in step B3 (step B4).

[0078] Next, the processing execution unit 22 determines whether the 3D model selected in step B4 is the first 3D model or the second 3D model (step B5).

[0079] If the result of the determination in step B5 indicates that the 3D model selected in step B4 is the first 3D model, the processing execution unit 22 determines the coordinates of the area to be operated on in the first 3D model (step B6).

[0080] Furthermore, after step B6 is executed, the processing unit 22 reads the data of the area to be operated on based on the coordinates determined in step B6 (step B7).

[0081] Next, the processing execution unit 22 performs the processing corresponding to the operation identified in step B3 on the data read in step B7 (step B8).

[0082] On the other hand, if the result of the determination in step B5 indicates that the 3D model selected in step B4 is the second 3D model, the processing execution unit 22 executes the processing corresponding to the operation identified in step B3 on the second 3D model (step B9).

[0083] After step B8 or B9 is completed, the display unit 23 displays the result of the processing on the screen of the display device 40 (step B10).

[0084] Thus, in Embodiment 2, a 3D model is selected in response to user input, processing is performed on the selected 3D model, and the results are displayed. This allows the user to perform the desired operations while avoiding processing delays due to increased processing load on the device.

[0085] [program] In Embodiment 2, the program is one that causes a computer to execute steps B1 to B10 shown in Figure 8. By installing and executing this program on a computer, the image display device 20 and the image display method can be realized. In this case, the computer's processor functions as a data selection unit 21, a processing execution unit 22, a presentation unit 23, an operation information analysis unit 24, and a data reading unit 25, and performs processing. In addition to general-purpose PCs and server computers, the computer can also be a smartphone, a tablet terminal device, etc.

[0086] Furthermore, in Embodiment 2, the program may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: a data selection unit 21, a processing execution unit 22, a presentation unit 23, an operation information analysis unit 24, and a data reading unit 25.

[0087] [Differentiation] Here, a modified example of the image display device 20 will be explained using Figure 9. Figure 9 shows the configuration of another example of the image display device.

[0088] As shown in Figure 9, the image display device 20 may also include a 3D model generation device 10 internally. Specifically, in a modified example, the image display device 20 includes, in addition to the configuration shown in Figure 7, a virtual camera placement unit 11, a rendering processing unit 12, a 3D model generation unit 13, a data acquisition unit 14, and an output unit 15.

[0089] In the modified version, the data reading unit 25 reads only the first 3D model from the database 30.

[0090] [Physical configuration] Here, a computer that implements a 3D model generation device or an image display device by executing the programs in Embodiments 1 and 2 will be described using Figure 10. Figure 10 is a block diagram showing an example of a computer that implements a 3D model generation device and an image display device.

[0091] As shown in Figure 10, the computer 110 comprises a CPU (Central Processing Unit) 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication.

[0092] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to, or instead of, the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0093] The CPU 111 loads the program in the embodiment, which consists of a set of codes stored in the storage device 113, into the main memory 112, and performs various calculations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0094] Furthermore, the program in this embodiment is provided stored on a computer-readable recording medium 120. The program in this embodiment may also be distributed over the internet via a communication interface 117.

[0095] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0096] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0097] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0098] Furthermore, the 3D model generation device and image display device can be realized not by a computer with a program installed, but by using hardware corresponding to each part, such as electronic circuits. Moreover, the 3D model generation device and image display device may be partially implemented by a program and the remaining parts by hardware. In the embodiment, the computer is not limited to the computer shown in Figure 10.

[0099] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 21) described below, but are not limited to the following descriptions.

[0100] (Note 1) A virtual camera placement unit arranges multiple virtual cameras in a three-dimensional space to capture a first three-dimensional model, A rendering processing unit that uses the position and orientation of each of the multiple virtual cameras to render the first 3D model and generate multiple rendered images, A 3D model generation unit generates a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. A 3D model generation device characterized by having the following features.

[0101] (Note 2) The virtual camera placement unit places the multiple virtual cameras such that the first 3D model fits within the rendering field of view. The 3D model generation device described in Appendix 1.

[0102] (Note 3) The virtual camera placement unit places some or all of the multiple virtual cameras such that some or all of the multiple rendering images include parts of the three-dimensional space other than the first three-dimensional model. The 3D model generation device described in Appendix 1.

[0103] (Note 4) The first 3D model is 3D point cloud data, The rendering processing unit performs rendering assuming that each point constituting the 3D point cloud data is a solid. The 3D model generation device described in Appendix 1.

[0104] (Note 5) The 3D model generation unit generates the second 3D model by inputting the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images into a machine learning model capable of generating 3D models. The 3D model generation device described in Appendix 1.

[0105] (Note 6) A data selection unit selects one of two 3D models based on user input: a first 3D model and a second 3D model created for the same object as the first 3D model, but with less data than the first 3D model. A processing execution unit that performs the operation corresponding to the selected 3D model, A display unit that presents the results of the above processing, It is equipped with An image display device characterized by the following features.

[0106] (Note 7) A virtual camera placement unit arranges multiple virtual cameras in a three-dimensional space to capture the first three-dimensional model, A rendering processing unit that uses the position and orientation of each of the multiple virtual cameras to render the first 3D model and generate multiple rendered images, A 3D model generation unit generates the second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images, Furthermore, it is equipped with The image display device described in Appendix 6.

[0107] (Note 8) A virtual camera placement step involves positioning multiple virtual cameras in a 3D space to capture a first 3D model, and A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step involves generating a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. A method for generating a 3D model, characterized by having [a certain feature].

[0108] (Note 9) In the virtual camera placement step, the plurality of virtual cameras are positioned such that the first 3D model is within the rendering field of view. The method for generating a 3D model as described in Appendix 8.

[0109] (Note 10) In the virtual camera placement step, some or all of the multiple virtual cameras are positioned such that some or all of the multiple rendering images include parts of the three-dimensional space other than the first three-dimensional model. The method for generating a 3D model as described in Appendix 8.

[0110] (Note 11) The first 3D model is 3D point cloud data, In the rendering step, rendering is performed assuming that each point constituting the 3D point cloud data is a solid. The method for generating a 3D model as described in Appendix 8.

[0111] (Note 12) In the 3D model generation step, the second 3D model is generated by inputting the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images into a machine learning model capable of generating a 3D model. The method for generating a 3D model as described in Appendix 8.

[0112] (Note 13) A data selection step in which, in response to user input, one of the following is selected: a first 3D model and a second 3D model created for the same object as the first 3D model, but with less data than the first 3D model. A processing execution step in which a process corresponding to the operation is performed on the selected 3D model, A presentation step, which presents the results of the above processing, An image display method characterized by having the following:

[0113] (Note 14) A virtual camera placement step involves arranging multiple virtual cameras in a three-dimensional space to capture the first three-dimensional model, A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step is performed, in which the second 3D model is generated using the position and orientation of each of the plurality of virtual cameras and each of the plurality of rendering images. Furthermore, The image display method described in Appendix 13.

[0114] (Note 15) A virtual camera placement step involves positioning multiple virtual cameras in a 3D space to capture a first 3D model, and A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step involves generating a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. A program that executes something.

[0115] (Note 16) In the virtual camera placement step, the plurality of virtual cameras are positioned such that the first 3D model is within the rendering field of view. The program described in Appendix 15.

[0116] (Note 17) In the virtual camera placement step, some or all of the multiple virtual cameras are positioned such that some or all of the multiple rendering images include parts of the three-dimensional space other than the first three-dimensional model. The program described in Appendix 15.

[0117] (Note 18) The first 3D model is 3D point cloud data, In the rendering step, rendering is performed assuming that each point constituting the 3D point cloud data is a solid. The program described in Appendix 15.

[0118] (Note 19) In the 3D model generation step, the second 3D model is generated by inputting the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images into a machine learning model capable of generating a 3D model. The program described in Appendix 15.

[0119] (Note 20) On the computer, A data selection step in which, in response to user input, one of the following is selected: a first 3D model and a second 3D model created for the same object as the first 3D model, but with less data than the first 3D model. A processing execution step in which a process corresponding to the operation is performed on the selected 3D model, A presentation step, which presents the results of the above processing, A program that executes something.

[0120] (Note 21) To the aforementioned computer, A virtual camera placement step involves arranging multiple virtual cameras in a three-dimensional space to capture the first three-dimensional model, A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step is performed, in which the second 3D model is generated using the position and orientation of each of the plurality of virtual cameras and each of the plurality of rendering images. To execute further, The program described in Appendix 20. [Industrial applicability]

[0121] According to this disclosure, it is possible to reduce the amount of data while suppressing a decrease in accuracy in 3D models. This disclosure is useful for computer systems that handle 3D models. [Explanation of Symbols]

[0122] 10. 3D Model Generation Device 11 Virtual Camera Placement Section 12 Rendering Processing Unit 13. 3D Model Generation Unit 14. Data Acquisition Unit 15 Output section 20 Image display devices 21 Data Selection Section 22 Processing Execution Unit 23 Presentation section 24 Operation information analysis section 25 Data reading section 30 databases 40 Display device 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A virtual camera placement unit arranges multiple virtual cameras in a three-dimensional space to capture a first three-dimensional model, A rendering processing unit that uses the position and orientation of each of the multiple virtual cameras to render the first 3D model and generate multiple rendered images, A 3D model generation unit generates a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. A three-dimensional model generation device characterized by having the following features.

2. The virtual camera placement unit places the plurality of virtual cameras such that the first three-dimensional model fits within the rendering field of view. A three-dimensional model generation apparatus according to claim 1.

3. The virtual camera placement unit places some or all of the multiple virtual cameras such that some or all of the multiple rendering images include parts of the three-dimensional space other than the first three-dimensional model. A three-dimensional model generation apparatus according to claim 1.

4. The first three-dimensional model described above is three-dimensional point cloud data, The rendering processing unit performs rendering assuming that each point constituting the three-dimensional point cloud data is a three-dimensional object. A three-dimensional model generation apparatus according to claim 1.

5. The 3D model generation unit generates the second 3D model by inputting the position and orientation of each of the plurality of virtual cameras and each of the plurality of rendering images into a machine learning model capable of generating a 3D model. A three-dimensional model generation apparatus according to claim 1.

6. A data selection unit selects one of two three-dimensional models based on user input: a first three-dimensional model and a second three-dimensional model created for the same object as the first three-dimensional model, but with less data than the first three-dimensional model. A processing execution unit that performs the operation corresponding to the selected three-dimensional model, A display unit that presents the results of the above processing, It is equipped with An image display device characterized by the following features.

7. A virtual camera placement step involves arranging multiple virtual cameras in a three-dimensional space to capture a first three-dimensional model, A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step involves generating a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. A method for generating a three-dimensional model, characterized by having [a certain feature].

8. A data selection step in which, in response to user input, one of a first three-dimensional model and a second three-dimensional model created for the same object as the first three-dimensional model, but with less data than the first three-dimensional model, is selected. A processing execution step in which a process corresponding to the operation is performed on the selected three-dimensional model, A presentation step, which presents the results of the above processing, An image display method characterized by having the following:

9. On the computer, A virtual camera placement step involves arranging multiple virtual cameras in a three-dimensional space to capture a first three-dimensional model, A rendering process step involves rendering the first 3D model using the positions and orientations of the multiple virtual cameras to generate multiple rendered images. A 3D model generation step involves generating a second 3D model using the position and orientation of each of the multiple virtual cameras and each of the multiple rendering images. A program that executes something.

10. On the computer, A data selection step in which, in response to user input, one of a first three-dimensional model and a second three-dimensional model created for the same object as the first three-dimensional model, but with less data than the first three-dimensional model, is selected. A processing execution step in which a process corresponding to the operation is performed on the selected three-dimensional model, A presentation step, which presents the results of the above processing, A program that executes something.

Images