System for generating a three dimensional model from video data

Abstract

To generate and browse a plurality of three dimensional model groups having different target areas and qualities in three dimensional reconstruction.SOLUTION: A system acquires video data by a data collection terminal, restores first posture information of the data collection terminal from the video data, generates one or more three dimensional partial models for one or more partial spaces in which an importance degree is determined to be higher than a threshold value in an entire space captured by the video data on the basis of the first posture information and the video data, restores second posture information of the data collection terminal from the video data, and generates a three dimensional entire model for the entire space on the basis of the second posture information and the video data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system for generating a three-dimensional model from video data. [Background technology]

[0002] Conventionally, methods using red, green, blue, and depth (RGBD) information are often used for high-density 3D reconstruction. For example, Patent Document 1 states, "We provide a method for reconstructing a scene as a 3D model using images acquired by a 3D sensor and an omnidirectional camera. The 3D sensor can be a Kinect (trademark) sensor, and the omnidirectional camera can be a wide field-of-view (FOV) camera. RGB images are acquired by the camera, and RGBD images are acquired by the 3D sensor. Both types of images are used together to construct a 3D model of the scene." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-510419 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques enable dense 3D reconstruction using RGBD information for a variety of scenes expected in industrial applications, such as large scenes and scenes with little texture. Conventional techniques typically generate a single 3D model with uniform quality across the entire region. However, 3D models generated by conventional techniques may not be suitable for industrial use cases.

[0005] Specifically, there are two requirements: one is to be able to view the entire target environment at a glance in order to understand the progress of work and the placement of items, and the other is to be able to check specific areas of the target environment in detail for equipment inspection, training, etc. The former requirement requires a single, low-quality 3D model of the entire environment (an overall model), while the latter requirement requires multiple 3D models of any quality corresponding to specific areas of the environment (a group of partial models). Conventional technology is unable to generate a 3D model that satisfies both requirements with a single measurement.

[0006] An aspect of the present invention aims to generate and view a group of multiple 3D models with different target areas and qualities in 3D reconstruction. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, a representative example of the present invention is a system for generating a 3D model from video data, comprising one or more processors and one or more storage devices, wherein the one or more storage devices store video data from a data collection terminal, and the one or more processors restore first posture information of the data collection terminal from the video data, and based on the first posture information and the video data, generate one or more 3D partial models for one or more subspaces determined to have an importance higher than a threshold in the overall space captured by the video data, restore second posture information of the data collection terminal from the video data, and generate a 3D overall model for the overall space based on the second posture information and the video data. [Effects of the Invention]

[0008] According to one aspect of the present invention, in 3D reconstruction, a group of multiple 3D models with different target areas and qualities can be generated and viewed. [Brief explanation of the drawings]

[0009] [Figure 1] Figure 1 shows the configuration of an industrial high-density 3D reconstruction system. [Figure 2] FIG. 2 is a flowchart showing the procedure of the partial model group generation process. [Figure 3] FIG. 3 is a flowchart showing the procedure for generating the whole model. [Figure 4] FIG. 4 is a diagram showing an example of the overall model display interface. [Figure 5] FIG. 5 is a diagram showing an example of a partial model display interface. [Figure 6] FIG. 6 is a diagram showing a schematic configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention. Furthermore, illustrations and descriptions of elements that are essential to the configuration of the invention but are well known may be omitted.

[0011] In the following explanation, information that produces an output for an input may be described using expressions such as "xxx table," but this information may be data of any structure. Therefore, "xxx table" can also be called "xxx information."

[0012] Furthermore, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0013] In the following explanation, processing may be described using the "program" as the subject. Since the program is executed by the processor unit to perform predetermined processing while appropriately using the storage unit and / or interface unit, the subject of the processing may be the processor unit (or a device such as a controller that has the processor unit).

[0014] The program may be installed in a device such as a computer, or may be stored in, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0015] Furthermore, a "processor unit" refers to one or more processors. The processor is typically a microprocessor such as a CPU (Central Processing Unit), but may also be other types of processors such as a GPU (Graphics Processing Unit). The processor may be single-core or multi-core. The processor may also be a processor in the broader sense, such as a hardware circuit that performs part or all of the processing (for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0016] In the following description, when elements of the same type are described without distinction, reference signs (or common signs among reference signs) may be used, and when elements of the same type are described with distinction, identification numbers (or reference signs) of the elements may be used. Furthermore, the number of elements shown in each drawing is an example and is not limited to the number shown.

[0017] Fig. 1 is a configuration diagram of an industrial high-density 3D reconstruction system. The industrial high-density 3D reconstruction system shown in Fig. 1 includes a data collection terminal 100, a 3D reconstruction device 110, and a 3D model display device 120.

[0018] The data collection terminal 100 includes an integrated sensor unit 101 , a data collection processing unit 102 , and a notification input unit 103 .

[0019] The integrated sensor unit 101 is composed of a sensor capable of acquiring RGBD information and other sensors that can be added as needed. The sensor capable of acquiring RGBD information is, for example, an RGBD camera or a stereo camera. The other sensor is, for example, an inertial measurement unit.

[0020] The data collection processing unit 102 collects data obtained from the integrated sensor unit and transmits it to the 3D reconstruction device 110 or stores it in a secondary storage device on the data collection terminal 100. Data collection means, for example, time-synchronizing and stacking data transmitted sequentially from each sensor of the integrated sensor unit in a chronological order.

[0021] The notification input unit 103 notifies people of information and accepts input from people, such as displaying acquired RGB information and inputting the start and end of data collection.

[0022] The three-dimensional reconstruction device 110 includes a partial model group SLAM processing unit 111, a partial model group generation processing unit 112, an overall model SLAM processing unit 113, an overall model generation processing unit 114, and a notification input unit 115.

[0023] The partial model group SLAM processing unit 111 performs SLAM processing on the data transmitted or stored by the data collection processing unit 102, and restores the attitude information (including position and direction information) of the data collection terminal 100 used to generate the partial model group. The restored attitude information of the data collection terminal 100 is transmitted to the partial model group generation processing unit 112, or is stored in a secondary storage device on the 3D reconstruction device 110.

[0024] The partial model group SLAM processing unit 111 can use a known SLAM technique that uses RGBD information, such as ORB-SLAM.

[0025] The partial model group generation processing unit 112 generates a partial model group by integrating the RGBD information and the posture information of the data collection terminal 100 for the data transmitted or stored by the data collection processing unit 102 and the partial model group SLAM processing unit 111. The generated partial model group is transmitted to the 3D model display device 120 or stored in a secondary storage device on the 3D reconstruction device 110.

[0026] The SLAM processing unit for overall model 113 performs SLAM processing on the data transmitted or stored by the data collection processing unit 102, and restores the posture information of the data collection terminal 100 to be used for generating the overall model. The restored posture information of the data collection terminal 100 is transmitted to the overall model generation processing unit 114, or is stored in a secondary storage device on the 3D reconstruction device 110.

[0027] The whole model SLAM processing unit 113 can use a known SLAM technique that uses RGBD information, such as ORB-SLAM.

[0028] The overall model generation processing unit 114 generates an overall model by integrating the RGBD information and the posture information of the data collection terminal 100 for the data transmitted or stored by the data collection processing unit 102 and the overall model SLAM processing unit 113. The generated overall model is transmitted to the 3D model display device 120 or stored in a secondary storage device on the 3D reconstruction device 110.

[0029] The notification input unit 115 notifies people of information and accepts human input, such as input to start 3D reconstruction, input of various parameters, and display of the progress of 3D model generation.

[0030] The partial model group SLAM processing unit 111 and the entire model SLAM processing unit 113 may be the same unit.

[0031] The processing format in the partial model group SLAM processing unit 111 and the partial model group generation processing unit 112 may be, for example, a format in which data transmitted or stored by the data collection processing unit 102 is processed sequentially one frame at a time, or a format in which all data stored by the data collection processing unit 102 is processed in batches.

[0032] The three-dimensional model display device 120 includes a notification input unit 121 .

[0033] The notification input unit 121 notifies people of information about the partial model group and the overall model transmitted or saved by the partial model group generation processing unit 112 and the overall model generation processing unit 114, and also accepts human input. For example, it selects a 3D model, visualizes the 3D model, changes the viewpoint of the 3D model, etc.

[0034] 2 is a flowchart showing the procedure for generating a partial model group in the partial model group generation processing unit 112. For example, a partial model group is generated from a video file currently being shot, or one or more shot video files. First, for a new partial model, a TSDF volume is generated (S200), which is a three-dimensional representation obtained using a truncated signed distance function (TSDF), which is a known distance function.

[0035] Next, the next data (frame) is received from the data collection processing unit 102 and the partial model group SLAM processing unit 111 (S201). Information obtained by SLAM processing includes the attitude (position and direction) of the data collection terminal 100 for that frame, and information on the position of the frame in the entire video area (positional relationship between frames). Next, the importance of the subject (frame) in the frame being processed is calculated (S202).

[0036] Next, it is determined whether the state of high importance continues (S203). If the state of high importance does not continue, the process returns to S201. If the state of high importance continues, the RGBD information for the current frame and the posture information of the data collection terminal 100 are integrated (stored) into a TSDF volume for a partial model at a target resolution set in advance (S204). This ensures that the resolution required for the partial model referenced for detailed information is ensured. The resolution is calculated from the texture resolution, the number of vertices of the mesh, etc.

[0037] Next, it is determined whether or not to end the processing at the current frame (S205). For example, the last frame in a video file or the end of video capture may be the cause of the processing ending. If the processing ends at the current frame (S205: YES), polygon meshes (the entities of the 3D partial models) are generated and saved from all TSDF volumes (S208), and the partial model group generation process ends.

[0038] If processing does not end with the current frame (S205: NO), it is determined whether the capacity of the TSDF volume for the partial model is larger than the computational resources of the 3D reconstruction device 110 (e.g., the capacity of the primary storage device) (S206). For example, it is determined whether it is larger than a preset threshold. If the capacity of the TSDF volume for the partial model is larger than the computational resources of the 3D reconstruction device 110, the TSDF volume for the partial model is saved and the process returns to S200. If the capacity of the TSDF volume for the partial model is equal to or smaller than the computational resources of the 3D reconstruction device 110, the process returns to S201. This allows the capacity of each TSDF volume for a partial model to be kept below a value appropriate for processing.

[0039] The TSDF volume in the partial model group generation processing procedure of FIG. 2 may be a volume obtained by a different distance function.

[0040] The importance of the subject (frame) in S202 can be calculated, for example, from the moving speed of the data collection terminal 100 in the frame being processed, the frequency of the RGBD information, etc. For example, the importance is expressed as a function of the moving speed of the frame and / or the frequency of the RGBD information, and the slower the moving speed, the higher the calculated importance, and the higher the calculated frequency, the higher the calculated importance. The frequency of the RGBD information is expressed, for example, as a function of the frequencies of R, G, B, and D, and some of these may be excluded. Alternatively, the importance for generating a partial model may be assigned to a pre-specified partial space.

[0041] In step S203, the determination of whether the high importance state continues can be made by using, for example, a moving average of the importance of frames surrounding the frame being processed. The surrounding frames can be, for example, a predetermined number of frames before the target frame or a predetermined number of frames before and after the target frame.

[0042] For mesh generation in S208, for example, the Marching Cubes algorithm can be used.

[0043] 3 is a flowchart showing the overall model generation processing procedure in the overall model generation processing unit 114. First, all data is received from the data collection processing unit 102 and the overall model SLAM processing unit 113 (S300). Next, for the received data, one 3D bounding box is calculated from the movement trajectory of the data collection terminal 100 (S301).

[0044] Next, the mesh volume (volume of the 3D model) after the entire model is reconstructed is estimated from the bounding box (S302). For example, the mesh volume is estimated based on the volume of the bounding box and the reference resolution. Next, a TSDF volume for the new entire model is generated (S303). Next, the next data (frame) is extracted from all data received from the data collection processing unit 102 and the entire model SLAM processing unit 113 (S304).

[0045] Next, it is determined whether a partial model group exists near the orientation (position and / or orientation) of the data collection terminal 100 in the frame being processed (S305). The determination of the vicinity may be made by referring to a preset threshold. If a partial model group does not exist near the orientation of the data collection terminal 100 in the frame being processed (S305: NO), the RGBD information in the current frame and the orientation information of the data collection terminal 100 are integrated into the TSDF volume for the entire model at a resolution such that the estimated mesh capacity after reconstructing the entire model is equal to or less than (e.g., equal to) the smaller of a preset upper limit mesh capacity that allows comfortable viewing of the mesh and the computational resources (e.g., the capacity of a primary storage device) of the 3D reconstruction device 110 (S306). The computational resource limit of the 3D reconstruction device 110 may be the same as the computational resource limit of the 3D reconstruction device 110 in generating the partial model. The same applies to step S307.

[0046] The resolution is determined by recalculating the mesh capacity estimated in step S302 using the resolution as a variable, comparing the calculated value with the smaller of the two values, and determining the resolution at which the difference is a predetermined value. At this time, it may be assumed that all frames have the same resolution.

[0047] If a group of partial models exists near the posture of the data collection terminal 100 in the frame being processed (S305: YES), the RGBD information in the current frame and posture information of the data collection terminal 100 are integrated into the TSDF volume for the entire model at a resolution such that the estimated mesh capacity after the entire model is reconstructed is sufficiently smaller than the smaller of the upper limit mesh capacity set in advance to allow comfortable viewing of the mesh and the computational resources of the 3D reconstruction device 110 (S307). This makes it possible to reduce the data capacity while suppressing degradation in the quality of the 3D model.

[0048] The resolution is determined by recalculating the mesh capacity estimated in step S302 using the resolution as a variable, comparing this value with the smaller of the two values, and determining the resolution at which the difference is a predetermined value. This predetermined value is greater than the predetermined value in step S306. At this time, it may be assumed that all frames have a common resolution. As described above, frames are classified into two types: frames in which a partial model exists nearby, and frames in which a partial model does not exist nearby, and a lower resolution is assigned to frames in which a partial model exists nearby. Note that all frames may be integrated at a common resolution so that the estimated mesh capacity condition is met.

[0049] Next, it is determined whether the current frame is the final frame (S308). If the current frame is the final frame (S308: YES), a mesh is generated from the TSDF volume for the entire model (S309), and the entire model generation process ends. If the current frame is not the final frame (S308: NO), the process returns to S304. The process of step S308 is the same as step S205.

[0050] The TSDF volume in the overall model generation processing procedure of FIG. 3 may be a volume obtained by a different distance function.

[0051] In S301, instead of calculating the bounding box of the movement trajectory of the data collection terminal, for example, principal component analysis can be used.

[0052] To estimate the mesh volume after reconstruction of the entire model from the bounding box in S302, for example, the volume of the bounding box can be used.

[0053] For mesh generation in S309, for example, the Marching Cubes algorithm can be used.

[0054] Fig. 4 is a diagram showing an example of a display interface for an entire model in the three-dimensional model display device 120. Fig. 5 is a diagram showing an example of a display interface for a partial model in the three-dimensional model display device 120.

[0055] The three-dimensional model display interface 400 is a user interface for selecting data to be displayed, displaying the entire model or a partial model, selecting a partial model, and the like.

[0056] When displaying the overall model, the three-dimensional model display interface 400 comprises an information display window 410, an overall model display area 420, a data selection button 430, and a partial model selection button 440.

[0057] The information display window 410 displays the data identifier, the data measurement date and time, the capacity of the three-dimensional model, and so on.

[0058] The overall model display area 420 is an area for displaying the overall model and for performing operations such as changing the display viewpoint of the overall model. In the overall model display area 420, an overlay display 421 is displayed, which shows the area where the partial model exists and its identifier.

[0059] The data selection button 430 is a button that is pressed when switching the display target of the three-dimensional model to one obtained by another measurement.

[0060] The partial model selection button 440 is a button that is pressed when switching the display to a partial model with an identifier corresponding to the label of the button.

[0061] When displaying a partial model, the three-dimensional model display interface 400 comprises an information display window 410 , a data selection button 430 , a partial model selection button 440 , a partial model display area 450 , and an entire model display button 460 .

[0062] The partial model display area 450 is an area for displaying the partial model and for performing operations such as changing the display viewpoint of the partial model. In the partial model display area 450, a satellite display 451 of the whole model is displayed.

[0063] The whole model display button 460 is a button that is pressed when switching the display to the whole model.

[0064] 6 shows an example of the hardware configuration of a computer 600 according to an embodiment of the present specification. The data collection terminal 100, the 3D reconstruction device 110, and the 3D model display device 120 may each have a computer configuration. Note that some or all of these devices may be implemented in a single computer, or each device may be implemented in a system including multiple computers.

[0065] The computer 600 includes a processor 601 that executes various programs, a primary storage device 602 that stores the various programs, and a secondary storage device 603 that stores various data. The processor 601 can include one or more cores, and the primary storage device 602 is, for example, a DRAM that includes a volatile storage area. The secondary storage device 603 is, for example, a hard disk drive (HDD) or flash memory, and can provide a non-volatile storage area.

[0066] The computer 600 further includes an output device 604 for presenting information to a user of the device, an input device 605 for inputting instructions, images, etc. from the user, and a communication device 606 for communicating with other devices. The components of the computer 600 are interconnected by a bus. A user may use a terminal connected to the computer 600 via a network instead of the input / output devices of the computer 600.

[0067] The functional units of the computer 600 can be implemented, for example, by the processor 601 operating in accordance with a program. The processor 601 reads and executes various programs from the primary storage device 602 as needed. The primary storage device 602 can store programs and data used by them. Each program and reference data is loaded, for example, from the secondary storage device 603 to the primary storage device 602, and executed and processed by the processor 601. Note that at least a portion of the functional units may be configured with logic circuits.

[0068] The output device 604 may include devices such as a display, a printer, and a speaker. The input device 605 may include devices such as a keyboard, a mouse, and a microphone. The output device 204 presents the results of input from the user and also presents the results of processing by the computer 600. Instructions from the user are input to the computer 600 by the input device 205. When another terminal is used, its input / output device functions in the same way, and the output device 604 and the input device 605 can be omitted.

[0069] The communication device 606 receives data transmitted from other devices connected via a network, and transmits the results of processing by the computer 600 to other devices. Note that some devices may be omitted.

[0070] As described above, according to the high-density 3D reconstruction system for industry of this embodiment, the data collected by the data collection terminal is subjected to 3D reconstruction processing by the SLAM processing unit for partial model groups, the partial model group generation processing unit, the SLAM processing unit for overall model, and the overall model generation processing unit of the 3D reconstruction device, respectively, thereby generating multiple 3D model groups with different target areas and qualities with a single measurement.

[0071] Furthermore, these multiple 3D models can be displayed and operated in an integrated manner on a 3D model display device, which makes it possible to easily satisfy two needs for industrial 3D models: a glance at the entire environment and detailed display of specific areas, and simplifies the process of generating and viewing industrial 3D models.

[0072] The present invention is not limited to the above-described embodiment, but includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, not only can the configurations be deleted, but also replacements and additions of configurations are possible.

[0073] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a recording medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the recording medium. In this case, the program code itself read from the recording medium implements the functions of the above-described embodiments, and the program code itself and the recording medium on which it is stored constitute the present invention. Examples of recording media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0074] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, Python, Shell, PHP, and Java (registered trademark).

[0075] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected. [Explanation of symbols]

[0076] 100...data collection terminal, 101...integrated sensor unit, 102...data collection processing unit, 103...notification input unit, 110...3D reconstruction device, 111...SLAM processing unit for partial model group, 112...partial model group generation processing unit, 113...SLAM processing unit for overall model, 114...overall model generation processing unit, 115...notification input unit, 120...3D model display device, 121...notification input unit, 400...3D model display interface, 410...information display window, 420...overall model display area, 421...overlay display, 430...data selection button, 440...partial model selection button, 450...partial model display area, 451...satellite display of partial model, 460...overall model display button

Claims

1. A system for generating a three-dimensional model from video data, one or more processors; one or more storage devices; the one or more storage devices store video data from the data collection terminal; The one or more processors: Recovering first attitude information of the data collection terminal from the video data; generating one or more three-dimensional partial models for one or more partial spaces determined to have importance higher than a threshold in an entire space captured by the video data, based on the first posture information and the video data; Recovering second attitude information of the data collection terminal from the video data; The system generates a three-dimensional global model for the global space based on the second pose information and the video data.

2. 10. The system of claim 1, The one or more processors generate each three-dimensional part model based on the importance of successive frames in the video data.

3. 10. The system of claim 1, The one or more processors: generating a volume for each three-dimensional part model using a predetermined distance function; The system integrates the frames of video data into a volume of each three-dimensional part model such that the volumetric capacity is within a threshold.

4. 10. The system of claim 1, The one or more processors: generating a volume for each three-dimensional part model using a predetermined distance function; The system integrates the frames of video data into the volume at a target resolution.

5. 10. The system of claim 1, The one or more processors: generating a volume of the three-dimensional overall model using a predetermined distance function; The system determines a resolution of the frames of image data to be integrated into the volume such that an estimated volume after reconstruction of the three-dimensional overall model is within a threshold.

6. 10. The system of claim 1, The one or more processors: generating the three-dimensional whole model after generating the one or more three-dimensional partial models; generating a volume of the three-dimensional overall model using a predetermined distance function; sequentially integrating frames of the video data into the volume at a determined resolution; The system determines a resolution of a frame determined to be a neighborhood of any of the one or more three-dimensional part models to be lower than the resolution of other frames.

7. 10. The system of claim 1, The one or more processors: generating a volume for each three-dimensional part model using a predetermined distance function; The system integrates the frames of image data into the volume of each three-dimensional part model at a target resolution.

8. 10. The system of claim 1, The one or more processors: displaying an image showing an area of ​​the one or more three-dimensional partial models on an image of the three-dimensional whole model; In response to a user selection from the one or more three-dimensional part models, the system displays an image of the selected three-dimensional part model.

9. 1. A method for generating a three-dimensional model from video data, performed by a system, comprising: The system, Acquire video data from a data collection terminal, Recovering first attitude information of the data collection terminal from the video data; generating one or more three-dimensional partial models for one or more partial spaces determined to have importance higher than a threshold in an entire space captured by the video data, based on the first posture information and the video data; Recovering second attitude information of the data collection terminal from the video data; generating a three-dimensional global model for the global space based on the second pose information and the video data.

Images