Three-dimensional model generation device, three-dimensional model generation method, and program

Abstract

To generate a three-dimensional model with small data size and high precision in short time.SOLUTION: A three-dimensional model generation device 10 comprises: a unit 132 for generating a plurality of object extraction images from which unnecessary areas except an object area are removed from a plurality of two-dimensional images obtained by photographing an object at multiple viewpoints; and a unit 133 for generating a three-dimensional model of the object by image matching of the plurality of object extraction images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a three-dimensional model generating device, a three-dimensional model generating method, and a program. [Background technology]

[0002] Conventionally, there are known techniques for automatically analyzing the degree of deterioration from an image of an object. For example, Patent Document 1 discloses a technique for extracting tires from a two-dimensional image including wheels using a segmentation technique, identifying text marked on the tires, and analyzing the text content to generate a text analysis result indicating the condition of the tires. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2022 / 0051391 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been an increasing need to measure various physical quantities from three-dimensional data of an object. However, the data size of three-dimensional data is large, which imposes a heavy processing load and reduces the accuracy of the object. In addition, when an object moves, its relationship with other objects changes, making it difficult to create a three-dimensional model of a moving object. In addition, the technology disclosed in Patent Document 1 does not assume the creation of a three-dimensional model of an object.

[0005] In view of the above circumstances, an object of the present disclosure is to provide a 3D model generation device, a 3D model generation method, and a program capable of generating a 3D model with a small data size and high accuracy in a short period of time. [Means for solving the problem]

[0006] The gist of the present disclosure for solving the above problems is as follows.

[0007] (1) A three-dimensional model generation device comprising: an acquisition unit that acquires a plurality of two-dimensional images of an object captured from multiple viewpoints; an unnecessary region removal unit that generates a plurality of object extraction images by removing unnecessary regions other than the object region from the plurality of two-dimensional images; and a three-dimensional model generation unit that generates a three-dimensional model of the object using the plurality of object extraction images. This configuration makes it possible to generate a 3D model with small data size and high accuracy in a short time. In addition, because the background of the object is removed as unnecessary area, it is possible to create a 3D model of a moving object.

[0008] (2) The 3D model generating device according to (1), wherein the 3D model generating unit performs equalization of a luminance histogram for each small region of the object extraction image, and then generates the 3D model. With this configuration, the contrast of the object extraction image can be enhanced, so that even if there is only a small change in the luminance of the object extraction image, feature points can be extracted and a highly accurate 3D model can be generated.

[0009] (3) A 3D model generating device as described in (1) or (2), wherein the multiple 2D images are obtained by photographing the rotating or moving object with a fixed camera, or by photographing the moving object while tracking it with a camera mounted on a drone. This configuration makes it possible to automatically and easily acquire multiple two-dimensional images.

[0010] (4) The three-dimensional model generating device according to any one of (1) to (3), wherein the unnecessary region removing unit generates the object extraction image by using a segmentation model. With this configuration, the object region can be detected with high accuracy.

[0011] (5) The three-dimensional model generating device according to (1) or (2), wherein the three-dimensional model generating unit generates a three-dimensional model of the object by photogrammetry using the plurality of object extraction images. This configuration makes it possible to generate a three-dimensional model with high accuracy.

[0012] (6) The three-dimensional model generating device according to any one of (1) to (5), wherein the object includes a tire and a rubber crawler. With this configuration, it is possible to perform measurements such as measuring the remaining groove depth of an object from a 3D model and diagnosing the state of wear.

[0013] (7) A three-dimensional model generation method in which a three-dimensional model generation device acquires a plurality of two-dimensional images of an object photographed from multiple viewpoints, generates a plurality of object extraction images from the plurality of two-dimensional images by removing unnecessary areas other than the object area, and generates a three-dimensional model of the object using the plurality of object extraction images. This procedure reduces the data size of the 3D model and improves its accuracy. In addition, the background of the object is removed as unnecessary area, so that the moving object can be modeled in 3D.

[0014] (8) A program for causing a computer to function as the three-dimensional model generating device according to any one of (1) to (6). This configuration reduces the data size of the 3D model and improves its accuracy. In addition, because the background of the object is removed as unnecessary area, it is possible to create a 3D model of a moving object. Effect of the Invention

[0015] According to the present disclosure, it is possible to generate a 3D model with a small data size and high accuracy in a short time, and also to create a 3D model of a moving object. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a three-dimensional model generation system according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of the configuration of a three-dimensional model generating device according to an embodiment. [Diagram 3] 1A to 1C are diagrams illustrating an example of an object extraction image generated by a three-dimensional model generating device according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a 3D model generated by a 3D model generating device according to an embodiment. [Diagram 5] 1 is a flowchart showing an example of a procedure of a three-dimensional model generating method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, one embodiment will be described in detail with reference to the drawings.

[0018] Fig. 1 shows an example of the configuration of a 3D model generation system according to an embodiment. The 3D model generation system 1 shown in Fig. 1 includes a 3D model generation device 10 and a camera 20. The 3D model generation system 1 is a system that captures an object 30 from multiple viewpoints using the camera 20 and generates a 3D model of the object 30 using the 3D model generation device 10. The 3D model may be 3D point cloud data, or may be data obtained by converting the 3D point cloud data into mesh data (polygon data), surface data (geometry data), or the like.

[0019] The target object 30 is, for example, a tire or a rubber crawler, but is not limited thereto as long as it is a part that receives force from the ground and transmits driving and braking forces to the main body. The tire is not particularly limited, but may be an OR (Off The Road) tire mounted on a construction vehicle, a mining vehicle, a truck bus tire, an airplane tire, a passenger car tire, or the like.

[0020] The camera 20 photographs the object 30 from multiple viewpoints from multiple directions to obtain multiple (multiple viewpoint) two-dimensional images. The camera 20 may be mounted on a device such as a drone (UAV; Unmanned Aerial Vehicle). For example, the multiple two-dimensional images are obtained by photographing the object 30 while moving the camera 20. Alternatively, the multiple two-dimensional images may be obtained by photographing the rotating or moving object 30 with a fixed camera 20, or may be obtained by photographing the moving object 30 while tracking it with a camera 20 mounted on a drone. Note that the above "rotation" means that the object 30 rotates (turns) around the fixed camera 20.

[0021] The three-dimensional model generating device 10 is a device that generates a three-dimensional model of an object 30 from a plurality of two-dimensional images. The three-dimensional model generating device 10 may include a built-in camera 20. The three-dimensional model generating device 10 may be a smart device such as a smartphone or a tablet.

[0022] Fig. 2 shows an example of the configuration of a 3D model generation device 10 according to an embodiment. The 3D model generation device 10 shown in Fig. 2 includes an input I / F (interface) 11, a camera I / F 12, a control unit 13, a storage unit 14, and an output I / F 15. The 3D model generation device 10 may further include a communication I / F such as a LAN (Local Area Network) I / F to enable communication with an external device.

[0023] The input I / F 11 is an interface between the 3D model generating device 10 and an input device, and the 3D model generating device 10 is connected to the input device via the input I / F 11. The input device is a keyboard, a pointing device, etc. The input I / F 11 receives user instructions and inputs various data from the input device.

[0024] The camera I / F 12 is an interface between the three-dimensional model generating device 10 and the camera 20, and the three-dimensional model generating device 10 is connected to the camera 20 via the camera I / F 12. The camera I / F 12 acquires three-dimensional coordinate data of the object 30 photographed by the camera 20. The camera I / F 12 outputs the three-dimensional coordinate data to the storage unit 14.

[0025] The storage unit 14 includes one or more memories, and may include, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. Each memory included in the storage unit 14 may function, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 14 stores any information used in the operation of the 3D model generation device 10. A part of the storage unit 14 may be provided outside the 3D model generation device 10.

[0026] The output I / F 15 outputs the processing result by the control unit 13 to a display device such as a display. The display device may be integrated with an input device such as a touch screen.

[0027] The control unit 13 performs various controls of the 3D model generating device 10. The control unit 13 includes an acquisition unit 131, an unnecessary region removal unit 132, and a 3D model generating unit 133. The control unit 13 may be configured with dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured with a processor, or may be configured including both.

[0028] The acquisition unit 131 acquires a plurality of two-dimensional images obtained by the camera 20 capturing images of the object 30 from multiple viewpoints.

[0029] The unnecessary region removing unit 132 generates a plurality of object extraction images by removing unnecessary regions (background images) other than the object region from the plurality of two-dimensional images acquired by the acquiring unit 131.

[0030] The unnecessary region removal unit 132 may generate an object extraction image from a two-dimensional image by using a trained model generated by supervised learning. The learning may be machine learning or deep learning. The trained model may be stored in the storage unit 14. For example, the trained model may be a segmentation model trained with a segmentation network. The segmentation may be any of semantic segmentation, instance segmentation, and panoptic segmentation. The unnecessary region removal unit 132 may generate an object extraction image by performing image analysis such as masking a specific color tone by capturing an image of the object 30 against a simple background such as a monochromatic wall or curtain.

[0031] 3 shows an example of an object extraction image generated by the unnecessary region removing unit 132. In this example, the unnecessary region removing unit 132 inputs a two-dimensional RGB image A obtained by photographing the object 30 to a segmentation model to obtain a segmentation image B in which an area a indicating the object 30 is specified. Then, the unnecessary region removing unit 132 generates an object extraction image C in which only the area a indicating the object 30 is extracted from the two-dimensional RGB image A. The unnecessary region removing unit 132 generates a plurality of object extraction images by performing this process on a plurality of two-dimensional RGB images.

[0032] The 3D model generation unit 133 generates a 3D model of the object 30 by image matching using a plurality of object extraction images generated by the unnecessary region removal unit 132. For example, the 3D model generation unit 133 can generate a 3D model of the object 30 from a plurality of object extraction images by photogrammetry. In photogrammetry, feature points of the object extraction images are extracted and feature point matching is performed to obtain tie points and estimate the position and orientation of the camera 20. Then, a 3D point cloud is generated by multi-view stereo measurement.

[0033] 4 shows an example of a 3D model generated by the 3D model generation unit 133. In this example, the 3D model generation unit 133 acquires a plurality of object extraction images C from the unnecessary area removal unit 132. The object extraction images C are images in which only the object 30 is extracted from two-dimensional RGB images in which the object 30 is photographed from different viewpoints. Then, the 3D model generation unit 133 generates a 3D model D of the object 30 from the plurality of object extraction images C using a technique such as photogrammetry.

[0034] Since the number of feature points is small in an object extraction image with little change in luminance (with a biased luminance distribution), the 3D model generation unit 133 may perform preprocessing to make it easier to extract feature points. For example, when the object 30 is a tire or a rubber crawler, the change in luminance of the object extraction image is small, but the number of feature points can be increased by highlighting dirt, scratches, etc. on the tire or rubber crawler. Therefore, the 3D model generation unit 133 may generate a 3D model of the object by image matching (photogrammetry) of the object extraction image in which the object extraction image has been subjected to luminance and contrast adjustment (for example, flattening the histogram for each small region and performing the preprocessing, etc.). By performing this preprocessing, the contrast of the object extraction image can be enhanced, so that even if the change in luminance of the object extraction image is small, feature points can be extracted and a highly accurate 3D model can be generated.

[0035] Since the background of the object is deleted as an unnecessary area, it is not necessary to fix the object 30. Therefore, a plurality of two-dimensional images may be obtained by photographing the rotating or moving object 30 with a fixed camera 20. For example, when the object 30 is a tire, a plurality of two-dimensional images can be automatically and easily obtained by measuring a moving car with a camera 20 fixed at a predetermined position. Also, a plurality of two-dimensional images may be obtained by photographing the moving object 30 while tracking it with a camera 20 mounted on a drone. Even in such a case, the three-dimensional model generating unit 133 can generate a three-dimensional model in the same manner.

[0036] <3D model generation method> Next, a three-dimensional model generating method according to an embodiment will be described with reference to a flowchart of FIG 5.

[0037] In step S101, the camera 20 captures images of the object 30 from multiple directions and multiple viewpoints.

[0038] In step S102, the acquisition unit 131 of the three-dimensional model generating device 10 acquires a plurality of two-dimensional images captured by the camera 20.

[0039] In step S103, the unnecessary region removing unit 132 of the three-dimensional model generating device 10 generates a plurality of object extraction images by removing unnecessary regions other than the object region from the plurality of two-dimensional images.

[0040] In step S104, the 3D model generating unit 133 of the 3D model generating device 10 performs equalization of the luminance histogram for each small region of the object extraction image. Note that this process is not essential.

[0041] In step S105, the 3D model generating unit 133 of the 3D model generating device 10 generates a 3D model of the object 30 by image matching of a plurality of object extraction images.

[0042] As described above, the 3D model generating device 10 reduces the amount of data by removing unnecessary regions using the unnecessary region removing unit 132, and then generates a 3D model from the object extraction image using the 3D model generating unit 133. Therefore, according to this embodiment, it is possible to generate a highly accurate 3D model in a short time. Also, it is possible to create a 3D model of a moving object.

[0043] <Program> A computer capable of executing program instructions can be used to function as the above-mentioned 3D model generating device 10. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

[0044] The computer includes a processor, a storage unit, an input unit, and an output unit. The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be composed of multiple processors of the same type or different types. The processor reads out a program from the storage unit and executes it to control each of the above components and perform various arithmetic processing. At least a part of the processing contents may be realized by hardware. The input unit is an input interface that receives a user's input operation and acquires information based on the user's operation, and is a pointing device, a keyboard, a microphone, or the like. The output unit is an output interface that outputs information, and is a display, a speaker, or the like.

[0045] The program may be recorded on a computer-readable recording medium. By using such a recording medium, the program can be installed on the computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be in a form that is downloaded from an external device via a network.

[0046] Although the above-mentioned embodiments have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiments, and various modifications or changes are possible without departing from the scope of the claims. For example, it is possible to integrate multiple component blocks shown in the configuration diagram of the embodiment, or to divide one component block. It is also possible to integrate multiple steps shown in the flowchart of the embodiment into one, or to divide one step. Contributing to the United Nations-led Sustainable Development Goals (SDGs)

[0047] The SDGs have been proposed to realize a sustainable society. One embodiment of the present invention is expected to contribute to "No. 9 - Building a foundation for industry and technological innovation." [Explanation of symbols]

[0048] 1. 3D model generation system 10. 3D model generation device 11 Input I / F 12 Camera I / F 13 Control section 14 Storage section 15 Output I / F 20 Camera 30 Objects 131 Acquisition Department 132 Unnecessary area removal section 133 3D model generation unit

Claims

1. an acquisition unit that acquires a plurality of two-dimensional images obtained by capturing an object from multiple viewpoints; an unnecessary region removing unit that generates a plurality of object extraction images by removing unnecessary regions other than the object region from the plurality of two-dimensional images; a three-dimensional model generation unit that generates a three-dimensional model of the object using the plurality of object extraction images; A three-dimensional model generating device comprising:

2. The three-dimensional model generating device according to claim 1 , wherein the three-dimensional model generating section performs equalization of a luminance histogram for each small region of the object extraction image before generating the three-dimensional model.

3. 3. The three-dimensional model generating device according to claim 1 or 2, wherein the plurality of two-dimensional images are obtained by photographing the rotating or moving object with a fixed camera, or by photographing the moving object while tracking it with a camera mounted on a drone.

4. The three-dimensional model generating device according to claim 1 , wherein the unnecessary region removing unit generates the object extraction image by using a segmentation model.

5. The three-dimensional model generating device according to claim 1 , wherein the three-dimensional model generating unit generates a three-dimensional model of the object by photogrammetry using the plurality of object extraction images.

6. The three-dimensional model generating device according to claim 1 or 2, wherein the object includes a tire and a rubber crawler.

7. A three-dimensional model generating device Obtaining a plurality of two-dimensional images of an object captured from multiple viewpoints; generating a plurality of object extraction images by removing unnecessary areas other than the object area from the plurality of two-dimensional images; generating a three-dimensional model of the object using the plurality of object extraction images; A three-dimensional model generating method comprising the steps of:

8. A program for causing a computer to function as the three-dimensional model generating device according to claim 1.

Images