Image processing method, image processing device, and image processing program

Abstract

In an image processing method according to the present disclosure, a computer acquires a segmentation result of a 2D image group corresponding to a 3D model, corrects the segmentation result of the 2D image group on the basis of the shape of the 3D model, and executes processing for performing segmentation of the 3D model on the basis of the corrected segmentation result of the 2D image group.

Description

Image processing method, image processing device, and image processing program

[0001] The present disclosure relates to an image processing method, an image processing device, and an image processing program.

[0002] In the segmentation of 3D models (point clouds / meshes), conventional techniques are known that use the results of 2D segmentation of 2D images obtained by rendering the 3D model or of 2D images that are the source of the 3D model.

[0003] Special Publication No. 2023-518908

[0004] "Segment Anything 3D", "Searched on April 9, 2020", Internet <URL: https: / / github.com / Pointcept / SegmentAnything3D>

[0005] However, the above-described conventional techniques are based on the premise that the 3D model and the 2D image group are perfectly matched without any misalignment or missing parts, etc. Therefore, if the 2D segmentation results for the 2D image group with incomplete matching are used as they are, there is a problem that the accuracy of the segmentation results for the 3D model will be poor.

[0006] For example, when a 3D model is acquired from a group of 2D images using photogrammetry or LiDAR (Light Detection and Ranging), the generated 3D model may be missing parts due to factors such as the shooting position and the shape of the subject. As an example, small objects (e.g., handrails, tree branches, etc.) reflected in the group of 2D images may be missing from the 3D model. In such cases, the missing objects in the 3D model will be included in the results of 2D segmentation of the group of 2D images, and using the results of 2D segmentation as is will result in inconsistencies in the segmentation results of the 3D model.

[0007] Therefore, the present disclosure provides an image processing method, an image processing device, and an image processing program that can improve the accuracy of segmentation of a 3D model.

[0008] It should be noted that the above problem or object is merely one of multiple problems or objects that can be solved or achieved by multiple embodiments disclosed in this specification.

[0009] The image processing method disclosed herein includes a computer obtaining segmentation results of a group of 2D images corresponding to a 3D model, modifying the segmentation results of the group of 2D images based on the shape of the 3D model, and segmenting the 3D model based on the modified segmentation results of the group of 2D images.

[0010] FIG. 1 is an explanatory diagram illustrating an overview of an embodiment of the present disclosure. FIG. 2 is an explanatory diagram illustrating an example of operation of an embodiment of the present disclosure. FIG. 3 is a block diagram illustrating an example of the functional configuration of an image processing device according to an embodiment of the present disclosure. FIG. 4 is a flowchart illustrating an example of operation of an image processing device according to an embodiment of the present disclosure. FIG. 5 is a flowchart illustrating an example of processing related to improving the accuracy of 2D segmentation. FIG. 6 is an explanatory diagram illustrating an example of a 2D segmentation result and a clustering result. FIG. 7 is an explanatory diagram illustrating an example of a 2D segmentation result. FIG. 8 is an explanatory diagram illustrating a comparison between a segmentation result of a 3D model according to an embodiment of the present disclosure and a conventional example. FIG. 9 is a flowchart illustrating an example of operation of an image processing device according to a modified example. FIG. 10 is an explanatory diagram illustrating an example of a display screen. FIG. 11 is a hardware configuration diagram illustrating an example of a computer.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] In this specification and drawings, similar components of the embodiments may be distinguished by adding at least one different alphabet and / or number after the same reference numeral. However, if there is no need to particularly distinguish between the similar components, the same reference numeral will be used.

[0013] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0014] (Outline of the embodiment) First, an outline of the embodiment of the present disclosure will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is an explanatory diagram illustrating an outline of the embodiment of the present disclosure. Fig. 2 is an explanatory diagram illustrating an operation example of the embodiment of the present disclosure.

[0015] As shown in FIG. 1 , an image processing device according to an embodiment of the present disclosure performs segmentation on a 3D model 1, such as a mesh or a point cloud, using a 2D segmentation result 2 of a 2D image corresponding to the 3D model 1, and generates a 3D model 4 with segmentation.

[0016] Here, the segmentation of the 3D model 1 identifies which class (e.g., tree, road, building, etc.) each mesh or point cloud included in the 3D model 1 is divided into. In the segmented 3D model 4, the class identified for each mesh or point cloud is assigned as a flag. In this embodiment, the 3D model 1 is point cloud data corresponding to a 3D object, but it goes without saying that this point cloud data may be replaced with mesh data.

[0017] The image processing device according to the embodiment of the present disclosure obtains a more accurate 2D segmentation result 3 by modifying the 2D segmentation result 2 based on the shape of the 3D model 1. Then, the image processing device according to the embodiment of the present disclosure generates a 3D model 4 with segmentation based on the 2D segmentation result 3 modified based on the shape of the 3D model 1.

[0018] 2 , the image processing device according to the embodiment of the present disclosure receives input of a 3D model 1 and a group of 2D images corresponding to the 3D model 1 (S1, S2). Here, the group of 2D images is, for example, a group of photographs taken of the object of the 3D model 1 from various positions and degrees of accuracy, or a group of images obtained by rendering the 3D model 1.

[0019] Next, the image processing device according to the embodiment of the present disclosure performs 2D segmentation on each of the 2D image groups (S3), thereby obtaining 2D segmentation results 2 for each image included in the 2D image group.

[0020] For 2D segmentation, well-known techniques can be applied, such as using machine learning models such as 2D segmentation-based models. Current 2D segmentation-based models (segment-anything) can segment almost anything in 2D images, making this method applicable regardless of the domain, such as indoors or outdoors. Furthermore, for 2D segmentation-based models, there is essentially no need to train them for each domain. While fine-tuning for each domain can be expected to further improve accuracy, even if this is done, a 2D training dataset can be prepared.

[0021] Next, the image processing device according to the embodiment of the present disclosure improves the accuracy of the 2D segmentation result 2 of each image included in the 2D image group by correcting the result to match the shape of the 3D model 1 based on the shape of the 3D model 1 (S4). As a result, the image processing device according to the embodiment of the present disclosure obtains a more accurate 2D segmentation result 3.

[0022] Next, the image processing device according to an embodiment of the present disclosure generates a 3D model 4 with segmentation by reflecting the 2D segmentation result 3, which has been corrected based on the shape of the 3D model 1, in the 3D model 1 (S5).

[0023] In this way, in the image processing device according to an embodiment of the present disclosure, a 3D model 4 with segmentation is generated using a 2D segmentation result 3 that matches the shape of the 3D model 1, so that accurate segmentation results can be obtained that are consistent with the segmentation results.

[0024] (Configuration Example of Embodiment) FIG. 3 is a block diagram showing a functional configuration example of an image processing apparatus according to an embodiment of the present disclosure.

[0025] As shown in Fig. 3, the image processing device 10 includes a communication unit 11, a storage unit 12, an input / output unit 13, and a control unit 14. Note that the configuration shown in Fig. 3 is a functional configuration, and the hardware configuration may be different from this. Furthermore, the functions of the image processing device 10 may be statically or dynamically distributed and implemented in multiple physically separated configurations. The image processing device 10 may also be configured by multiple information processing devices.

[0026] The communication unit 11 is a communication interface for wireless communication with other devices. The communication unit 11 communicates with other devices via, for example, the Internet. The communication unit 11 functions as a communication means of the image processing device 10. The communication unit 11 communicates with other devices under the control of the control unit 14.

[0027] The storage unit 12 is a data readable / writable storage device such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a hard disk, etc. The storage unit 12 stores various data such as input data 121, calculation data 122, and processing results 123. The storage unit 12 functions as a storage means of the image processing device 10.

[0028] The input data 121 is data input via the communication unit 11 or the input / output unit 13. The input data 121 includes, for example, a 3D model 1 and a group of 2D images corresponding to the 3D model 1.

[0029] The calculation data 122 is various data generated during the calculation process of the control unit 14. For example, the calculation data 122 includes various data generated during the process of obtaining the segmented 3D model 4. As an example, the calculation data 122 includes 2D segmentation results 2 and 3, a depth map 5, a clustering result 6, etc. (see FIG. 5 ).

[0030] The processing result 123 is a result of the calculation process performed by the control unit 14. For example, the processing result 123 includes a 3D model 4 with segmentation.

[0031] The control unit 14 is a controller that controls each unit of the image processing device 10. The control unit 14 is realized by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). For example, the control unit 14 is realized by the processor executing various programs stored in a storage device inside the image processing device 10 using a RAM (Random Access Memory) or the like as a working area.

[0032] The control unit 14 may be realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). A CPU, an MPU, an ASIC, and an FPGA can all be considered as controllers.

[0033] The control unit 14 includes a 3D reconstruction unit 141, a 2D segmentation processing unit 142, a segmentation result correction unit 143, and a label assignment unit 144. Each block constituting the control unit 14 (the 3D reconstruction unit 141, the 2D segmentation processing unit 142, the segmentation result correction unit 143, and the label assignment unit 144) is a functional block that indicates the function of the control unit 14.

[0034] These functional blocks may be software blocks or hardware blocks. For example, each of the above-mentioned functional blocks may be a software module implemented by software (including a microprogram), or may be a circuit block on a semiconductor chip (die). Of course, each functional block may be a processor or an integrated circuit. The functional blocks may be configured in any manner. Note that the control unit 14 may be configured by functional units different from the above-mentioned functional blocks.

[0035] The 3D reconstruction unit 141 is a processing unit that reconstructs the 3D model 1 from the group of 2D images. Specifically, the 3D reconstruction unit 141 reconstructs the 3D model 1 from the group of 2D images included in the input data 121 by using a known 3D model reconstruction method such as photogrammetry or LiDAR.

[0036] Note that if the input data 121 includes the 3D model 1, the functional configuration of the 3D reconstruction unit 141 may be unnecessary. For example, if photogrammetry or LiDAR is performed by an external device, and then the input data 121 including the 3D model 1 and a group of 2D images corresponding to the 3D model 1 is received from the external device, the 3D reconstruction unit 141 is unnecessary.

[0037] The 2D segmentation processing unit 142 is a processing unit that performs 2D segmentation on each of the 2D image groups. Specifically, as described above, the 2D segmentation processing unit 142 performs 2D segmentation by applying a known method, such as using a machine learning model such as a 2D segmentation-based model. The 2D segmentation processing unit 142 stores the 2D segmentation results 2 of each image included in the 2D image group as calculation data 122 in the storage unit 12.

[0038] The segmentation result correction unit 143 is a processing unit that corrects the 2D segmentation result 2 of each image included in the 2D image group based on the shape of the 3D model 1 (details will be described later in the operation example). The segmentation result correction unit 143 stores the corrected 2D segmentation result 3 in the storage unit 12 as calculation data 122.

[0039] The labeling unit 144 is a processing unit that performs segmentation of the 3D model 1 based on the corrected 2D segmentation result 3, thereby generating a segmented 3D model 4 in which a class based on the corrected 2D segmentation result 3 is assigned as a flag for each mesh or point cloud. Specifically, the labeling unit 144 generates the segmented 3D model 4 by reflecting the corrected 2D segmentation result 3 in the 3D model 1 using a known method such as the above-mentioned Segment Anything 3D.

[0040] 4 is a flowchart showing an example of the operation of the image processing device according to the embodiment of the present disclosure. As shown in FIG. 4, the image processing device 10 receives input of a group of 2D images via the communication unit 11 or the input / output unit 13 (S11). The control unit 14 stores the data of the input image group as input data 121 in the control unit 14.

[0041] Here, the image group is a group of photographs taken from various positions and degrees of accuracy of the object (target object) that is the subject of the 3D model 1. Note that the image group may also be a group of 2D images obtained by rendering the 3D model 1.

[0042] Next, the 3D reconstruction unit 141 reconstructs the 3D model 1 from the images included in the input data 121 based on a known reconstruction method such as photogrammetry or LiDAR (S12).

[0043] Next, the 2D segmentation processing unit 142 performs 2D segmentation on each image (2D image) included in the image group (S13). The 2D segmentation processing unit 142 stores the 2D segmentation result 2 obtained by this 2D segmentation in the control unit 14 as calculation data 122.

[0044] Next, the segmentation result correction unit 143 corrects the 2D segmentation result 2 of each image included in the image group based on the shape of the 3D model 1, thereby improving the accuracy of the 2D segmentation result 2 (S14). The segmentation result correction unit 143 stores the corrected 2D segmentation result 3, which has improved the accuracy of the 2D segmentation result 2, in the control unit 14 as calculation data 122.

[0045] The labeling unit 144 reads the corrected 2D segmentation result 3 included in the calculation data 122 and reflects it in the 3D model 1, thereby generating a 3D model 4 with segmentation (S15). The labeling unit 144 stores the generated 3D model 4 in the control unit 14 as the processing result 123.

[0046] 5 is a flowchart showing an example of a process for improving the accuracy of 2D segmentation. More specifically, FIG. 5 shows an example of the process related to S14 described above.

[0047] 5 , for each image (target image) included in the image group, the segmentation result correction unit 143 generates a depth map 5 corresponding to the shape of the 3D model 1 when viewed from the same camera position as the target image (S20). Specifically, the segmentation result correction unit 143 generates the depth map 5 based on the 3D model 1 by using well-known techniques in 3DCG and the like. The depth map 5 is also called a depth map, and represents the depth of field from the camera position for each pixel of the image.

[0048] Next, the segmentation result correction unit 143 clusters the 2D segmentation result 2 of the target image, i.e., the areas to be used as flag masks when reflecting the 3D model 4 with segmentation, according to the depth values ​​(distance) of the depth map 5 (S21).

[0049] Specifically, the segmentation result correction unit 143 clusters the 2D segmentation result 2 into clusters with similar depth values. As a result, the segmentation result correction unit 143 obtains a clustering result 6 in which the 2D segmentation result 2 is clustered into areas that are relatively close, medium distance, long distance, etc. from the camera.

[0050] 6 is an explanatory diagram showing an example of a 2D segmentation result 2 and a clustering result 6. Note that FIG. 6 illustrates only the result corresponding to the foreground tree included in the 2D segmentation result 2.

[0051] 6, in the 2D segmentation result 2, the segmentation 21 corresponding to a tree may also include buildings behind the tree. In contrast, in the clustering result 6, the segmentation 21 is clustered according to the shape (depth value) of the 3D model 1, so that it can be divided into a cluster 61 corresponding to the tree in the foreground and clusters 62, 63, and 64 corresponding to parts farther away from the tree (such as buildings in the background).

[0052] Next, the segmentation result correction unit 143 corrects the 2D segmentation result 2 for each image (target image) included in the image group based on the clustering result 6. Specifically, the segmentation result correction unit 143 determines, as the 2D segmentation result, the main components among the clusters 61 to 64 corresponding to the segmentation 21 included in the clustering result 6 (S22).

[0053] As an example, for the segmentation 21 corresponding to the tree described above, the segmentation result correction unit 143 sets the cluster 61 that occupies the largest area in the image as the 2D segmentation result. Note that the method of determining the main component is not limited to being based on area. For example, the segmentation result correction unit 143 may set, as the 2D segmentation result, one of the clusters 61 to 64 that is similar in shape to the flag name (tree, road, building, etc.) of the segmentation 21 as the main component.

[0054] 7 is an explanatory diagram showing an example of a 2D segmentation result, in which, like FIG. 6, only the result corresponding to the foreground tree is shown.

[0055] 7, in the 2D segmentation result 2 of the simple image group, segmentation 21 corresponding to the foreground tree contains noise such as buildings in the background. In contrast, in the corrected 2D segmentation result 3, segmentation 31 corresponding to the foreground tree matches the shape of the 3D model 1. Specifically, segmentation 31 does not contain noise such as buildings in the background.

[0056] Next, the labeling unit 144 assigns labels to the 3D model 1 based on the corrected 2D segmentation result 3 for the selected region of the 3D model 1 that corresponds to the target image (S23). As a result, the image processing device 10 obtains a 3D model 4 to which a semantics level (segmentation) has been assigned.

[0057] 8 is an explanatory diagram illustrating a comparison between the segmentation results of a 3D model according to an embodiment of the present disclosure and a conventional example. In FIG. 8, 3D models 4 and 7 are representations of the same landscape as models with segmentation. 3D model 4 is a model with segmentation according to an embodiment of the present disclosure. 3D model 7 is a model with segmentation according to a conventional example. A tree 101, a lamp 102, and a hedge 103 included in the landscape are arranged in the order of closest distance from the foreground (camera position): tree 101, lamp 102, hedge 103.

[0058] As shown in FIG. 8, in the 3D model 4 with segmentation, segmentation is performed in accordance with the shape of the 3D model 1, and the tree 101, the lamp 102, and the hedge 103 are colored differently without noise.

[0059] In contrast, the conventional 3D model 7 with segmentation is affected by small objects (for example, handrails) reflected in the 2D images, causing noise in the hedge 103 and the like.

[0060] (Variation) The image processing device 10 uses a 3D shape (e.g., depth information) to improve the accuracy of the 2D segmentation result 2. Therefore, the 3D model 1 to be used may be a combination of a 2D depth map and a 2D segmentation result.

[0061] Furthermore, the main components can be determined fully automatically by, for example, comparing the areas of each cluster and selecting the largest one, but it is also possible to consider an interaction where clusters are presented as candidates and the user selects the correct one. In this modified example, we will explain a case where the main components are determined by presenting and selecting the candidates to the user.

[0062] 9 is a flowchart showing an example of the operation of the image processing device 10 according to the modified example. As shown in Fig. 9, the image processing device 10 according to the modified example differs from the above-described embodiment in that it presents each mask candidate (clusters 61 to 64) to the user (S14a) and accepts the selection of each mask by the user (S14b).

[0063] Specifically, the segmentation result correction unit 143 presents the clusters 61 to 64 corresponding to the segmentation 21 included in the clustering result 6 to the user via the display screen of the input / output unit 13 (S14a). The user checks the clusters 61 to 64 displayed on the display screen and selects a cluster to be used as a major component, i.e., a cluster to be used as the 2D segmentation result (S14b). As a result, the segmentation result correction unit 143 sets the cluster selected by the user as the major component and uses it as the 2D segmentation result.

[0064] 10 is an explanatory diagram illustrating an example of a display screen. As shown in Fig. 10, a screen G1 includes a clustering result 6 and a 2D segmentation result 2. The user operates a cursor G11 on the screen G1 to select a cluster to be used as a main component from among clusters 61 to 64 included in the clustering result 6.

[0065] Furthermore, the screen G1 is configured to display the 2D segmentation result 2 together with the clustering result 6. Therefore, the user can refer to the 2D segmentation result 2 and select a cluster to be used as a main component from among the clusters 61 to 64.

[0066] The segmentation result correction unit 143 may be configured to present the clustering results to the user when the clustering results satisfy a predetermined condition. Here, the predetermined condition is set in advance in a memory or the like as a condition that requires presentation and confirmation to the user. As an example, a condition is set in advance that would make a cluster likely to be erroneously selected by a method of comparing the areas of the clusters and selecting the largest one, such as when the difference in area between the clusters is small.

[0067] In a configuration in which the clustering results are presented to the user when they satisfy certain conditions, the 2D segmentation results 2 for each image included in the image group do not have to be presented to the user and selected one by one, thereby reducing the burden on the user.

[0068] Furthermore, in the embodiment of the present disclosure, an example has been described in which the segmentation result correction unit 143 clusters the 2D segmentation results 2 based on depth values ​​corresponding to the shape of the 3D model 1, but the clustering of the 2D segmentation results 2 corresponding to the shape of the 3D model 1 is not limited to depth values. For example, the segmentation result correction unit 143 may generate a topological map corresponding to the shape of the 3D model 1 and perform clustering of the 2D segmentation results 2 based on this topological map. As an example, the segmentation result correction unit 143 may perform clustering of the 2D segmentation results 2 for portions surrounded by objects such as trees based on the topological map.

[0069] (Effect) As described above, the control unit 14 of the image processing device 10 obtains the 2D segmentation result 2 of the 2D image group corresponding to the 3D model 1. The control unit 14 of the image processing device 10 modifies the 2D segmentation result 2 based on the shape of the 3D model 1, and performs segmentation of the 3D model 1 based on the modified 2D segmentation result 3.

[0070] As a result, the image processing device 10 segments the 3D model 1 using the 2D segmentation result 3 that is matched to the shape of the 3D model 1, so that it is possible to obtain a segmentation result of the 3D model 1 with high accuracy.

[0071] Furthermore, the control unit 14 clusters the 2D segmentation results 2 of the 2D image group in accordance with the shape of the 3D model 1, and corrects the 2D segmentation results 2 of the 2D image group based on the clustering results.

[0072] As a result, the image processing device 10 can obtain a 2D segmentation result 3 corrected based on the clustering result corresponding to the shape of the 3D model 1 from the 2D segmentation result 2.

[0073] Furthermore, the control unit 14 clusters the 2D segmentation results 2 of the 2D image group in accordance with the depth of field indicated by the shape of the 3D model 1 .

[0074] By performing clustering according to the depth of field in this way, the image processing device 10 can obtain a 2D segmentation result 3 that is accurately corrected according to the depth.

[0075] In addition, the control unit 14 generates a depth map 5 corresponding to the shape of the 3D model 1 for each of the 2D image groups, and clusters the 2D segmentation results 2 of the 2D image groups based on the generated depth map 5.

[0076] This allows the image processing device 10 to perform clustering corresponding to the depth of field with high accuracy based on the depth map 5.

[0077] The control unit 14 also modifies the 2D segmentation result 2 of the 2D image group based on the area occupied by each clustering result in the image.

[0078] This allows the image processing device 10 to modify the 2D segmentation result 2 based on the area occupied by each clustering result image, and, for example, to eliminate noise with a small area in the 2D segmentation result 2 of a group of 2D images.

[0079] Furthermore, the control unit 14 determines the clustering result with the largest area as the corrected segmentation result.

[0080] As a result, the image processing device 10 can obtain a 2D segmentation result 3 that has been corrected so as to retain the characteristic result with the largest area.

[0081] Furthermore, the control unit 14 presents each of the clustering results to the user, and sets the clustering result selected by the user from the presented clustering results as the corrected segmentation result.

[0082] As a result, the image processing device 10 can obtain a 2D segmentation result 3 that has been corrected by presenting and selecting it for the user.

[0083] Furthermore, when presenting the clustering results, the control unit 14 also presents the segmentation results to the user.

[0084] This allows the user to select the clustering results while referring to the segmentation results.

[0085] Furthermore, if the clustering result satisfies a predetermined condition, the control unit 14 presents the clustering result to the user.

[0086] As a result, in the image processing device 10, if the clustering result satisfies a predetermined condition, the segmentation can be corrected by presenting it to the user and selecting it.

[0087] In the image processing device 10, the 3D model 1 is a model reconstructed in 3D based on a group of 2D images.

[0088] This allows the image processing device 10 to perform segmentation on the 3D model 1 reconstructed based on the group of 2D images.

[0089] In the image processing device 10 , the 2D image group is a group of photographs of an object corresponding to the 3D model 1 .

[0090] This allows the image processing device 10 to perform segmentation of the 3D model 1 based on a group of photographs of the object corresponding to the 3D model 1.

[0091] (Hardware Configuration Example) Information devices such as the image processing device 10 according to the above-described embodiments are realized by a computer 1000 configured as shown in Fig. 11, for example. Fig. 11 is a hardware configuration diagram showing an example of the computer 1000. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an SSD (Solid State Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0092] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the SSD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the SSD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0093] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0094] The SSD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the SSD 1400 is a recording medium that records the image processing program according to the present disclosure, which is an example of the program data 1450. The SSD 1400 may be another non-temporary recording medium, such as a hard disk drive (HDD).

[0095] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0096] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a touch panel, keyboard, mouse, microphone, and camera via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, and printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

[0097] For example, when the computer 1000 functions as the image processing device 10 according to the embodiment, the CPU 1100 of the computer 1000 executes a presentation program loaded onto the RAM 1200 to realize functions of a 3D reconstruction unit 141, a 2D segmentation processing unit 142, a segmentation result correction unit 143, a label assignment unit 144, etc. The image processing program according to the present disclosure and data in the storage unit 12 are stored in the SSD 1400. The CPU 1100 reads and executes program data 1450 from the SSD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0098] Note that the present technology can also be configured as follows. (1) An image processing method in which a computer executes the following processes: acquire segmentation results of a group of 2D images corresponding to a 3D model; correct the segmentation results of the group of 2D images based on the shape of the 3D model; and segment the 3D model based on the corrected segmentation results of the group of 2D images. (2) The image processing method described in (1), in which the correcting process clusters the segmentation results of the group of 2D images in correspondence with the shape of the 3D model and corrects the segmentation results of the group of 2D images based on the clustering results. (3) The image processing method described in (2), in which the correcting process clusters the segmentation results of the group of 2D images in correspondence with the depth of field indicated by the shape of the 3D model. (4) The image processing method according to (3), wherein the correction process generates a depth map corresponding to the shape of the 3D model for each of the 2D images, and clusters the segmentation results of the 2D images based on the generated depth map. (5) The image processing method according to (2), wherein the correction process corrects the segmentation results of the 2D images based on the area occupied by each of the clustering results in the image. (6) The image processing method according to (5), wherein the correction process sets the clustering result with the largest area as the corrected segmentation result. (7) The image processing method according to any one of (2) to (6), wherein the computer further performs a process of presenting each of the clustering results to a user, and setting a clustering result selected by the user from the presented clustering results as the corrected segmentation result. (8) The image processing method according to (7), wherein the computer further performs a process of presenting the segmentation result to the user when presenting the clustering result. (9) The image processing method according to (7), wherein the computer presents the clustering result to the user when the clustering result satisfies a predetermined condition.(10) The image processing method according to any one of (1) to (9), wherein the 3D model is a model reconstructed in 3D based on the group of 2D images. (11) The image processing method according to any one of (1) to (10), wherein the group of 2D images is a group of photographs of an object corresponding to the 3D model. (12) An image processing device comprising: a control unit that acquires segmentation results of the group of 2D images corresponding to a 3D model, corrects the segmentation results of the group of 2D images based on a shape of the 3D model, and performs segmentation of the 3D model based on the corrected segmentation results of the group of 2D images. (13) An image processing program that causes a computer to execute processes of acquiring segmentation results of the group of 2D images corresponding to a 3D model, correcting the segmentation results of the group of 2D images based on the shape of the 3D model, and segmenting the 3D model based on the corrected segmentation results of the group of 2D images.

[0099] DESCRIPTION OF SYMBOLS 1, 4, 7... 3D model 2, 3... 2D segmentation result 5... Depth map 6... Clustering result 10... Image processing device 11... Communication unit 12... Memory unit 13... Input / output unit 14... Control unit 21, 31... Segmentation 61 to 64... Cluster 101... Tree 102... Light 103... Hedge 121... Input data 122... Calculation data 123... Processing result 141... 3D reconstruction unit 142... 2D segmentation processing unit 143... Segmentation result correction unit 144... Label assignment unit 1000... Computer 1050... Bus 1100... CPU 1200... RAM 1300... ROM 1400... SSD 1450... Program data 1500... Communication interface 1550... External network 1600... Input / output interface 1650... Input / output device G1... Screen G11... Cursor

Claims

1. An image processing method in which a computer performs the following processes: obtain segmentation results of a group of 2D images corresponding to a 3D model; modify the segmentation results of the group of 2D images based on the shape of the 3D model; and segment the 3D model based on the modified segmentation results of the group of 2D images.

2. The image processing method according to claim 1, wherein the correction process clusters the segmentation results of the group of 2D images in accordance with the shape of the 3D model, and corrects the segmentation results of the group of 2D images based on the clustering results.

3. The image processing method according to claim 2, wherein the correction process clusters the segmentation results of the group of 2D images in accordance with the depth of field indicated by the shape of the 3D model.

4. The image processing method of claim 3, wherein the modifying process generates a depth map corresponding to the shape of the 3D model for each of the 2D images, and clusters the segmentation results of the 2D images based on the generated depth map.

5. The image processing method according to claim 2, wherein the modifying process modifies the segmentation results of the 2D images based on the area occupied by each of the clustering results in the image.

6. The image processing method according to claim 5, wherein the correction process uses the clustering result with the largest area as the corrected segmentation result.

7. The image processing method according to claim 2, further comprising the step of: presenting each of the clustering results to a user; and selecting a clustering result selected by the user from the presented clustering results as the corrected segmentation result.

8. The image processing method according to claim 7, further comprising the step of: presenting the segmentation result to the user together with the clustering result.

9. The image processing method according to claim 7, wherein the computer presents the clustering result to the user if the clustering result satisfies a predetermined condition.

10. The image processing method according to claim 1, wherein the 3D model is a model reconstructed in 3D based on the group of 2D images.

11. The image processing method according to claim 1, wherein the group of 2D images is a group of photographs of an object corresponding to a 3D model.

12. An image processing device comprising: a control unit that acquires segmentation results of a group of 2D images corresponding to a 3D model; modifies the segmentation results of the group of 2D images based on the shape of the 3D model; and segments the 3D model based on the modified segmentation results of the group of 2D images.

13. An image processing program that causes a computer to perform the following processes: obtain segmentation results of a group of 2D images corresponding to a 3D model; modify the segmentation results of the group of 2D images based on the shape of the 3D model; and segment the 3D model based on the modified segmentation results of the group of 2D images.

Images