Three-dimensional model editing system and method

Abstract

This three-dimensional model editing system 11 comprises: a three-dimensional model acquisition unit 21 that acquires a plurality of three-dimensional models including a mesh or a point group; an interface 22 that receives a user's editing instruction for a target three-dimensional model to be edited among the plurality of three-dimensional models; and a three-dimensional model editing unit 23 that extracts feature quantities of the plurality of three-dimensional models, adjusts a feature quantity of the target three-dimensional model on the basis of the extracted feature quantities and the editing instruction, and edits the target three-dimensional model on the basis of the adjusted feature quantity.

Description

3D model editing system and method

[0001] The present invention relates to a three-dimensional model system and method.

[0002] 3D models are primarily used in industry and the entertainment industry to represent or simulate the shape and appearance of objects in three-dimensional space. In these industries, 3D CAD models and data formats such as point clouds are used using technologies such as 3D Computer Aided Design (3DCAD) software and LiDAR (Light Detection and Ranging). With the rise of not only industrial design and maintenance but also businesses such as the Metaverse and digital twins, the efficiency and cost reduction of 3D data creation are also gaining attention.

[0003] Furthermore, it is expected that the demand for utilizing 3D data accumulated at each site will increase. In recent years, generative artificial intelligence (AI) has been active in areas such as text and images. Although generative AI that handles 3D models is still in the development stage, considering the utilization of 3D data in fields such as the metaverse and its ability to create new data, it is expected that it will attract more attention and develop further in the future.

[0004] At sites where 3D data is being accumulated, it is common for 3D data with similar shapes and uses to be repeatedly created. Therefore, it is necessary to either edit and use existing 3D models, or to have 3D generation AI learn from existing 3D models to generate new 3D models.

[0005] Patent Literature 1 discloses a three-dimensional model editing system for editing three-dimensional models. This three-dimensional model editing system receives a textured three-dimensional model as input and acquires feature points that represent the characteristics of the three-dimensional model and feature points that represent the characteristics of the three-dimensional model on a two-dimensional image of the three-dimensional model viewed from a predetermined viewpoint. The three-dimensional model editing system then uses these feature points to match corresponding feature points and deforms the three-dimensional model to match the edited two-dimensional image. The edited two-dimensional image is then projected onto the deformed three-dimensional model to apply a texture.

[0006] Similarly, Non-Patent Document 1, which allows editing of 3D models, is premised on generating new 3D models, is unable to handle existing 3D models, and furthermore, input is limited to sketches.

[0007] Japanese Patent Application Laid-Open No. 2022-28090

[0008] Guillard, Benoit et al. “Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches.” 2021 IEEE / CVF International Conference on Computer Vision (ICCV) (2021): 13003-13012.

[0009] Patent Document 1 aims to facilitate editing of 3D models of subjects captured using multiple image capture devices. The user must edit the rendered image of the 3D model and reflect changes in feature points in the edited image in the 3D model. Therefore, the method relies on the presence of texture, which makes it easy to capture feature points in 3D and 2D space, and is difficult to apply to 3D CAD models that do not have texture. Furthermore, the method requires precise editing of the 2D image by the user, which makes editing time-consuming.

[0010] In Non-Patent Document 1, a 3D model can be generated using a sketch, but other inputs are not possible, and editing of existing 3D models is not covered. Furthermore, inputs are limited to sketches, and the intended use is generation, not editing.

[0011] Therefore, an object of the present invention is to provide a technique for easily editing a three-dimensional model.

[0012] Therefore, the present invention has been made in consideration of the above-mentioned problems, and comprises an acquisition unit that acquires multiple 3D models including meshes or point clouds, an interface that accepts user editing instructions for a target 3D model to be edited among the multiple 3D models, and an editing unit that extracts feature amounts of the multiple 3D models, adjusts the feature amounts of the target 3D model based on the extracted feature amounts and the editing instructions, and edits the target 3D model based on the adjusted feature amounts.

[0013] According to the present invention, a three-dimensional model can be easily edited.

[0014] 1 is a block diagram showing an example of the configuration of a three-dimensional model editing system according to Example 1. FIG. 2 is a flowchart showing an example of a three-dimensional model editing process according to Example 1. FIG. 3 is a flowchart showing a feature amount adjustment process according to Example 1. FIG. 4 is a diagram showing an example of a user interface according to Example 1. FIG. 5 is a flowchart showing an example of an editing process according to Examples 2 and 3. FIG. 6 is a flowchart showing an example of a sketch acquisition process according to Example 4. FIG. 7 is a block diagram showing an example of a user editing instruction process according to Example 4.

[0015] Hereinafter, several embodiments will be described in detail with reference to the drawings.

[0016] FIG. 1 is a block diagram illustrating an example of the configuration of a three-dimensional model editing system according to a first embodiment.

[0017] 1, a 3D model editing system 11 executes a 3D model editing process. The 3D model editing system 11 includes a 3D model acquisition unit 21 as an example of an "acquisition unit," a user interface 22 as an example of an "interface," a 3D model editing unit 23 as an example of an "editing unit," and a 3D model output unit 24.

[0018] The three-dimensional model editing system 11 further includes at least one processor 101 and at least one memory 102. The processor 101 reads and executes predetermined computer programming stored in the memory 102 to realize the functions described below.

[0019] The 3D model acquisition unit 21 reads, for example, a 3D model acquired by scanning using 3D CAD or LiDAR, which are design documents. The 3D model acquisition unit 21 can read formats for saving 3D models that are commonly used in industry. The 3D model acquisition unit 21 outputs the acquired 3D model to the user interface 22. At the same time, the 3D model acquisition unit 21 performs preprocessing on the read 3D model. For example, the 3D model acquisition unit 21 samples the 3D model surface, acquires a point cloud, and normalizes the point cloud coordinates. The 3D model acquisition unit 21 outputs the processed 3D model to the 3D model editing unit 23.

[0020] The user interface 22 includes a communication interface and a user interface. The user interface 22 places the 3D model input from the 3D model acquisition unit 21 in a three-dimensional space. The user interface 22 also has a camera that allows the user to observe the 3D model. The camera accepts user operations. The user can sketch editing instructions on a two-dimensional projection of the observed 3D model while changing the viewpoint by operating the camera, for example, on a two-dimensional line drawing-style image of the target 3D model acquired from the same viewpoint. The sketch may be a two-dimensional monochrome image. The user interface 22 outputs the acquired user editing instructions and viewpoint information to the 3D model editing unit 23.

[0021] The three-dimensional model editing unit 23 processes the three-dimensional model acquired by the three-dimensional model acquisition unit 21, for example, and edits the three-dimensional model based on user editing instructions input from the user interface 22. Furthermore, the three-dimensional model editing unit 23 outputs the edited three-dimensional model again to the user interface 22, or outputs it as a final result via the three-dimensional model output unit 24, depending on the user's decision.

[0022] As shown in FIG. 1, the three-dimensional model editing unit 23 includes a user editing instruction processing unit 31, a feature adjustment unit 32, a two-dimensional image rendering unit 33, a three-dimensional model feature calculation unit 34 as an example of a “feature calculation unit”, and a three-dimensional model generation unit 35.

[0023] The user edit instruction processor 31 receives a 2D monochrome image edited by the user and a 2D projection image of the 3D model from a selected viewpoint. Based on these images, the user edit instruction processor 31 calculates the area edited by the user. The user edit instruction processor 31 uses the calculated area as a mask to restrict the feature amount adjustment unit 32 (described later) so that the 3D model outside the edited area remains unchanged.

[0024] The three-dimensional model feature amount calculation unit 34 calculates the feature amounts of the three-dimensional model after the three-dimensional model acquisition unit 21 has performed preprocessing, and outputs the calculated feature amounts to the feature amount adjustment unit 32 .

[0025] The 3D model generation unit 35 generates a 3D model whose shape matches the original 3D model based on the feature values ​​calculated by the 3D model calculation unit 34. The generated 3D model accurately reflects the class information and semantic information of the target 3D model. For example, it can reflect features such as the fact that a car has four wheels. Furthermore, the 3D model generation unit 35 can generate a new 3D model based on the feature values ​​adjusted by the feature value adjustment unit 32. Here, the 3D model may be an explicit shape representation such as a mesh or point cloud, or an implicit function representation such as an SDF (Signed Distance Function).

[0026] The 3D model feature calculation unit 34 and the 3D model generation unit 35 constitute a 3D generation AI. The 3D generation AI may be, for example, an autoencoder that handles 3D data. In addition to the 3D model, class information and caption data associated with the 3D model are used to train the 3D generation AI. If the training data includes class information, the trained 3D model feature calculation unit 34 can also function as a 3D model classification mechanism. The 3D generation AI is not limited to a specific structure or architecture.

[0027] The feature amount adjustment unit 32 optimizes the feature amounts based on a sketch, which is a user editing instruction. The feature amount adjustment unit 32 outputs the feature amounts to the 3D model generation unit 35. The 3D model generation unit 35 generates a 3D model based on the input feature amounts. The 3D model generation unit 35 then inputs the generated 3D model to the 2D image rendering unit 33. The 2D image rendering unit 33 generates a contour image of the generated 3D model from the same viewpoint as the viewpoint selected by the user.

[0028] Based on the new contour image, the feature adjustment unit 32 calculates the difference between the new contour image and the sketch, which is the user's editing instruction, and adjusts the feature in a direction that reduces the difference. The 3D model generation unit 35 generates a new 3D model based on the adjusted feature. The feature adjustment unit 32, the 2D image rendering unit 33, and the 3D model generation unit 35 repeat the above-mentioned process multiple times using the new 3D model, and the feature adjustment unit 32 outputs the adjusted feature. The 3D model generation unit 35 generates a final 3D model based on the adjusted feature, places it on the user interface 22 again, and waits for the user's next editing operation. If the current editing result is the final result, the editing result is output to the 3D model output unit 24.

[0029] FIG. 2 is a flowchart illustrating an example of a three-dimensional model editing process according to the first embodiment.

[0030] The three-dimensional model acquisition unit 21 receives input of a three-dimensional model (S101). For example, the three-dimensional model acquisition unit 21 reads the three-dimensional model input by the user via the user interface 22 by operating a client program (the three-dimensional model acquisition unit 21 and the user interface 22).

[0031] Next, the three-dimensional feature amount calculation unit 34 calculates the feature amounts of the input three-dimensional model, and places the three-dimensional model on the user interface 22 (S102).

[0032] Next, the user operates the user interface 22 to select a viewpoint from which editing is desired, draw a sketch on the two-dimensional projection of the target three-dimensional model, and input editing instructions (S103).

[0033] Next, in step S103, the user editing instruction processing unit 31 acquires a sketch created by the user as a mask for restricting the editing area. The sketch and mask, which are the user editing instructions, are input to the feature amount adjustment unit 32. The feature amount adjustment unit 32 optimizes the feature amounts based on the processed sketch and mask (S104). Details of the feature amount adjustment process will be described later with reference to FIG. 3. The feature amount adjustment unit 32 also outputs a three-dimensional model, which is an intermediate result, as necessary.

[0034] Next, the feature amounts optimized in step S103 are input to the 3D model generation unit 35 to generate the final result, and the 3D model generation unit 35 outputs the generated 3D model as the editing result (S105).

[0035] Next, the user interface 22 determines whether the user has confirmed the editing results and whether to end the editing (S106). If the determination result in step S106 is false (S106: NO), a new 3D model is set as the new target for editing, and the process returns to step S102. If the determination result in step S106 is true (S106: YES), the 3D model generation unit 35 outputs the last output 3D model as the final result to the 3D model output unit 24, and the 3D model output unit 24 outputs the final result in the form of a mesh, a point cloud, or the like.

[0036] Next, the three-dimensional model editing process is completed, and the processing steps of the three-dimensional model editing system 11 are finished.

[0037] FIG. 3 is a flowchart illustrating the feature amount adjustment process according to the first embodiment.

[0038] The feature amount adjustment unit 32 receives the sketch and viewpoint information from the user editing instruction processing unit 31 (S21). The feature amount adjustment unit 32 adjusts the feature amount based on the user sketch and viewpoint.

[0039] Specifically, the 2D image rendering unit 33 renders a 2D contour image of the 3D model based on the viewpoint information (S22). The 2D image rendering unit 33 calculates the difference between the rendered contour image and the user sketch (S23). For example, the inter-image error is calculated using a mean squared error (MSE) that calculates the distance between images, or a 2D chamfer distance that can calculate the distance between monochrome images. The feature adjustment unit 32 adjusts the feature based on the calculated inter-image error.

[0040] The feature adjustment unit 32 can adjust the features using an optimization method used in deep learning, etc. For example, Adam or Stochastic Gradient Descent (SGD) is used to optimize the features based on the inter-image error (S24).

[0041] Furthermore, the feature amount adjustment unit 32 processes the images used to calculate the inter-image error based on the area mask calculated by the user editing instruction processing unit 31, and excludes areas not to be edited from the error calculation. This makes it possible to limit the editing area to only the area specified by the user.

[0042] The optimized feature quantities are input to the 3D model generation unit 35 to generate a new 3D model. The 3D model generation unit 35 generates a new 3D model based on the adjusted feature quantities (S25). The 3D model generation unit 35 outputs the generated 3D model to the 2D image rendering unit 33. Typically, steps S21 to S25 are repeated until a predetermined optimization termination condition is met. The termination condition for the optimization process can be determined as appropriate, for example, by manually terminating the optimization through visual confirmation by the user, or by automatically terminating the optimization process after a predetermined number of optimization iterations. Furthermore, the 3D model generated during the optimization process for rendering a 2D contour image can be output as an intermediate result in the form of a mesh or a point cloud, as needed.

[0043] In response to the input, the three-dimensional model output unit 24 outputs data including three-dimensional model data in a data storage format designated by the user.

[0044] The 2D image rendering unit 33 renders the contour of the input 3D model and outputs a 2D image. Here, the input 3D model matches the format of the output of the 3D model generation unit 35. For example, when an SDF is output, the 2D image rendering unit 33 receives the SDF as input and renders a 2D image.

[0045] FIG. 4 is a diagram illustrating an example of a user interface according to the first embodiment.

[0046] The user interface 22 displays user settings 41, intermediate results 42, an operation area 43, and edit result candidates 44.

[0047] The user settings 41 accepts settings to be used in the optimization process, for example, parameters and setting terms such as constraints and termination conditions.

[0048] The intermediate result 42 is available as needed. The intermediate result 42 is an area that displays a 3D model generated as an intermediate result by the 3D model generation unit 35 based on the feature quantities output during the optimization process.

[0049] The operation area 43 allows the user to operate the viewpoint and check the target 3D model or the resulting 3D model. The operation area 43 also allows the user to fix the viewpoint and give editing instructions by sketching as shown in the figure.

[0050] A three-dimensional model 51 and a user sketch 52 are displayed in the operation area 43. The three-dimensional model 51 is either a three-dimensional model acquired by the three-dimensional model acquisition unit 21 or a three-dimensional model after editing has been completed.

[0051] The user sketch 52 is an editing instruction that indicates what kind of editing the user wants to perform on the 3D model that is the editing target. For example, the user sketch 52 is sketched by imagining the outline and wheel positions after editing to shorten the length of the car shown in Figure 4.

[0052] The editing result candidate 44 is an area for displaying a plurality of editable editing results when the same editing instruction is used to generate the edited results.

[0053] The user can use the user interface 22 shown in FIG. 4 to perform editing operations on the three-dimensional model.

[0054] According to this configuration, the 3D model editing system 11 includes a 3D model acquisition unit 21, an interface 22, and a 3D model editing unit 23. The 3D model acquisition unit 21 acquires multiple 3D models including meshes or point clouds. The interface 22 accepts user editing instructions for a target 3D model to be edited among the multiple 3D models. The 3D model editing unit 23 extracts feature amounts of the multiple 3D models, adjusts the feature amounts of the target 3D model based on the extracted feature amounts and the editing instruction, and edits the target 3D model based on the adjusted feature amounts.

[0055] This allows users to easily edit 3D models through simple editing input, improving editing efficiency. For example, when editing a 3D model of a car, if you want to change the position of a wheel, you can simply sketch the new wheel position and shape within the selected viewpoint, and the other paired wheel will also move to the new position and change its shape.

[0056] The user's viewpoint with respect to the target three-dimensional model and a sketch from this viewpoint are input as editing instructions to the interface 22. This allows for user operability and simplifies editing.

[0057] The 3D model editing unit 23 includes a feature amount adjustment unit 32 that adjusts the feature amounts of the target 3D model based on the difference between the sketch at the viewpoint and the contour of the target 3D model, and a 3D model generation unit 35 that regenerates the target 3D model based on the feature amounts of the target 3D model adjusted by the feature amount adjustment unit 32. This makes it possible to easily regenerate the 3D model.

[0058] The plurality of 3D models are the class information and rendering images of these 3D models, which allows a new 3D model to be appropriately generated.

[0059] The three-dimensional model editing unit 23 identifies an edited area in the target three-dimensional model based on the difference between the sketch at the viewpoint and the two-dimensional projected image of the target three-dimensional model at the viewpoint, and sets areas other than the identified edited area as an edit-prohibited area. This makes it possible to determine whether editing is necessary for each area.

[0060] A three-dimensional model editing system 11 according to the second embodiment will be described with reference to FIG.

[0061] In the second embodiment, when the user operates the viewpoint on the user interface 22 to sketch editing instructions in step S104 of the first embodiment, the possibility of unexpected changes occurring in the shape of the editing result observed from other viewpoints due to editing from a single viewpoint is reduced. The sketch input process, which is the main change, will be explained while showing it in FIG. 5.

[0062] FIG. 5 is a flowchart illustrating an example of an editing process according to the second and third embodiments.

[0063] As shown in FIG. 5 , in step S103, the user selects a viewpoint and creates a sketch. Because a single viewpoint is input, in the next step S201, the 3D model editing system 11 prompts the user to select a different viewpoint and provide editing instructions from that viewpoint. The number of user editing instructions required by the 3D model editing system 11, in this case the number of sketches and the number of viewpoint information, is a preset number. For example, the number of user editing instructions is set by the user in the user setting section 41 shown in FIG. 4 , which is an example of the user interface 22. In step S202, the 3D model editing system 11 monitors the number of user editing instructions and determines whether the set number has been reached. If the determination result in step S202 is true (S202: YES), the process proceeds to step S104. If the determination result in step S202 is false (S202: NO), the process returns to step S201 and provides feedback to the user to continue requesting editing instructions from a different viewpoint.

[0064] In step S104, the feature amount adjustment unit 32 optimizes the feature amounts, as described in the first embodiment. However, although the user editing instruction processing unit 31 performs processing before the user editing instruction is input to the feature amount adjustment unit 32, the use of sketches from multiple viewpoints ensures consistency of the area outside the editing area before and after editing. Therefore, the processing by the user editing instruction processing unit 31 can be skipped.

[0065] The feature adjustment unit 32 optimizes the feature values ​​based on the multiple sketches and viewpoint information. The optimization process corresponds to the steps shown in FIG. 3. However, in the calculation of the image differences in S22 to S24, it is necessary to calculate the differences between the rendered line drawings from multiple viewpoints and the user-input sketches. The multiple calculation results are integrated, for example, by calculating their sum. Furthermore, the user may set, for each viewpoint, a priority (weight) for the reflection of editing instructions from each viewpoint in the editing results in the user setting area 41 of the user interface 22. In this way, when integrating the multiple calculation results, by multiplying the calculation results from each viewpoint by a weight and adding them up, the editing instructions from the viewpoint that the user most wants to reflect can be prioritized.

[0066] By making such a change, it is possible to reduce the possibility that a shape change that is not intended by the user occurs in the edited result viewed from a viewpoint different from the designated edit viewpoint due to single viewpoint editing.

[0067] A three-dimensional model editing system 11 according to the third embodiment will be described with reference to FIGS.

[0068] In the third embodiment, when the user operates the viewpoint on the user interface 22 to sketch editing instructions in step S103 of the second embodiment, editing from only a single viewpoint reduces the possibility of unexpected changes occurring in the shape of the editing result observed from other viewpoints. The main changes from the second embodiment will be described with reference to Figs. 1 and 5.

[0069] In FIG. 1 , the user editing instruction processing unit 31 that processes user input calculates the editing area. However, this step is omitted in Example 2 because multiple viewpoints are input. Calculation of the editing area is also omitted in Example 3. For a single viewpoint input from the user, existing image generation AI, such as stable diffusion or Zero123 based on it, is used to generate sketch images of multiple other viewpoints. Then, the sketch images and viewpoint information for each viewpoint are used for feature adjustment in the feature adjustment unit 32.

[0070] By making such a change, even if the user inputs only an editing instruction for a single viewpoint in the second embodiment, the input conditions of the second embodiment can be met by generating editing instructions for multiple viewpoints.

[0071] A three-dimensional model editing system 11 according to a fourth embodiment will be described with reference to FIGS.

[0072] In Example 4, when the user inputs an ambiguous input instruction in Examples 1, 2, and 3, for example, when editing a car with four doors, there is no explicit editing image, and the user inputs the edited image in a text prompt, such as "a car with two doors."

[0073] FIG. 6 is a flowchart illustrating an example of a sketch acquisition process according to the fourth embodiment.

[0074] 6, in the sketch acquisition process, it is determined whether the user has input a vague input, for example, a text prompt (S311). If the determination result of step S311 is true (S311: YES), the 3D model editing system 11 proceeds to a dedicated processing flow S312. If the determination result of step S311 is false (S311: NO), it proceeds to a normal processing S315. The normal processing S315 is the processing executed in Examples 1, 2, and 3, for example, S103.

[0075] In S312, the three-dimensional model editing system 11 calls the image generation model 53 (see FIG. 7), which is a mechanism that allows editing using text prompts. The image generation model 53 is a mechanism that can generate two-dimensional images using text prompts, such as stable diffusion. The processing flow shown in FIG. 6 is explained below.

[0076] First, when the user operates the user interface 22 to input prompt text, the three-dimensional model editing system 11 determines the type of input. That is, the three-dimensional model editing system 11 determines whether the input is prompt text (S311). If the determination result in step S311 is true (S311: YES), the process proceeds to S312.

[0077] Next, the user operates the 3D model and selects a viewpoint for editing via the user interface 22. The 3D model editing system 11 acquires a 2D projection image of the 3D model as seen from the selected viewpoint (S312).

[0078] Next, the 3D model editing system 11 calls the image generation model 53 using the modified user editing instruction processing unit 31, and generates a possible edited image using the text prompt and the 2D image acquired in the previous step S312 (S313). Note that the generated image is identical to the original 2D image except for the edited parts.

[0079] Next, a sketch, which is a contour image, is extracted from the edited image generated in the previous step S313 (S314). The sketch is extracted using an image processing method such as a Sobel filter.

[0080] On the other hand, if the determination result in step S311 is false (S311: NO), the acquired sketch is output to the feature amount processing unit 32, and the process proceeds to the next processing stage (S315).

[0081] FIG. 7 is a block diagram illustrating an example of a user edit instruction process according to the fourth embodiment.

[0082] When user input text 51 is input, user edit instruction processing unit 31 calculates prompt features using an image generation model 53, for example, stable diffusion.

[0083] Next, the user editing instruction processor 31 inputs the calculated prompt features and the rendered image 52 acquired in the previous step into the image generation model 53 to regenerate an image 55. The user editing instruction processor 31 adjusts the prompt features while calculating the reconstruction loss of the image based on the regenerated image 55 and the rendered image 52. Here, the reconstruction loss is a function that calculates the inter-image loss. For example, the reconstruction loss is the mean squared error (MSE).

[0084] After completing the feature adjustment, the user editing instruction processing unit 31 fine-tunes the image generation model 53 based on the adjusted feature 56 so that it can generate an edited image. In this process, the adjusted feature 56 is fixed, the image generation model 53 is made trainable, and the image is regenerated based on the adjusted feature 56 and the rendered image 52. The user editing instruction processing unit 31 fine-tunes the image generation model 53 based on the regenerated image 57 and the image reconstruction loss of the rendered image 52.

[0085] Finally, the user editing instruction processing unit 31 fuses the adjusted features 56 and the prompt features 54 with different weights, and generates a predictive edited image 59 via an adjusted model 58. In the process up to this point, a plurality of different predictive edited images 59 can be output by selecting the weights.

[0086] The image editing process of the modified user editing instruction processing unit 31 can also be replaced with an editing method that uses image generation AI such as DragDiffusion or MasaCtrl.

[0087] Furthermore, the rendering image 52 can also be obtained directly as a line drawing. Therefore, step S314 can be omitted due to the editing effect of the predictive editing image 59.

[0088] By making such changes, even if the user's editing purpose is unclear, it is possible to express the image in text, use the image generation AI to generate multiple 2D images that are editing instructions, and edit the 3D model based on each. Therefore, by outputting multiple mutually different 3D model editing results, feedback can be provided to the user, providing hints for editing and creating 3D models and improving creativity.

[0089] The present invention is not limited to the above-described embodiments, but includes other variations in which the inputs and constraints used for optimization are different. The 3D model system 11 including the user interface described in this specification is a collection of multiple functions and is not limited to the configuration shown in the drawings. For example, the 3D model acquisition unit 21 may be implemented as part of the functions of the user interface 22. The processing steps in each processing process do not need to be performed in order, as long as no contradictions arise.

[0090] 11... 3D model editing system, 21... 3D model acquisition unit, 22... user interface, 23... 3D model editing unit, 32... feature amount adjustment unit, 35... 3D model generation unit, 59... predicted edited image

Claims

1. A 3D model editing system comprising: an acquisition unit that acquires multiple 3D models including meshes or point clouds; an interface that accepts user editing instructions for a target 3D model to be edited among the multiple 3D models; and an editing unit that extracts features of the multiple 3D models, adjusts the features of the target 3D model based on the extracted features and the editing instructions, and edits the target 3D model based on the adjusted features.

2. The 3D model editing system according to claim 1, wherein a user's viewpoint with respect to the target 3D model and a sketch from that viewpoint are input to the interface as the editing instructions.

3. The 3D model editing system of claim 2, wherein the editing unit comprises: a feature amount adjustment unit that adjusts the feature amounts of the target 3D model based on the difference between the sketch and the contour of the target 3D model at the viewpoint; and a 3D model generation unit that regenerates the target 3D model based on the feature amounts of the target 3D model adjusted by the feature amount adjustment unit.

4. The three-dimensional model editing system according to claim 1, wherein the plurality of three-dimensional models are class information and rendering images of the three-dimensional models.

5. The 3D model editing system according to claim 2, wherein the editing unit identifies an edited area in the target 3D model based on the difference between the sketch at the viewpoint and the 2D projected image of the target 3D model at the viewpoint, and designates areas other than the identified edited area as areas where editing is prohibited.

6. The three-dimensional model editing system of claim 2, wherein the editing instructions are text prompts, and the editing unit generates a plurality of predicted edited images by editing a rendering image at the viewpoint based on the text prompts, and adjusts the feature quantities of the target three-dimensional model based on the generated plurality of predicted edited images.

7. A three-dimensional model editing system as described in claim 2, wherein a plurality of viewpoints of the target three-dimensional model and sketches from the plurality of viewpoints are input to the interface as the editing instructions, and the editing unit adjusts the feature quantities of the target three-dimensional model based on differences between the sketches from each of the plurality of viewpoints and the contour of the target three-dimensional model.

8. A three-dimensional model editing system as described in claim 2, wherein a single viewpoint of the target three-dimensional model and a sketch at that viewpoint are input to the interface as the editing instructions, and the editing unit adjusts the feature quantities of the target three-dimensional model based on the difference between the sketch of the viewpoint and the contour of the target three-dimensional model.

9. The 3D model editing system according to claim 1, wherein the editing unit regenerates the target 3D model based on the feature quantities of the target 3D model, and the interface displays the regenerated target 3D model as an intermediate result.

10. The 3D model editing system according to claim 1, wherein the acquisition unit acquires the plurality of 3D models and class features and semantic features associated with the plurality of 3D models as learning data, and the editing unit extracts features of each of the plurality of 3D models, learns the extracted features, and adjusts the features of the target 3D model based on the learned features and the editing instructions, and edits the target 3D model based on the adjusted features.

11. A three-dimensional model editing method using a three-dimensional model editing system, comprising the steps of: acquiring a plurality of three-dimensional models including a mesh or a point cloud; receiving a viewpoint for a target three-dimensional model to be edited from the plurality of three-dimensional models and a sketch at that viewpoint; extracting feature amounts for each of the plurality of three-dimensional models, and adjusting the feature amounts of the target three-dimensional model based on the difference between the sketch at the viewpoint and the contour of the target three-dimensional model; and editing the target three-dimensional model based on the adjusted feature amounts of the target three-dimensional model.

Images