Method, device, medium and program product for modifying a three-dimensional model

Abstract

The application provides a three-dimensional model modification method, device, medium and program product. It relates to the technical field of computer image technology. The method comprises the following steps: generating a modification guide vector and a geometry guide vector; generating a modification geometry hidden vector based on the modification guide vector and the geometry guide vector, and generating an initial three-dimensional model based on the modification geometry hidden vector; sampling and encoding the surface of the initial three-dimensional model according to a first sampling mode to obtain a first geometry hidden vector, sampling and encoding the surface of a three-dimensional base model according to the first sampling mode to obtain a second geometry hidden vector, and performing smoothing fusion processing on the first geometry hidden vector and the second geometry hidden vector based on a region to be modified to obtain a third geometry hidden vector; and generating a modified three-dimensional model according to the third geometry hidden vector. The application is used to provide a technical solution capable of responding to a text instruction and improving the fusion effect of a region to be modified and an original model.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to a method, device, medium and program product for modifying a three-dimensional model. Background Technology

[0002] With the widespread application of 3D modeling technology in fields such as industrial design, games and film, and architectural decoration, the demand for local modifications to 3D models is increasing. Users expect to be able to quickly and accurately modify specified areas of the 3D base model while preserving the original geometric features of the non-modified areas of the 3D base model, ensuring the overall coordination and integrity of the modified model.

[0003] Current methods for modifying 3D models largely rely on specialized modeling software, requiring operators to manually select areas and adjust parameters. This approach is technologically advanced, inefficient, and difficult for non-professionals to master. Some existing solutions attempt to drive modifications through commands, but these fail to effectively integrate text commands, the original 3D model, and the selected area, making it difficult to accurately predict the outcome of text-based modifications to the selected area.

[0004] Such solutions generally have two major drawbacks: First, they cannot accurately respond to text commands, and the modification results deviate significantly from the text descriptions; second, the selected area after modification has poor integration with the remaining area of ​​the original model, which easily leads to problems such as abrupt transitions and discontinuities, and it is difficult to fully preserve the inherent geometric features of the non-selected areas of the original model, thus damaging the overall coordination of the model. Summary of the Invention

[0005] In view of the deficiencies of the prior art, this application provides a method, device, medium and program product for modifying a three-dimensional model, so as to provide a technical solution that can accurately respond to text commands and improve the fusion effect between the area to be modified and the remaining area of ​​the original model.

[0006] Firstly, this application provides a method for modifying a three-dimensional model, the method comprising: Based on the modification instructions obtained from the 3D base model, a modification guidance vector is generated; where the modification instructions are text modification instructions. Based on the region to be modified in the obtained 3D base model, and the 3D base model itself, generate geometric guiding vectors; Based on the modified guiding vector and geometric guiding vector, a modified geometric implicit vector is generated, and based on the modified geometric implicit vector, an initial 3D model is generated; The surface of the initial 3D model is sampled and encoded according to the first sampling method to obtain the first geometric latent vector. The surface of the 3D base model is sampled and encoded according to the first sampling method to obtain the second geometric latent vector. Based on the region to be modified, the first geometric latent vector and the second geometric latent vector are smoothly fused to obtain the third geometric latent vector. The modified 3D model is generated based on the third geometric latent vector.

[0007] In one optional implementation, generating the modification guidance vector based on the obtained modification instructions for the 3D base model includes: Generate a target image based on the text modification instructions, and the information described by the target image conforms to the text modification instructions; The target image is encoded to obtain an image guidance vector, which is then used as the modification guidance vector.

[0008] In one alternative implementation, generating geometric guidance vectors based on the region to be modified in the acquired 3D base model and the 3D base model includes: The region to be modified in the 3D base model is sampled using the second sampling method, and the region to be unmodified in the 3D base model is sampled using the third sampling method to obtain the target sampling point set; wherein, the sampling density of the second sampling method is less than the sampling density of the third sampling method; The position of each sampling point in the acquired target sampling point set is encoded to obtain the geometric guidance vector.

[0009] In one alternative implementation, generating the modified geometric latent vector based on the modified guiding vector and the geometric guiding vector includes: Cross-attention processing is applied to the modified guiding vector and the geometric guiding vector to obtain the modified geometric latent vector.

[0010] In one alternative implementation, generating the initial 3D model based on modified geometric latent vectors includes: The modified geometric latent vectors are input into a pre-trained 3D generative model to obtain an initial 3D model; the pre-trained 3D generative model includes a geometric latent vector processing module, a geometric encoder, and a geometric decoder.

[0011] In one optional implementation, based on the region to be modified, the first geometric latent vector and the second geometric latent vector are smoothly fused to obtain the third geometric latent vector, including: Based on the region to be modified, the first region to be modified and the first region not to be modified are marked in the first geometric implicit vector, and the second region to be modified and the second region not to be modified are marked in the second geometric implicit vector. Keeping the features of the first non-modified region in the first geometric latent vector and the features of the second non-modified region in the second geometric latent vector unchanged, the features of the first non-modified region in the first geometric latent vector and the features of the second non-modified region in the second geometric latent vector are processed by a preset smoothing fusion algorithm to obtain the third geometric latent vector.

[0012] In one alternative implementation, generating the modified 3D model based on the third geometric latent vector includes: The third geometric latent vector is input into the pre-trained 3D generative model, and the geometric latent vector processing module and geometric decoder are used to process the third geometric latent vector sequentially to generate the modified 3D model.

[0013] Secondly, embodiments of this application also provide an electronic device, including: a memory and a processor; The memory is configured to store computer program instructions; The processor is configured to execute computer program instructions that enable the electronic device to implement the method of modifying the three-dimensional model as described in any of the first aspects.

[0014] Thirdly, embodiments of this application also provide a computer-readable storage medium, including: computer program instructions; The electronic device executes computer program instructions, enabling the electronic device to implement the method of modifying the three-dimensional model as described in any of the first aspects.

[0015] Fourthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the method for modifying any of the three-dimensional models in the first aspect.

[0016] This application provides a method, device, medium, and program product for modifying a three-dimensional model. The method includes: generating a modification guidance vector based on a modification instruction of an acquired three-dimensional base model; generating a geometric guidance vector based on the region to be modified of the acquired three-dimensional base model and the three-dimensional base model; generating a modification geometric latent vector based on the modification guidance vector and the geometric guidance vector, and generating an initial three-dimensional model based on the modification geometric latent vector; sampling the surface of the initial three-dimensional model according to a first sampling method to obtain a first geometric latent vector, sampling the surface of the three-dimensional base model according to the first sampling method to obtain a second geometric latent vector; performing a smooth fusion process on the first geometric latent vector and the second geometric latent vector based on the region to be modified to obtain a third geometric latent vector; and generating a modified three-dimensional model based on the third geometric latent vector.

[0017] Based on this, the embodiments of this application complete the local modification of the 3D model through subsequent technical steps according to the text modification instructions and the area to be modified of the obtained 3D base model. This effectively solves the problems of existing technologies that rely on professional software, are cumbersome to operate, and are inefficient. It allows non-professionals to quickly achieve local modification of 3D models, significantly improves modification efficiency, and adapts to the modification needs of multiple fields such as industrial design and cultural and creative creation.

[0018] Furthermore, this application embodiment generates modification guidance vectors and geometric guidance vectors separately, corresponding to the modification requirements of the text modification command and the location information of the area to be modified, respectively. The two are then merged to generate a modification geometric latent vector, which can accurately predict the result of the area to be modified after adjustment by the modification command. This effectively solves the defect of the existing solution that cannot effectively combine the modification command, the original 3D model and the selected area, resulting in a large deviation between the modification result and the user's requirements, and ensures that the modified area to be modified conforms to the requirements of the text modification command.

[0019] Furthermore, the embodiments of this application are based on geometric guiding vectors, which means that the entire modification process only adjusts the area to be modified, without changing the features of the non-modified area. This can completely preserve the original geometric features such as shape, structure, and spatial position of the non-modified area of ​​the 3D base model, avoiding the problem of the modification process destroying the overall structure of the model in the prior art, and ensuring that the modified model maintains geometric consistency with the non-modified area of ​​the original 3D base model.

[0020] Finally, the embodiments of this application adopt the same first sampling method to sample the initial three-dimensional model and the three-dimensional base model respectively, obtain the corresponding first geometric latent vector and second geometric latent vector, and then perform smooth fusion processing on the two based on the region to be modified. This effectively solves the problem of abrupt connection and discontinuity between the modified region and the non-modified region in the existing solution, realizes the natural transition between the modified region and the non-modified region, ensures the overall coordination and integrity of the modified model, and improves the modification quality of the three-dimensional model. Attached Figure Description

[0021] Figure 1 A flowchart illustrating the steps of a method for modifying a three-dimensional model, as provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. It is understood that the terms “first,” “second,” etc., as used herein may be used to describe various information or data, but these elements are not limited by these terms. These terms are only used to distinguish the information described in the first instance from another piece of information. For example, without departing from the scope of this application, first action information may be referred to as second action information, and similarly, second action information may be referred to as first action information. Both first action information and second action information are action information, but they are not the same action information.

[0024] With the widespread application of 3D modeling technology in fields such as industrial design, games and film, and architectural decoration, the demand for local modifications to 3D models is increasing. Users expect to be able to quickly and accurately modify specified areas of the 3D base model while preserving the original geometric features of the non-modified areas of the 3D base model, ensuring the overall coordination and integrity of the modified model.

[0025] Current methods for modifying 3D models largely rely on specialized modeling software, requiring operators to manually select areas and adjust parameters. This approach is technologically advanced, inefficient, and difficult for non-professionals to master. Some existing solutions attempt to drive modifications through commands, but these fail to effectively integrate text commands, the original 3D model, and the selected area, making it difficult to accurately predict the outcome of text-based modifications to the selected area.

[0026] Such solutions generally have two major drawbacks: First, they cannot accurately respond to text commands, and the modification results deviate significantly from the text descriptions; second, the selected area after modification has poor integration with the remaining area of ​​the original model, which easily leads to problems such as abrupt transitions and discontinuities, and it is difficult to fully preserve the inherent geometric features of the non-selected areas of the original model, thus damaging the overall coordination of the model.

[0027] The technical solutions shown in this application will now be described in detail through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; for identical or similar content, the description will not be repeated in different embodiments.

[0028] Based on the above technical deficiencies, referring to Figure 1 This application provides a method for modifying a three-dimensional model, the method including: S100: Based on the modification instructions obtained from the 3D base model, generate modification guidance vectors. The modification instructions are text modification instructions.

[0029] In this embodiment of the application, the text modification instruction is used to express the intention to modify the shape, structure, form or appearance of a specified area of ​​the three-dimensional base model in natural language.

[0030] Subsequently, semantic parsing and feature encoding can be performed on the text modification instructions to extract the modification semantics, morphological constraints and style features corresponding to the text instructions, and map them into high-dimensional feature vectors with a unified representation form, thereby generating modification guidance vectors.

[0031] The modified guiding vector is used to quantify and express the semantic requirements of the text modification instructions. It serves as a semantic guide for the subsequent reconstruction of the 3D model's region to be modified, enabling the model generation process to follow the modification intent of the text instructions for constrained generation.

[0032] S200 generates geometric guidance vectors based on the region to be modified in the obtained 3D base model and the 3D base model.

[0033] In this embodiment, point cloud sampling can be performed on the surface of the three-dimensional base model based on the overall structural information of the three-dimensional base model and the selected area to be modified on the three-dimensional base model. Combined with the spatial position boundary information of the area to be modified, feature extraction and latent vector encoding are performed on the sampled point cloud through a preset encoding network to obtain feature vectors that can characterize the overall geometric contour, topological structure, size ratio and spatial position constraints of the area to be modified of the three-dimensional base model, so as to generate geometric guidance vectors.

[0034] This geometric guidance vector preserves the original geometric features of the unmodified regions of the 3D base model while carrying spatial definition information of the regions to be modified. It is used to constrain the subsequent 3D model generation process, ensuring that the overall geometric shape remains unchanged and that compliant reconstruction is performed only in the specified regions to be modified.

[0035] S300 generates modified geometric latent vectors based on modified guiding vectors and geometric guiding vectors, and generates an initial 3D model based on the modified geometric latent vectors.

[0036] In this embodiment, the modification guidance vector representing the semantic modification of the text is fused and associated with the geometric guidance vector representing the overall geometric features of the three-dimensional base model and the spatial constraints of the region to be modified. By combining the text modification intention and the geometric structure constraints of the original model, a modification geometric latent vector with both semantic modification features and model geometric prior is generated. Then, the modification geometric latent vector is input into a pre-trained three-dimensional generation network, and the geometric latent vector is analyzed and reconstructed in three-dimensional space. The decoded output is the initial three-dimensional model.

[0037] The initial 3D model both conforms to the shape requirements of the area to be modified according to the text modification instructions and maintains the original geometric features of the non-modified areas of the 3D base model.

[0038] S400, the surface of the initial 3D model is sampled according to the first sampling method to obtain the first geometric latent vector. The surface of the 3D base model is sampled and encoded according to the first sampling method to obtain the second geometric latent vector. Based on the region to be modified, the first geometric latent vector and the second geometric latent vector are smoothly fused to obtain the third geometric latent vector.

[0039] In this embodiment, a unified first sampling method is adopted to sample the surface of the initial three-dimensional model and the surface of the three-dimensional base model in the same way. The sampling data of the initial three-dimensional model is encoded to obtain a first geometric latent vector, which carries the new geometric features of the modified area of ​​the initial three-dimensional model. The sampling data of the three-dimensional base model is encoded to obtain a second geometric latent vector, which retains the inherent geometric features of the unmodified area of ​​the original model.

[0040] Based on the predefined boundaries of the region to be modified, the first and second geometric latent vectors are divided into regions. Smoothing and fusion processing is performed only on the feature parts corresponding to the region to be modified, while the original features of the unmodified region remain unchanged. Feature discontinuities are eliminated by feature smoothing transition and boundary feature interpolation fitting. The fused regional features are integrated with the unmodified regional features to finally generate a third geometric latent vector that combines the geometric shape of the original model with the new shape of the modified region and has a continuous and smooth transition of the regional boundary.

[0041] S500 generates a modified 3D model based on the third geometric latent vector.

[0042] In this embodiment, the fused third geometric latent vector is input into the pre-trained 3D generative model, which performs feature parsing and spatial geometric reconstruction on the third geometric latent vector. The latent vector features are mapped to the occupancy status of each sampling point in the 3D space by the geometric decoder, and then a continuous mesh model is generated by the surface reconstruction algorithm, and finally the modified 3D model is output.

[0043] The generated modified 3D model not only conforms to the shape requirements of the text modification instructions in the area to be modified, but also fully preserves the original geometric features of the non-modified areas of the 3D base model. Furthermore, it ensures a smooth and continuous transition of features between the area to be modified and the non-modified areas, without obvious breaks or abrupt connections, thus guaranteeing the geometric consistency and appearance integrity of the overall model.

[0044] Based on the above description, the embodiments of this application complete the local modification of the 3D model through subsequent technical steps according to the text modification instructions and the area to be modified of the obtained 3D base model. This effectively solves the problems of existing technologies that rely on professional software, are cumbersome to operate, and are inefficient. It allows non-professionals to quickly achieve local modification of 3D models, significantly improves modification efficiency, and adapts to the modification needs of multiple fields such as industrial design and cultural and creative creation.

[0045] Furthermore, this application embodiment generates modification guidance vectors and geometric guidance vectors separately, corresponding to the modification requirements of the text modification command and the location information of the area to be modified, respectively. The two are then merged to generate a modification geometric latent vector, which can accurately predict the result of the area to be modified after adjustment by the modification command. This effectively solves the defect of the existing solution that cannot effectively combine the modification command, the original 3D model and the selected area, resulting in a large deviation between the modification result and the user's requirements, and ensures that the modified area to be modified conforms to the requirements of the text modification command.

[0046] Furthermore, the embodiments of this application are based on geometric guiding vectors, which means that the entire modification process only adjusts the area to be modified, without changing the features of the non-modified area. This can completely preserve the original geometric features such as shape, structure, and spatial position of the non-modified area of ​​the 3D base model, avoiding the problem of the modification process destroying the overall structure of the model in the prior art, and ensuring that the modified model maintains geometric consistency with the non-modified area of ​​the original 3D base model.

[0047] Finally, the embodiments of this application adopt the same first sampling method to sample the initial three-dimensional model and the three-dimensional base model respectively, obtain the corresponding first geometric latent vector and second geometric latent vector, and then perform smooth fusion processing on the two based on the region to be modified. This effectively solves the problem of abrupt connection and discontinuity between the modified region and the non-modified region in the existing solution, realizes the natural transition between the modified region and the non-modified region, ensures the overall coordination and integrity of the modified model, and improves the modification quality of the three-dimensional model.

[0048] In one optional implementation, generating the modification guidance vector based on the obtained modification instructions for the 3D base model includes: First, a target image is generated based on the text modification instructions, and the information described by the target image conforms to the text modification instructions.

[0049] In this embodiment, a text modification instruction input by a user is received. This instruction expresses the modification requirements for the shape, structure, style, or details of a specified area to be modified in a 3D base model in natural language. Then, a preset text-image generation model is invoked to perform semantic understanding and parsing of the text modification instruction. According to the shape, form, style, and structural requirements expressed by the instruction, a corresponding target image is generated so that the visual content, outline shape, and structural details presented by the target image completely match the modification intent of the text modification instruction.

[0050] The pre-built text-image generation model is a pre-trained, embedded AI model that receives natural language text commands, automatically understands the semantics, stylistic requirements, and structural descriptions of the text, and generates a target image that matches the text's intent from scratch. Its working principle is as follows: it receives input text modification commands, performs semantic parsing, keyword extraction, and stylistic intent understanding; it learns the shape, outline, structure, style, and detail features described in the text; based on the visual prior knowledge learned through massive training, it renders and generates the corresponding two-dimensional target image; the generated target image serves as a visualization carrier of the text's semantics, transforming the abstract text modification intent into a concrete visual form, facilitating subsequent conversion into vector features to constrain the modification of the three-dimensional model.

[0051] Then, the target image is encoded to obtain the image guidance vector, which is then used as the modification guidance vector.

[0052] In this embodiment, a pre-trained image coding model can be used to extract deep features and encode vectors in the generated target image to obtain contour features, structural features, morphological style and detail texture information contained in the target image. The two-dimensional visual image information is mapped into a high-dimensional feature vector that can be recognized by the machine and participate in the three-dimensional model generation operation to obtain the image guidance vector. The image guidance vector is then directly used as the modification guidance vector to provide semantic guidance and morphological prior for the geometric reconstruction of the area to be modified in the subsequent three-dimensional model.

[0053] The pre-trained image coding model is a convergent visual feature extraction model trained on a large-scale image dataset. It is specifically designed to extract deep features from the target image output by the aforementioned text-to-image generation model, converting the two-dimensional image into a high-dimensional feature vector that can be computed by the machine and used for three-dimensional generation. Its working principle is as follows: it receives the target image output by the text-to-image generation model; extracts high-level semantic information such as contour features, structural features, morphological proportions, and style textures from the image layer by layer; discards irrelevant pixel details, retaining core representations related to shape and structure; and compresses and maps the entire image into a fixed-dimensional high-dimensional vector, i.e., the image guidance vector, which is used as a modified guidance vector in subsequent three-dimensional latent vector fusion and model generation.

[0054] Based on the above description, the embodiments of this application first convert text instructions into concrete target images, and then encode them into feature vectors, realizing the conversion of text semantics to two-dimensional images to high-dimensional guiding vectors, and achieving effective conversion of natural language into three-dimensional generated recognizable features.

[0055] Furthermore, in this embodiment, the target image is encoded into a unified format image guidance vector and used as a modification guidance vector. The feature dimensions are standardized and the semantic representation is stable, which can be adapted to subsequent geometric guidance vectors, providing a reliable and unified semantic constraint benchmark for 3D latent vector generation and model reconstruction.

[0056] Furthermore, users only need to input natural language text to convey their intentions, without the need for professional modeling operations or manual drawing of reference shapes. The system automatically completes the text-image conversion and feature encoding, enabling text-driven local modifications to the 3D model.

[0057] Optionally, based on the region to be modified in the obtained 3D base model and the 3D base model, the geometric guidance vector is generated, including: First, the region to be modified in the 3D base model is sampled according to the second sampling method, and the region to be unmodified in the 3D base model is sampled according to the third sampling method to obtain the target sampling point set; wherein, the sampling density of the second sampling method is less than the sampling density of the third sampling method.

[0058] In this embodiment of the application, the three-dimensional base model is pre-divided into regions to be modified and regions not to be modified; sampling is performed using independent sampling methods for different regions: the regions to be modified in the three-dimensional base model are sampled according to the second sampling method, while the regions not to be modified in the three-dimensional base model are sampled according to the third sampling method; wherein, the sampling density of the second sampling method is configured to be less than the sampling density of the third sampling method.

[0059] In other words, a relatively sparse sampling strategy is used for the area to be modified, with a relatively loose distribution of sampling points and a smaller number of points; while a higher-density, denser sampling method is used for the unmodified area, with a more detailed arrangement of sampling points and more complete spatial point coverage. The area to be modified only requires a small number of sampling points to mark its spatial range, regional boundaries, and overall positional outline, without needing too many detailed points; while the unmodified area, as the main part that needs to completely preserve its original shape, outline structure, and overall topological relationship, is densely sampled using a higher-density third sampling method, which can comprehensively capture the surface changes, outline trends, spatial proportions, and subtle geometric structures of the unmodified area.

[0060] The sampling points obtained from the region to be modified using the second sampling method are combined with the sampling points obtained from the non-modified region using the third sampling method to form a complete set of target sampling points.

[0061] Second, the position of each sampling point in the acquired target sampling point set is encoded to obtain the geometric guiding vector.

[0062] In this embodiment, the target sampling point set includes all sampling points of the three-dimensional base model's modified and unmodified regions. Each sampling point carries unique three-dimensional spatial coordinates, relative positional relationships, and region affiliation information. First, the three-dimensional spatial position data, spatial distribution relationships between points, and region boundary association information of all sampling points in the set are extracted. Then, the aforementioned positional features and spatial topological features are input into a preset encoding network. The encoding network performs feature extraction, dimensionality compression, and high-dimensional mapping on the discrete sampling point coordinates, converting the discrete point cloud position information in three-dimensional space into a unified-dimensional latent space feature representation that can be recognized and processed by subsequent modules. Finally, a geometric guidance vector is output.

[0063] This geometric guide vector contains geometric constraint information such as the overall outline of the 3D base model, the spatial location of each region, the boundary range, and the structural topology, which is used to constrain the generation and modification range of the 3D model in subsequent processes.

[0064] Based on the above description, this application embodiment uses low-density sampling in the region to be modified, requiring only a small number of sampling points to complete region localization and boundary definition, reducing the number of redundant sampling points, lowering the computational load of feature encoding, and improving vector generation efficiency. A higher sampling density is set for the unmodified region, which can finely depict the original contour, surface curvature, structural details, and spatial topological relationships of the unmodified region, accurately preserving the inherent geometric features of the original model, and avoiding deformation, distortion, or structural shift in the unmodified region during the modification process.

[0065] By employing a differentiated deployment strategy of sparse sampling in the region to be modified and dense sampling in the unmodified region, the geometric details of the main region of the original model are preserved to the greatest extent while ensuring accurate positioning of the region to be modified. This allows the generated geometric guiding vectors to both constrain the scope of local modifications and stably maintain the geometric consistency of the overall model, providing reliable geometric prior constraints for subsequent vector fusion, model reconstruction, and smooth transition.

[0066] In one alternative implementation, generating the modified geometric latent vector based on the modified guiding vector and the geometric guiding vector includes: Cross-attention processing is applied to the modified guiding vector and the geometric guiding vector to obtain the modified geometric latent vector.

[0067] In this embodiment, the modification guidance vector, which represents the semantic intent of text modification, is input-matched with the geometric guidance vector, which represents the overall geometric structure and regional spatial constraints of the three-dimensional base model. The two sets of vectors are subjected to cross-attention processing. The attention mechanism is used to model the weights of the feature association, semantic correspondence, and spatial position association between the modification guidance vector and the geometric guidance vector, thereby strengthening the effective feature components that match and constrain each other and weakening irrelevant and redundant feature components.

[0068] After bidirectional feature interaction and adaptive weight allocation through cross-attention, the semantic guidance features of the text and the geometric constraint features of the model are integrated to output a modified geometric latent vector that combines the text modification intention, the overall geometric shape of the three-dimensional base model, and the positional constraints of the region to be modified.

[0069] Based on the above description, a cross-attention mechanism is used to achieve bidirectional deep feature interaction between the modification guidance vector and the geometric guidance vector, establishing a precise correlation between textual semantics and 3D geometric structure. Furthermore, through adaptive attention weight allocation, effective features are highlighted and redundant interference features are suppressed, ensuring that the generated modification geometric latent vectors not only conform to the shape requirements of the text modification instructions but also adhere to the overall geometric structure and regional boundary constraints of the original 3D base model. This improves the rationality and adaptability of feature fusion, avoiding feature fragmentation and semantic-geometric mismatch problems caused by simple splicing, and providing high-quality latent vector representations for subsequently generating an initial 3D model with a reasonable structure and shape that conforms to the instructions.

[0070] Optionally, generating an initial 3D model based on modified geometric latent vectors includes: The modified geometric latent vectors are input into a pre-trained 3D generative model to obtain an initial 3D model. The pre-trained 3D generative model includes a geometric latent vector processing module, a geometric encoder, and a geometric decoder. The geometric latent vector processing module is used to generate target geometric latent vectors based on the modified guiding latent vectors and random noise. The geometric encoder is used to map target spatial points to target geometric latent vectors. The geometric decoder is used to map target geometric latent vectors to target spatial points to obtain the initial 3D model.

[0071] In this embodiment of the application, the process of generating an initial 3D model based on modified geometric latent vectors is as follows: the modified geometric latent vectors are input into a pre-trained 3D generative artificial intelligence model, and the initial 3D model is obtained through inference.

[0072] The pre-trained 3D generative model includes a geometric encoder, a geometric latent vector processing module, and a geometric decoder.

[0073] The geometric latent vector processing module receives two types of inputs: modified and guided latent vectors and random noise. It uses its built-in network structure to iteratively optimize the random noise layer by layer. At the same time, it combines the semantic, morphological and geometric constraint information of the modified and guided latent vectors to perform feature correction and guidance, gradually converging and reconstructing the disordered random noise into a target geometric latent vector with regular features and conforming to preset constraints.

[0074] The geometric encoder takes target spatial points in three-dimensional space as input, extracts and encodes features such as the three-dimensional coordinates, position distribution, and topological relationships of adjacent points of each spatial point, and maps discrete spatial point geometric information into standardized target geometric latent vectors, thus completing the representation transformation from three-dimensional spatial points to latent vector space.

[0075] The geometry decoder receives the target geometric latent vector output by the aforementioned module, analyzes and restores the latent space features, and maps the abstract target geometric latent vector back to the three-dimensional target space points. Combining the space point occupancy determination and mesh reconstruction algorithm, it completes the construction of the three-dimensional shape and finally generates a complete initial three-dimensional model.

[0076] Based on the above description, a pre-trained 3D generative model is used to automatically generate the initial 3D model from modified geometric latent vectors. This eliminates the need for manual setting of modeling parameters and manual adjustment of the model structure, significantly reducing the operational threshold for text editing of 3D models and improving the efficiency of model generation and local modification.

[0077] Among them, the geometric latent vector processing module combines the modified guided latent vector with random noise to perform feature reconstruction. Based on random noise, it can gradually optimize and converge to obtain a compliant target geometric latent vector according to the semantic and geometric constraints of the guided latent vector. This ensures that the generation process strictly conforms to the text modification intention and the geometric prior constraints of the 3D base model, guaranteeing that the shape of the area to be modified is generated reasonably and without distortion.

[0078] The geometric encoder realizes standardized encoding of three-dimensional spatial points into latent vectors, unifies the data representation form, enables discrete point information to participate in the internal feature calculation of the model efficiently, and ensures the compatibility of data interaction between modules.

[0079] The geometric decoder completes the inverse mapping from latent vectors to points in three-dimensional space, stably reconstructing continuous and topologically sound three-dimensional geometric shapes, and successfully outputting an initial three-dimensional model that meets the modification requirements.

[0080] By modularly cooperating with the geometric latent vector processing module, the geometric encoder, and the geometric decoder, the latent vector feature optimization, spatial point mapping, and 3D model reconstruction are achieved. The generated initial 3D model has a regular structure and natural surface transitions, laying a reliable foundation for subsequent smooth latent vector fusion and the final high-quality 3D model output.

[0081] In one optional implementation, based on the region to be modified, the first geometric latent vector and the second geometric latent vector are smoothly fused to obtain the third geometric latent vector, including: Based on the region to be modified, the first region to be modified and the first region not to be modified are marked in the first geometric implicit vector, and the second region to be modified and the second region not to be modified are marked in the second geometric implicit vector. Keeping the features of the first non-modified region in the first geometric latent vector and the features of the second non-modified region in the second geometric latent vector unchanged, the features of the first non-modified region in the first geometric latent vector and the features of the second non-modified region in the second geometric latent vector are processed by a preset smoothing fusion algorithm to obtain the third geometric latent vector.

[0082] In this embodiment of the application, based on the region to be modified, the first geometric latent vector and the second geometric latent vector are smoothly fused to obtain the third geometric latent vector. The specific process is as follows: First, according to the pre-defined range of the region to be modified, the first region to be modified and the first region not to be modified are marked in the first geometric latent vector representing the features of the initial three-dimensional model; at the same time, the second region to be modified and the second region not to be modified are marked in the second geometric latent vector representing the features of the original three-dimensional base model, so as to achieve accurate alignment and matching of the two sets of latent vectors in the region dimension.

[0083] Subsequently, feature partitioning preservation and fusion operations are performed: the first unmodified region features of the first geometric latent vector and the second unmodified region features of the second geometric latent vector are kept unchanged, fully preserving the inherent geometric representation of the unmodified regions of the original model; only for the first unmodified region features of the first geometric latent vector and the second unmodified region features of the second geometric latent vector, a preset smoothing fusion algorithm is called to perform cross-vector feature interpolation, overfitting, and weight balancing, weakening the abrupt differences between the two sets of features and achieving a smooth transition of region features. The unchanged unmodified region features are integrated with the smoothed unmodified region features to generate a third geometric latent vector with continuous overall features and natural boundary transitions.

[0084] Based on the above description, this application embodiment simultaneously calibrates the regions to be modified and the regions not to be modified in two sets of geometric latent vectors, achieving precise alignment between latent vector features and model space regions, providing a basis for differentiated feature processing. Furthermore, by keeping the features of the two regions not to be modified unchanged, the contours, structures, and geometric features of the original 3D base model's unmodified regions are fully inherited, avoiding deformation and distortion of irrelevant regions during the fusion process. Only the features of the regions to be modified are subjected to smooth fusion operations, specifically eliminating feature breaks and abrupt differences at the boundaries between the initial modified model and the original model, ensuring a smooth and natural connection between the regions to be modified and the surrounding structures. This results in the third geometric latent vector obtained through fusion possessing both the main features of the original model and the new morphological features of the modified regions, with continuous and balanced feature transitions between regions, providing a high-quality latent vector representation for the subsequent reconstruction of a final 3D model that is overall coordinated and without obvious splicing traces.

[0085] In one alternative implementation, generating the modified 3D model based on the third geometric latent vector includes: The third geometric latent vector is input into the pre-trained 3D generative model, and the geometric latent vector processing module and geometric decoder are used to process the third geometric latent vector sequentially to generate the modified 3D model.

[0086] In this embodiment, the third geometric latent vector is input into the pre-trained 3D generative model. First, the geometric latent vector processing module uses the third geometric latent vector as the guiding latent vector, combines random noise for iterative optimization and latent space feature correction, performs regularization optimization on the fusion features of the third geometric latent vector, removes feature perturbations and numerical mutations generated during the fusion process, and outputs the target geometric latent vector adapted to the 3D geometric reconstruction task.

[0087] Subsequently, the geometric decoder receives the target geometric latent vector, maps the target geometric latent vector inversely to the coordinate points in the three-dimensional target space, determines the model occupancy status of each spatial coordinate point, and completes geometric topology restoration and surface fitting in combination with the mesh reconstruction algorithm. Finally, it generates a modified three-dimensional model with complete overall structure, faithful geometric features of non-modified areas, and the modified areas conform to the text modification intention, and the boundary transitions of the areas are smooth and continuous.

[0088] Based on the above description, this application reuses the same pre-trained 3D generation model for inference, ensuring a unified model architecture and consistent representation standards, thus avoiding feature bias and style fragmentation caused by switching between multiple models. Specifically, the geometric latent vector processing module further optimizes and corrects the third geometric latent vector, correcting feature perturbations generated during the fusion process, resulting in more regular latent vector representations and a more reasonable geometric structure. The geometric decoder completes 3D spatial reconstruction based on the optimized latent vectors, accurately restoring the original geometric shape of the unmodified region while maintaining the shape of the region to be modified in accordance with the intent of the text command. This results in a smooth and continuous transition between the boundary between the region to be modified and the unmodified region, without obvious splicing marks or structural breaks, improving the integrity, smoothness, and visual consistency of the modified 3D model.

[0089] The following describes a specific embodiment of the method for modifying the three-dimensional model provided in this application: Build and pre-train an artificial intelligence model for 3D model generation, which can be referenced from the Clay model architecture. It consists of three parts: a geometric encoder, a geometric latent vector processing module, and a geometric decoder.

[0090] The geometric encoder is a 3D model encoding module whose function is to compress the input 3D model into geometric latent vectors. Specifically, it performs a set number of point cloud samplings on the surface of the input geometric model to obtain a set of 3D point cloud coordinates; the geometric encoder extracts spatial position features, contour structure features, and topological relationship features from this set of point cloud coordinates, and maps and encodes the discrete point cloud information into geometric latent vectors of fixed dimensions, realizing the standardized representation of the 3D geometric structure into the latent vector space.

[0091] The geometric latent vector processing module is used to iteratively reconstruct and optimize compliant geometric latent vectors from random noise based on the input guiding latent vector. A preferred implementation is as follows: this module consists of a 24-layer Transformer structure combined with a cross-attention module; the multi-layer Transformer is responsible for iteratively constraining and optimizing the features of the initial random noise layer by layer, gradually correcting the noise distribution and converging to latent vector features that conform to geometric laws; the cross-attention module specifically introduces and integrates external guiding latent vector information, so that the entire noise optimization process is constrained by guiding semantics and geometric boundaries, ensuring that the generation direction conforms to the preset modification intention.

[0092] The geometry decoder is a 3D model decoding module whose function is to inversely reconstruct a complete 3D geometric model from the latent geometric vectors. The specific process is as follows: the latent geometric vectors to be decoded are input to the geometry decoder, along with the coordinates of a series of regularly arranged spatial points in 3D space; based on the characteristics of the latent vectors, the geometry decoder predicts whether each spatial point is occupied by a 3D model entity, and outputs the occupancy determination result for each spatial point; then, the moving cube algorithm is used to extract isosurfaces, fit meshes, and reconstruct surfaces from the spatial occupancy results, ultimately generating a continuous, smooth, and topologically complete 3D mesh model.

[0093] This embodiment employs a point cloud guidance mechanism to ensure that the generated result maintains geometric consistency with the original input 3D model. The implementation method is as follows: uniform fixed-point sampling is performed on the surface of the original input 3D model to collect a fixed number of surface point clouds; the spatial coordinates of these point clouds are then fed into a geometric encoder for feature encoding to obtain corresponding geometric latent vectors. These geometric latent vectors are then directly used as guidance vectors and input into the geometric latent vector processing module. Using the geometric features of the original model's point cloud as prior constraints, the overall outline, size proportions, and topological structure of the subsequently generated model closely match the original input model.

[0094] Specific implementation steps may include: selecting the original 3D base model to be edited, and defining the area to be modified by spatial cube annotation; that is, defining the local space of the model with a preset cube bounding box, defining the model area inside the cube space as the area to be modified, and defining the model area outside the cube space as the non-modified area.

[0095] Generate target image based on text modification instructions: Receive natural language text modification instructions input by the user, use existing text-image generation AI (Artificial Intelligence) models to analyze the semantics of the text and automatically generate a target image that matches the visual content, so that the shape, outline, structure and style details of the target image completely match the modification intent of the text modification instructions.

[0096] Differentiated sampling is performed on the original 3D base model: high-density sampling is performed on the non-modified areas, with the number of sampling points reaching hundreds of thousands, so as to preserve the geometric details of the main body of the original model in detail; uniform sparse sampling is performed on the areas to be modified, and the sampling points of these parts are uniformly marked as modifiable unknown, indicating that the area can be reconstructed according to text commands.

[0097] All sampling points with location information are fed into a geometric encoder to obtain geometric guidance vectors representing the overall structure and region boundaries of the original model. Simultaneously, a DINO V2 pre-trained image encoding model is used to extract high-level visual features and encode vectors from the target image, yielding image guidance vectors. The geometric guidance vectors and image guidance vectors are then fused using cross-attention features, bidirectionally linking the text semantic morphology with the geometric constraints of the original model to generate the final geometric guidance latent vector.

[0098] The initial modified model is generated through a single decoding process: the final geometric latent vector is input into a pre-trained 3D generative model; the geometric latent vector processing module combines random noise with the latent vector for feature reconstruction and optimization; then, the geometric decoder performs spatial point occupancy prediction and Marching Cube mesh reconstruction, outputting the modified model. This modified model ensures that the unmodified regions fully inherit the inherent geometry of the original model, and the shape of the regions to be modified conforms to the description requirements of the text modification instructions; however, the boundary between the regions to be modified and the unmodified regions is prone to abrupt surface changes, harsh boundaries, and uneven transitions.

[0099] The preprocessing for secondary decoding is as follows: First, approximately 20,000 points are uniformly sampled on the surface of the modified model 1 and fed into the geometric encoder to obtain the first geometric latent vector. The feature positions of the corresponding regions to be modified are marked on the dimension of the latent vector. Second, approximately 20,000 points are also sampled on the surface of the original 3D base model, ensuring that the number and distribution of sampling points inside and outside the modified region are completely consistent with the sampling rules of the modified model. The second geometric latent vector is obtained by encoding with the geometric encoder.

[0100] Based on the predefined boundaries of the region to be modified, the features corresponding to the unmodified regions in the first and second geometric latent vectors remain completely unchanged; only the features of the regions to be modified in the two sets of latent vectors are interpolated, weighted and transitioned using the Gaussian smoothing fusion algorithm to eliminate feature abrupt changes and boundary faults, and a third geometric latent vector with continuous features and natural boundary transitions is generated.

[0101] The fused third geometric latent vector is then input into the pre-trained 3D generative model. The model is then processed sequentially through the geometric latent vector processing module for feature correction and optimization, and the geometric decoder for 3D spatial reconstruction. Finally, the modified model is generated as the final output model.

[0102] It should be understood that the point cloud sampling and geometric encoder-guided approach can preserve the geometric contours, surface details, and topological structure of the unmodified areas of the original 3D base model to the greatest extent possible, ensuring global geometric consistency. Through text-to-image intermediate transformation, image encoding, and cross-attention fusion of geometric guidance vectors, natural language text commands can be used to precisely drive 3D local shape modifications, resulting in accurate semantic understanding and generated shapes that match the user's intent.

[0103] Furthermore, it can quickly achieve regional differentiation modifications with a single decoding to meet the requirements of text modeling; through secondary sampling, latent vector smooth fusion and secondary decoding, it effectively solves the problems of abrupt transitions and surface discontinuities at the boundary between the modified area and the original model, achieving seamless and smooth connection.

[0104] Secondly, this application also provides an electronic device. Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 2 As shown, the electronic device may include: transceiver 121, processor 122, and memory 123.

[0105] Processor 122 executes computer execution instructions stored in memory, causing processor 122 to perform the scheme in the above method embodiments. Processor 122 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0106] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0107] Transceiver 121 can be used to obtain the task to be run and its configuration information.

[0108] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0109] This application also provides a chip for executing instructions, which is used to execute the technical solutions of the methods described in the above embodiments.

[0110] Thirdly, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the technical solutions of the methods described in the above embodiments.

[0111] Fourthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solutions of the methods in the above embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0113] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0114] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0115] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0116] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0117] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0118] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0119] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0120] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0121] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for modifying a three-dimensional model, characterized in that, The methods for modifying the three-dimensional model include: Based on the modification instructions of the acquired 3D base model, a modification guidance vector is generated; wherein, the modification instructions are text modification instructions. Based on the region to be modified in the obtained 3D base model and the 3D base model, a geometric guiding vector is generated; Based on the modified guiding vector and the geometric guiding vector, a modified geometric implicit vector is generated, and based on the modified geometric implicit vector, an initial 3D model is generated; The surface of the initial 3D model is sampled and encoded according to the first sampling method to obtain the first geometric latent vector. The surface of the 3D base model is sampled and encoded according to the first sampling method to obtain the second geometric latent vector. Based on the region to be modified, the first geometric latent vector and the second geometric latent vector are smoothly fused to obtain the third geometric latent vector. Based on the third geometric latent vector, a modified 3D model is generated.

2. The method for modifying a three-dimensional model according to claim 1, characterized in that, The step of generating the modification guidance vector based on the modification instructions of the acquired 3D base model includes: A target image is generated according to the text modification instructions; wherein the information described by the target image conforms to the text modification instructions; The target image is encoded to obtain an image guidance vector, and the image guidance vector is used as the modified guidance vector.

3. The method for modifying a three-dimensional model according to claim 1, characterized in that, The step of generating geometric guidance vectors based on the obtained region to be modified in the 3D base model and the 3D base model includes: The region to be modified in the three-dimensional base model is sampled according to a second sampling method, and the region to be unmodified in the three-dimensional base model is sampled according to a third sampling method to obtain a target sampling point set; wherein, the sampling density of the second sampling method is less than the sampling density of the third sampling method; The positions of each sampling point in the acquired target sampling point set are encoded to obtain the geometric guidance vector.

4. The method for modifying a three-dimensional model according to claim 1, characterized in that, The step of generating the modified geometric latent vector based on the modified guiding vector and the geometric guiding vector includes: The modified guiding vector and the geometric guiding vector are subjected to cross-attention processing to obtain the modified geometric latent vector.

5. The method for modifying a three-dimensional model according to any one of claims 1-4, characterized in that, The process of generating the initial 3D model based on the modified geometric latent vectors includes: The modified geometric latent vectors are input into a pre-trained 3D generative model to obtain the initial 3D model; wherein the pre-trained 3D generative model includes a geometric latent vector processing module, a geometric encoder, and a geometric decoder.

6. The method for modifying a three-dimensional model according to claim 5, characterized in that, The step of smoothly fusing the first and second geometric latent vectors based on the region to be modified to obtain the third geometric latent vector includes: Based on the region to be modified, the first region to be modified and the first region not to be modified are marked in the first geometric latent vector, and the second region to be modified and the second region not to be modified are marked in the second geometric latent vector. Keeping the features in the first geometric latent vector corresponding to the first non-modified region and the features in the second geometric latent vector corresponding to the second non-modified region unchanged, the features in the first geometric latent vector corresponding to the first modified region and the features in the second geometric latent vector corresponding to the second modified region are processed using a preset smoothing fusion algorithm to obtain the third geometric latent vector.

7. The method for modifying a three-dimensional model according to claim 5, characterized in that, The step of generating the modified 3D model based on the third geometric latent vector includes: The third geometric latent vector is input into the pre-trained 3D generative model, and the geometric latent vector processing module and the geometric decoder are used to process the third geometric latent vector sequentially to generate the modified 3D model.

8. An electronic device, characterized in that, include: A memory and a processor; the memory is configured to store computer program instructions; the processor is configured to execute the computer program instructions, causing the electronic device to implement the method for modifying a three-dimensional model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, include: Computer program instructions; The electronic device executes the computer program instructions, causing the electronic device to implement the method for modifying the three-dimensional model as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method for modifying the three-dimensional model as described in any one of claims 1-7.

Images