A multi-view consistency three-dimensional object stylization method and device

Abstract

The application discloses a multi-view consistency three-dimensional object stylization method and device, and relates to the technical field of three-dimensional modeling. The method steps are as follows: obtaining a target three-dimensional object and its text description; generating an initial guide image through a text-to-image diffusion model, extracting orthogonal four-view features through multi-view diffusion, obtaining an image embedding vector through weighted fusion, and simultaneously extracting a text embedding encoding vector; rendering the object geometry structure and the initial texture to obtain a depth object graph; constructing a synchronous multi-diffusion model, synchronously generating latent images of each view based on the depth object graph and the two types of embedding vectors, and obtaining consistent prior views; calculating the loss of the prior views and the depth graph, distributing adaptive weights according to the loss of each view, and performing weighted summation; and finally, through iterative updating of the texture map by reverse gradient propagation until convergence, consistent stylized texture is obtained. Through multi-view guidance enhancement and adaptive weight constraint, the application solves the problems of color inconsistency and multi-head phenomenon in multi-view texture splicing.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a method and apparatus for stylizing 3D objects with multi-view consistency. Background Technology

[0002] Automatic 3D object generation is an important research direction in the fields of computer graphics and computer vision, with broad application prospects, especially playing a key role in 3D modeling, video game development, virtual reality, and augmented reality. Among these, texture generation is one of the core components of automatic 3D object generation, directly affecting the visual realism and expressiveness of 3D models.

[0003] Currently, texture generation methods for 3D objects are mainly divided into two categories: 3D optimization-based generation methods and 2D iteration-based texture generation methods.

[0004] The first type of method aims to train a 3D diffusion model directly on a 3D dataset in order to reduce the reliance on multiple diffusion models when generating textures on 3D shaped surfaces.

[0005] Existing techniques propose a 3D mesh texture generation scheme combining point cloud and UV mapping. This method first samples a point cloud on the surface of a 3D mesh, using point positions, normal vectors, and UV coordinates as feature inputs to a diffusion model. By progressively adding noise to the point cloud texture and performing a denoising process, the model generates a texture distribution on the point cloud. To maintain the geometric continuity of the texture, this method fully utilizes the geometric information of the 3D point cloud during the diffusion process and uses UV mapping to project the point cloud texture onto a 2D texture image, thereby eliminating the gap problem commonly found in traditional UV texture generation. This type of method has significant limitations: because it is trained directly on large-scale 3D datasets, and the quantity and diversity of existing 3D datasets are far less than those of 2D image datasets, the textures generated after model training are often blurry and lack rich texture variations. Furthermore, when extending the 3D diffusion model to higher resolutions or more complex geometries, the computational cost increases significantly, posing a substantial challenge to scalability.

[0006] The second type of method utilizes a pre-trained 2D diffusion model as prior knowledge, generating textures through iterative rendering and leveraging techniques such as depth imaging and image inpainting to endow the 2D diffusion model with a certain degree of 3D information understanding. For example, the texture-to-texture (Text2Tex) method employs a pre-trained depth-aware text-to-image diffusion model, rendering the target mesh from multiple viewpoints and progressively generating partial textures before back-projecting them into texture space. To handle stretching and inconsistency artifacts caused by viewpoint rotation, Text2Tex employs a view partitioning technique. It guides the diffusion process by calculating the similarity map between the normal vectors of visible finites and the current viewing direction, applying different diffusion intensities to different regions to supplement missing appearances and update stretching artifacts. Simultaneously, to avoid texturing failures due to improper viewpoint order definition or incomplete coverage, Text2Tex proposes an automatic viewpoint selection technique, progressively selecting the next optimal viewpoint and generating and updating based on the confidence of the largest relative region contained in each candidate viewpoint. Although this type of method demonstrates strong capabilities in generating diverse and high-resolution textures, its iterative rendering process is prone to introducing 3D inconsistencies, reducing generation efficiency. Furthermore, this type of method tends to produce shadow effects in textures that originate from the backbone of the diffusion model. Due to the failure to fully utilize the semantic information of user prompts and the inconsistencies between multiple views, the generated textures often fail to fully meet user expectations and may exhibit obvious seams or texture discontinuities from different perspectives.

[0007] In summary, existing 3D object texture generation technologies still suffer from problems such as texture blurring, insufficient diversity, 3D inconsistency, low computational efficiency, and difficulty in fully responding to user semantic needs. Summary of the Invention

[0008] This invention aims to at least improve one of the technical problems existing in the prior art. To this end, this invention proposes a multi-view consistent 3D object stylization method.

[0009] The technical solution of the present invention is as follows: A multi-view consistent 3D object stylization method, comprising: S1: Obtain the target 3D object and its corresponding text description; S2: Construct a multi-view guidance enhancement model, including a text-to-image diffusion model and a multi-view diffusion model. Input the text description into the text-to-image diffusion model to obtain an initial guidance image x. Input the image x into the multi-view diffusion model for diffusion and preprocessing to extract the image features of the orthogonal four-view set xn. Further weighted fusion is then used to obtain the image embedding vector E. The text encoder is used to extract the features of the text description and form the text embedding encoding vector T. S3: Obtain the geometric network structure and initial texture map of the target 3D object, and render it to obtain a rendering depth map and an object map with texture color from multiple perspectives. S4: Construct a synchronous multi-diffusion model, based on the rendered depth map, the image embedding vector E and the text embedding encoding vector T, to synchronously generate potential image variables at the same angle for each viewpoint, and obtain a prior view with multi-view consistency; S5: After calculating the loss of the prior view and the object map with texture and color, the adaptive weight is dynamically calculated based on the loss of each view, and the loss of each view is weighted and summed to obtain the total loss. S6: Update the texture map of the target 3D object through backward gradient propagation. Repeat the update step until the loss converges to obtain a stylized and consistent object texture map.

[0010] In one possible technical solution, the processing procedure of S2 further includes: The text description is input into the text-to-image diffusion model to obtain the initial guiding image x. The initial guiding image x is input into a multi-view diffusion model for diffusion to generate a multi-view set xm from different perspectives; The multi-view set xm is preprocessed to obtain the orthogonal four-view set xn; The image encoder based on the multi-view diffusion model encodes the images in the orthogonal four-view set xn, extracts the image features of the orthogonal four-view set xn, and uses different weights to perform weighted fusion to form the image embedding vector E; The text encoder based on the text-to-image diffusion model extracts text features and forms a text embedding encoding vector T.

[0011] In one possible technical solution, further, in step S5, the specific formula for calculating the loss between the prior view and the rendered depth map is as follows: in, Number the viewpoints to represent the first viewpoint. One perspective, Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where H is the image height calculated by the input loss, W is the image width calculated by the input loss, h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image.

[0012] In one possible technical solution, further, in step S5, the formula for calculating the total loss is as follows: , in, Number the viewpoints to represent the first viewpoint. One perspective, For the number of viewpoints, For dynamic weights from different perspectives, Set the L2 regularization constraint to 0.1. Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image.

[0013] In one possible technical solution, S4 further includes: The image features of the guiding images from different viewpoints in the orthogonal four-view set xn are weighted and fused using the following formula: in, This represents the fusion weight of each guide image. Indicates the first The image embedding vector of the guide image.

[0014] In one possible technical solution, the processing procedure of S4 further includes: The rendered depth maps of the target 3D object generated from various viewpoints are uniformly input into the synchronous multidiffusion model, and the denoising batch size is set to the total number of viewpoints. N ; The model performs a synchronous denoising process, during which a multi-view guided enhancement module is activated, which utilizes pre-trained text embedding encoding vectors. T and image embedding vector E As a joint constraint, a priori view with consistency across multiple perspectives is generated.

[0015] A multi-view consistent 3D object stylization apparatus, wherein, for implementing the stylization method described above, includes: The acquisition module is used to acquire the target 3D object and its corresponding text description; The first construction module is used to construct a multi-view guidance enhancement model, including a text-to-image diffusion model and a multi-view diffusion model. The text description is input into the text-to-image diffusion model to obtain an initial guidance image x. The image is then input into the multi-view diffusion model for diffusion and preprocessing to extract the image features of the orthogonal four-view set xn. The features are further weighted and fused to obtain the image embedding vector E. The text encoder is used to extract the features of the text description and form the text embedding encoding vector T. The rendering module is used to obtain the geometric network structure and initial texture map of the target 3D object, and render it to obtain a rendering depth map and an object map with texture color from multiple perspectives. The second construction module is used to construct a synchronous multi-diffusion model, which generates potential image variables at the same angle for each viewpoint based on the rendered depth map, the image embedding vector E and the text embedding encoding vector T, to obtain a prior view with multi-view consistency. The calculation module is used to calculate the loss of the prior view and the object map with texture and color, and then dynamically calculate the adaptive weight based on the loss of each view. The loss of each view is weighted and summed to obtain the total loss. The update module is used to update the texture map of the target 3D object through backpropagation. The update steps are repeated until the loss converges, resulting in a stylized and consistent object texture map.

[0016] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multi-view consistent 3D object stylization method as described above.

[0017] A computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the multi-view consistent 3D object stylization method described above.

[0018] The multi-view consistent 3D object stylization method of the present invention has the following advantages compared with the prior art: This invention guides a synchronous diffusion model to generate prior views of an object with consistent style and appearance from different viewpoints through a multi-view guidance enhancement module. The core advantage of this module lies in its ability to first fuse the feature information from various viewpoints of the initial guided object image, and then inject it along with the feature information of the guiding text through a decoupled cross-attention mechanism into the synchronous diffusion model to guide its denoising process. This generates an initial prior view of the object with consistent style and appearance for subsequent training and optimization. This addresses the multifaceted appearance problem inherent in multi-view consistency.

[0019] This invention employs an adaptive weighted multi-view texture constraint method. This method automatically selects the optimal view weight to adapt to different viewpoints, allowing the core viewpoint with more feature information to dominate the optimization direction, while the secondary viewpoint with less information plays an auxiliary constraint role. This ensures that the multi-view rendering results of objects with optimized texture maps do not have obvious inconsistencies in color representation at texture stitching points.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a multi-view consistent 3D object stylization method according to an embodiment of the present invention; Figure 2 This is a network architecture diagram of the multi-view guidance enhancement model according to an embodiment of the present invention; Figure 3 This is an overall architecture diagram of adaptive weighted multi-view texture constraints according to an embodiment of the present invention; Figure 4 This is a comparison chart of the stylistic test results of the present invention and existing technologies on baked bread; Figure 5 This is a comparison chart of the stylization test results of the present invention and existing technologies on boy dolls wearing specific styles of clothing. Detailed Implementation

[0023] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0024] It should be noted that when a component is said to be "fixed to" another component, it can be directly attached to the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component.

[0025] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0026] In the specification, claims, and accompanying drawings of this invention, the terms "first," "second," "third," etc., are used to distinguish different objects and not to describe a particular order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, it may include a series of steps or units, or optionally, steps or units not listed, or other steps or units inherent to these processes, methods, products, or devices.

[0027] The accompanying drawings show only the parts relevant to the invention and not all of them. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0028] The terms “component,” “module,” “system,” “unit,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or distributed between two or more computers. Furthermore, these units can be executed from various computer-readable media on which various data structures are stored. Units can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from a second unit interacting with another unit between a local system, a distributed system, and / or a network; for example, the Internet interacting with other systems via signals).

[0029] Example 1 like Figures 1 to 3 As shown, this embodiment provides a method for stylizing 3D objects with multi-view consistency, which includes: S1: Obtain the target 3D object and its corresponding text description; S2: Construct a multi-view guidance enhancement model, including a text-to-image diffusion model and a multi-view diffusion model. Input the text description into the text-to-image diffusion model to obtain an initial guidance image x. Input the x into the multi-view diffusion model for diffusion and preprocessing to extract the image features of the orthogonal four-view set xn. Further weighted fusion is then used to obtain the image embedding vector E. The text encoder is used to extract the features of the text description and form the text embedding encoding vector T.

[0030] It should be noted that, in this embodiment, the processing procedure of S2 includes: The text description is input into the text-to-image diffusion model to obtain the initial guiding image x. The initial guiding image x is input into a multi-view diffusion model for diffusion to generate a multi-view set xm from different perspectives; The multi-view set xm is preprocessed to obtain the orthogonal four-view set xn; The image encoder based on the multi-view diffusion model encodes the images in the orthogonal four-view set xn, extracts the image features of the orthogonal four-view set xn, and uses different weights to perform weighted fusion to form the image embedding vector E; The text encoder based on the text-to-image diffusion model extracts text features and forms a text embedding encoding vector T.

[0031] Specifically, such as Figure 2 As shown, the text description is input into the text-to-image diffusion model to obtain the initial guiding image x, which is the object reference image in the figure. The object reference image is input into a multi-view diffusion model for diffusion to generate multiple images of the object from different perspectives, and then preprocessed to obtain orthogonal four-view images. The image encoder based on the multi-view diffusion model encodes the images in the orthogonal four views, extracts the image features of the orthogonal four views, and uses different weights to perform weighted fusion to form the image embedding vector E; The text encoder based on the text-to-image diffusion model extracts text features (such as a round loaf of bread with various grain seeds sprinkled on its surface) and forms a text embedding encoding vector T.

[0032] S3: Obtain the geometric network structure and initial texture map of the target 3D object, and render it to obtain a rendering depth map and an object map with texture color from multiple perspectives. S4: Construct a synchronous multi-diffusion model, based on the rendered depth map, the image embedding vector E and the text embedding encoding vector T, to synchronously generate potential image variables at the same angle for each viewpoint, and obtain a prior view with multi-view consistency; S5: After calculating the loss between the prior view and the object map with texture and color, adaptive weights are dynamically calculated based on the loss amount of each viewpoint. The losses of each viewpoint are then weighted and summed to obtain the total loss. The specific formula for calculating the loss between the prior view and the rendered depth map is as follows: in, Number the viewpoints to represent the first viewpoint. One perspective, Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where H is the image height calculated by the input loss, W is the image width calculated by the input loss, h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image. The formula for calculating the total loss is as follows: , in, Number the viewpoints to represent the first viewpoint. One perspective, For the number of viewpoints, For dynamic weights from different perspectives, Set the L2 regularization constraint to 0.1. Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image.

[0033] S6: Update the texture map of the target 3D object through backward gradient propagation. Repeat the update step until the loss converges to obtain a stylized and consistent object texture map.

[0034] It should be noted that, in this embodiment, step S4 includes: The image features of the guiding images from different viewpoints in the orthogonal four-view set xn are weighted and fused using the following formula: in, This represents the fusion weight of each guide image. Indicates the first The image embedding vector of the guide image.

[0035] It should be noted that, in this embodiment, the processing in step S4 includes: 1. Input the rendered depth maps of the target 3D object generated from various viewpoints into the synchronous multidiffusion model, and set the denoising batch size to the total number of viewpoints. N ; 2. The model performs a synchronous denoising process, during which the multi-view guided enhancement module is activated. This module utilizes pre-trained text embedding encoding vectors. T and image embedding vector E As a joint constraint, a priori view with consistency across multiple perspectives is generated.

[0036] This embodiment provides the following specific implementation examples: Multi-view image diffusion networks include image diffusion networks and multi-view diffusion networks. The image diffusion network is currently the mainstream SDXL, version stable-diffusion-xl-base-1.0. After the text description is fed into SDXL, the initial guiding image x is obtained. The initial guide image x is then fed into a multi-view diffusion model to generate other images from different perspectives of the guide image. The multi-view diffusion network uses an MV-Adapter, which takes a single object image as input and outputs a set of images. The specific formula is as follows: , in, It is a collection of generated multiviews. The reasoning process representing the multi-view diffusion model. and It consists of the input reference image and its corresponding text description. It is a camera matrix. The batch number is set to 6. The azimuth (azi) follows In this case, the azi value is set to 0 in the fixed six views. After generating the multi-view set, it is further filtered because the multi-view set contains... The content in the set highly overlaps with the content in the remaining four views in the set. To eliminate viewpoint redundancy and reduce feature conflicts, this case only retains four standard orthogonal reference viewpoint images, i.e. The set of orthogonal four views xn. The image features of images from different perspectives are weighted and fused using the following formula: , in, This represents the fusion weight of each guide image. Indicates the first The image embedding vector of the guide image.

[0037] Considering that the front and back images contain more features and information than the two sides, the view weights are set in the order of front, back, left, and right. The images are then weighted and fused to form the final image embedding vector. At this point, the text description information is then... t The text is fed into the text encoder in SDXL to form the final text embedding encoding vector. T Then the fused image is embedded into the encoding vector. and text embedding encoding vector T The synchronous multidiffusion model is injected with a decoupled cross-attention mechanism via an IP-Adapter, as shown in the following formula: , in, Represents the final attention mechanism The weight representing the text, Represents the Query projection matrix. The Key projection matrix represents the text branch. The Value projection matrix represents the text branch. The key projection matrix represents the image branch. The Value projection matrix represents the image branch. The key-value projection matrix representing the text attention branch is used for text-guided image denoising. Represents the time step of the synchronous multidiffusion model during denoising. t Latent variables, The key-value projection matrix for the image attention branch. As the weight of the image, in this case, it will be Setting them to 0.5 will collectively guide the final denoising process of the synchrotron multidiffusion model. This injection guidance process is significant because it guides the synchrotron multidiffusion model to maintain stylistic consistency during denoising and effectively obtain information about the back and sides of the object, maintaining consistency with the main view during the corresponding view denoising process. This allows the object to obtain a multi-view consistent prior view in the final back projection.

[0038] The geometric network structure and initial texture map of the target 3D object are obtained, and then rendered to obtain a rendering depth map and an object map with textured colors from multiple viewpoints; the number of viewpoints here... Ten views are selected, eight of which are rendered at 45° intervals horizontally, and the other two are the top and bottom views of the target 3D object. These ten views ensure coverage of most of the target 3D object's texture area. After rendering the depth maps of the target 3D object from these ten views, a simultaneous multidiffusion model is used to denoise and generate ten object views at the same angle, which are then used as the ground truth views. Next, the RGB loss for each individual view is calculated using the MSE method, as shown in the following formula: , in, Number the viewpoints to represent the first viewpoint. One perspective, Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where H is the image height calculated by the input loss, W is the image width calculated by the input loss, h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image. After calculating the RGB loss for a single viewpoint, the loss is first weighted using initial weights, which are uniformly set to 0.1 for each viewpoint. After calculating the total loss, backpropagation is performed to update the texture map. And the loss weights for each perspective. The formula is as follows: , in, Number the viewpoints to represent the first viewpoint. One perspective, For the number of viewpoints, For dynamic weights from different perspectives, Set the L2 regularization constraint to 0.1. Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image.

[0039] It should be noted that in this case, the adaptive weight optimizer used is the Adam optimizer, and the learning rate is... The weights are set to 0.01. After the initial weight settings, the optimizer adaptively updates them based on the loss. Furthermore, to avoid over-amplifying the weights of certain views during training and optimization, thus neglecting the losses of other views, this paper also uses an L2 regularization constraint, i.e., in the formula... And set it to 0.1. The adaptive weighted multi-view constraint plays a crucial role in the backprojection process because it can... The weights of the losses from different perspectives are dynamically adjusted so that the core perspectives with more feature information, such as the frontal view, dominate the optimization direction, while the secondary perspectives with less feature information serve as auxiliary constraints. This approach preserves the accuracy of the core perspectives while maintaining the consistency of the secondary perspectives, and the weighted total loss is optimized accordingly. It guides the model to learn a unified texture that is compatible with all viewpoints, rather than a texture for a single viewpoint. The advantage of this is that the final rendered multi-view images of objects will not have obvious gaps in texture seams and colors, thus obtaining a consistent texture map.

[0040] To verify the beneficial effects of this invention compared to existing technologies, the applicant conducted comparative tests on a multi-view consistent 3D object stylization method and apparatus proposed in this invention, as shown in the following figures. Figure 4 and Figure 5 As shown.

[0041] I. Comparative Experiment Setup This invention is compared and evaluated with existing mainstream 3D object stylization methods, including: Synchronous Multi-View Diffusion (SyncMVD), Texture-to-Texture (Tex2Tex), Texture Generation (TEXTure), 3D Rendering (Paint3D), Paint-it, and CasTex, in the same experimental environment. The experiments use the same text description inputs ("crispy baked bread" and "boy doll wearing specific style clothing") and 3D geometric models to objectively evaluate the performance of each method in terms of style reproduction accuracy, multi-view consistency, and texture fit.

[0042] II. Experimental Results and Analysis (I) Performance of the Invention Experimental results show that, compared with the existing technologies, the multi-view consistent 3D object stylization method and apparatus proposed in this invention exhibit superior performance, mainly reflected in: Style capture and mapping accuracy: from visualization results ( Figure 4 , Figure 5As can be seen, compared with existing 3D object stylization methods, this invention can accurately capture the stylistic features contained in the text description and effectively map them onto the surface of the 3D object. At the same time, this method strictly follows the geometric topology of the 3D object, ensuring that the final generated texture map not only conforms to the text semantics but also highly matches the spatial morphology of the 3D object.

[0043] Multi-view consistency: This invention excels in multi-view consistency. Taking the texture stylization effect of "bread" and "boy doll" as examples, the detail representation in both the front and rear views is excellent. The crispy granular texture of the bread surface and the pattern and color of the boy doll's clothing present a natural and realistic visual effect, and maintain a high degree of stylistic consistency and geometric fit under different viewing angles, fully meeting the stylistic requirements of the text description.

[0044] The above experimental results strongly demonstrate the reliability and effectiveness of the multi-view guided enhancement module and adaptive weight multi-view texture constraint method proposed in this invention. The two work together: the former obtains a multi-view set of style cue images through a multi-view diffusion model, and injects this multi-view set along with text descriptions into the synchronous multi-view diffusion process using a decoupled cross-attention mechanism, guiding the model to generate style-consistent object multi-view prior images, providing reliable prior constraints for subsequent texture back-projection; the latter focuses on the multi-view nature of texture mapping. Figure 1 Consistency optimization utilizes an adaptive weighted multi-view texture constraint mechanism to achieve a natural transition and global uniformity of stylized textures on the surface of 3D objects, effectively avoiding texture distortion and viewpoint inconsistency caused by optimization with the same weights. The synergistic effect of these two methods significantly improves the quality and realism of texture mapping in text-driven 3D object stylization tasks.

[0045] (ii) Performance defects of the comparison method In contrast, various existing methods have varying degrees of shortcomings in the aforementioned tasks: The SyncMVD method produces bread with distorted graininess, deviating significantly from the "crispy baking style" described in the text; the facial features of the boy doll are rough, and the stylistic features of the clothing patterns are blurred.

[0046] The texture-to-texture (Tex2Tex) method produces bread textures with blurred layers, and the clothing patterns in the rear view of the doll are severely disjointed from the style in the front view, lacking consistency across multiple perspectives.

[0047] Texture generation method: The generated bread texture is unevenly distributed and the color saturation deviates from the text description; the overall tone of the doll is significantly off.

[0048] The Paint3D method produces bread and dolls with structural distortion and obvious inconsistencies from multiple perspectives.

[0049] The Paint-it method produces bread textures with abnormal spots, which is completely inconsistent with the "uniform and crisp texture" required by the text; the texture and color of the doll's clothing deviate significantly from the target style.

[0050] The Cascaded Texture (CasTex) method produces bread textures with relatively ideal details, but the stylistic features of the doll's face and clothing are blurred, presenting basic color blocks that match the geometric shape, which is difficult to meet the task requirements of text-driven 3D object stylization.

[0051] Based on the above comparative experimental results, the method proposed in this invention is significantly superior to existing technologies in terms of style restoration accuracy, texture naturalness, and multi-view consistency. It can effectively solve technical problems such as texture distortion, view inconsistency, and style deviation in text-driven 3D object stylization, and has good industrial application prospects and practical value.

[0052] Example 2 This embodiment provides a multi-view consistent 3D object stylization device, wherein, for implementing the stylization method described above, it includes: The acquisition module is used to acquire the target 3D object and its corresponding text description; The first construction module is used to construct a multi-view guidance enhancement model, including a text-to-image diffusion model and a multi-view diffusion model. The text description is input into the text-to-image diffusion model to obtain an initial guidance image x. The image is then input into the multi-view diffusion model for diffusion and preprocessing to extract the image features of the orthogonal four-view set xn. The features are further weighted and fused to obtain the image embedding vector E. The text encoder is used to extract the features of the text description and form the text embedding encoding vector T. The rendering module is used to obtain the geometric network structure and initial texture map of the target 3D object, and render it to obtain a rendering depth map and an object map with texture color from multiple perspectives. The second construction module is used to construct a synchronous multi-diffusion model, which generates potential image variables at the same angle for each viewpoint based on the rendered depth map, the image embedding vector E and the text embedding encoding vector T, to obtain a prior view with multi-view consistency. The calculation module is used to calculate the loss of the prior view and the object map with texture and color, and then dynamically calculate the adaptive weight based on the loss of each view. The loss of each view is weighted and summed to obtain the total loss. The update module is used to update the texture map of the target 3D object through backpropagation. The update steps are repeated until the loss converges, resulting in a stylized and consistent object texture map.

[0053] The multi-view consistent 3D object stylization method of the present invention has the following advantages compared with the prior art: This invention employs a multi-view guidance enhancement module to guide a synchronous diffusion model in generating prior views of an object with consistent style and appearance from different viewpoints for subsequent model optimization and training. The core advantage of this module lies in its ability to first fuse the feature information from various viewpoints of the initial guided object image, and then, through a decoupled cross-attention mechanism, inject it along with the feature information of the guiding text into the synchronous diffusion model to guide its denoising process. This generates an initial prior view of the object with consistent style and appearance for subsequent training and optimization. This addresses the multifaceted appearance phenomenon under the multi-view consistency problem.

[0054] This invention employs an adaptive weighted multi-view texture constraint method. This method automatically selects the optimal view weight to adapt to different viewpoints, allowing the core viewpoint with more feature information to dominate the optimization direction, while the secondary viewpoint with less information plays an auxiliary constraint role. This ensures that the multi-view rendering results of objects with optimized texture maps do not have obvious inconsistencies in color representation at texture stitching points.

[0055] Optionally, embodiments of the present invention also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a multi-view consistent 3D object stylization method and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0056] This invention also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described multi-view consistent 3D object stylization method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0057] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0058] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0059] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0060] Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. The reference to "embodiment" in this specification means that a specific feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily indicate the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in this specification can be combined with other embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0061] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-view consistent three-dimensional object stylization method, comprising: include: S1: Obtain the target 3D object and its corresponding text description; S2: Construct a multi-view guidance enhancement model, including a text-to-image diffusion model and a multi-view diffusion model. Input the text description into the text-to-image diffusion model to obtain an initial guidance image x. Input the image x into the multi-view diffusion model for diffusion and preprocessing to extract the image features of the orthogonal four-view set xn. Further weighted fusion is then used to obtain the image embedding vector E. The text encoder is used to extract the features of the text description and form the text embedding encoding vector T. S3: Obtain the geometric network structure and initial texture map of the target 3D object, and render it to obtain a rendering depth map and an object map with texture color from multiple perspectives. S4: Construct a synchronous multi-diffusion model, based on the rendered depth map, the image embedding vector E and the text embedding encoding vector T, to synchronously generate potential image variables at the same angle for each viewpoint, and obtain a prior view with multi-view consistency; S5: After calculating the loss of the prior view and the object map with texture and color, the adaptive weight is dynamically calculated based on the loss of each view, and the loss of each view is weighted and summed to obtain the total loss. S6: Update the texture map of the target 3D object through backward gradient propagation. Repeat the update step until the loss converges to obtain a stylized and consistent object texture map.

2. The multi-view consistent three-dimensional object stylization method of claim 1, wherein, The processing procedure of S2 includes: The text description is input into the text-to-image diffusion model to obtain the initial guiding image x. The initial guiding image x is input into a multi-view diffusion model for diffusion to generate a multi-view set xm from different perspectives; The multi-view set xm is preprocessed to obtain the orthogonal four-view set xn; The image encoder based on the multi-view diffusion model encodes the images in the orthogonal four-view set xn, extracts the image features of the orthogonal four-view set xn, and uses different weights to perform weighted fusion to form the image embedding vector E; The text encoder based on the text-to-image diffusion model extracts text features and forms a text embedding encoding vector T.

3. The multi-view consistent 3D object stylization method according to claim 1, characterized in that, In S5, the specific formula for calculating the loss between the prior view and the rendered depth map is as follows: in, Number the viewpoints to represent the first viewpoint. One perspective, Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where H is the image height calculated by the input loss, W is the image width calculated by the input loss, h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image.

4. The multi-view consistent 3D object stylization method according to claim 3, characterized in that, In S5, the formula for calculating the total loss is as follows: ， in, Number the viewpoints to represent the first viewpoint. One perspective, For the number of viewpoints, For dynamic weights from different perspectives, Set the L2 regularization constraint to 0.

1. Representing the The pixel values ​​of the textured and colored object image generated from each viewpoint. Representing the The pixel values ​​of potential image variables under each viewpoint, where h is the vertical pixel coordinate of the image, w is the horizontal pixel coordinate of the image, and c is the color channel number of the image.

5. The multi-view consistent 3D object stylization method according to claim 1, characterized in that, S4 includes: The image features of the guiding images from different viewpoints in the orthogonal four-view set xn are weighted and fused using the following formula: in, This represents the fusion weight of each guide image. Indicates the first The image embedding vector of the guide image.

6. The multi-view consistent 3D object stylization method according to claim 1, characterized in that, The processing procedure of S4 includes: The rendered depth maps of the target 3D object generated from various viewpoints are uniformly input into the synchronous multidiffusion model, and the denoising batch size is set to the total number of viewpoints. N ; The model performs a synchronous denoising process, during which a multi-view guided enhancement module is activated, which utilizes pre-trained text embedding encoding vectors. T and image embedding vector E As a joint constraint, a priori view with consistency across multiple perspectives is generated.

7. A multi-view consistent 3D object stylization device, characterized in that, For implementing the stylization method as described in any one of claims 1 to 6, comprising: The acquisition module is used to acquire the target 3D object and its corresponding text description; The first construction module is used to construct a multi-view guidance enhancement model, including a text-to-image diffusion model and a multi-view diffusion model. The text description is input into the text-to-image diffusion model to obtain an initial guidance image x. The image is then input into the multi-view diffusion model for diffusion and preprocessing to extract the image features of the orthogonal four-view set xn. The features are further weighted and fused to obtain the image embedding vector E. The text encoder is used to extract the features of the text description and form the text embedding encoding vector T. The rendering module is used to obtain the geometric network structure and initial texture map of the target 3D object, and render it to obtain a rendering depth map and an object map with texture color from multiple perspectives. The second construction module is used to construct a synchronous multi-diffusion model, which generates potential image variables at the same angle for each viewpoint based on the rendered depth map, the image embedding vector E and the text embedding encoding vector T, to obtain a prior view with multi-view consistency. The calculation module is used to calculate the loss of the prior view and the object map with texture and color, and then dynamically calculate the adaptive weight based on the loss of each view. The loss of each view is weighted and summed to obtain the total loss. The update module is used to update the texture map of the target 3D object through backpropagation. The update steps are repeated until the loss converges, resulting in a stylized and consistent object texture map.

8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multi-view consistent 3D object stylization method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the multi-view consistent 3D object stylization method as described in any one of claims 1 to 6.

Images