Method for training three-dimensional content generation model, electronic device, and storage medium

Abstract

The present disclosure relates to a method for training a three-dimensional content generation model, an electronic device and a storage medium, wherein the method comprises: obtaining a first sample, a second sample and a third sample, the first sample being composed of data of different modalities in pairs, the second sample comprising an image with perspective information, and the third sample being a sequence comprising text data, an image with perspective information and three-dimensional geometric data; training an initial model using the first sample to learn visual appearance rules and geometric structure priors, to obtain a first model; training the first model using the first sample and the second sample to learn the projection relationship between three-dimensional geometry and perspective images, to obtain a second model; and training the second model using the third sample to learn the alignment relationship between text, images and three-dimensional geometry, to obtain a three-dimensional content generation model. The present scheme improves the multi-perspective geometric consistency and three-dimensional structure fidelity of generating three-dimensional content using the three-dimensional content generation model.

Description

Technical Field

[0001] This disclosure relates to the field of Artificial Intelligence Generated Content (AIGC) technology, and in particular to a training method, electronic device, and storage medium for a three-dimensional content generation model. Background Technology

[0002] Cross-modal 3D content generation aims to directly generate 3D content based on text input. Among related technologies, text-driven 3D content generation methods are mainly divided into two categories: cascaded and native. Cascaded methods first generate a 2D image from text, then use fractional distillation, iterative optimization, etc., to upscale the 2D result into 3D content. This type of method relies on intermediate image proxies, making it prone to problems such as geometric distortion, inconsistencies across multiple perspectives, and semantic-structural disconnect. Furthermore, it suffers from slow inference speed and difficulty in achieving end-to-end controllable generation. Native 3D generation models can directly map text to 3D models, but are limited by the scarcity of high-quality 3D asset data, the lack of uniformity in text-image-3D modal representation, and interference between appearance generation and geometry generation tasks. This results in weak model generalization ability, insufficient generation stability, and low semantic alignment accuracy.

[0003] It is evident that existing text-driven 3D content generation solutions, under conditions of scarce 3D data, struggle to simultaneously achieve cross-modal semantic alignment accuracy and 3D geometric fidelity, easily leading to problems such as semantic misalignment and multi-view geometric distortion. Therefore, how to improve the geometric reliability and fidelity of 3D generation under limited 3D data conditions has become an urgent problem to be solved. Summary of the Invention

[0004] The main objective of this disclosure is to propose a training method, electronic device, and storage medium for a 3D content generation model, aiming to solve the technical problem in the prior art where the geometric fidelity of 3D content generation is difficult to meet the requirements due to the scarcity of 3D data.

[0005] Firstly, this disclosure provides a training method for a 3D content generation model, including: Acquire a first sample, a second sample, and a third sample. The first sample is composed of pairs of data from different modalities. The second sample includes an image with viewpoint information. The third sample is a sequence of text data, an image with viewpoint information, and three-dimensional geometric data. The different modalities include text, images, and three-dimensional geometry. The initial model is trained using the first sample, so that the initial model learns the visual appearance rules and geometric structure priors, thus obtaining the first model; The first model is trained using the first sample and the second sample, so that the first model learns the projection relationship between three-dimensional geometry and viewpoint image, and a second model is obtained. The second model is trained using the third sample, enabling it to learn the alignment relationships between text, images, and 3D geometry, thus obtaining a 3D content generation model.

[0006] Secondly, this disclosure provides a training apparatus for a three-dimensional content generation model, comprising: The sample acquisition module is used to acquire a first sample, a second sample, and a third sample. The first sample is composed of pairs of data from different modalities. The second sample includes an image with viewpoint information. The third sample is a sequence of text data, an image with viewpoint information, and three-dimensional geometric data. The different modalities include text, images, and three-dimensional geometry. The pre-training module is used to train the initial model using the first sample, so that the initial model learns the visual appearance rules and geometric structure priors to obtain the first model; The continuous training module is used to train the first model using the first sample and the second sample, so that the first model learns the projection relationship between three-dimensional geometry and viewpoint image to obtain the second model; The supervised training module is used to train the second model using the third sample, so that the second model learns the alignment relationship between text, images and 3D geometry, and obtains a 3D content generation model.

[0007] Thirdly, this disclosure provides an electronic device, including: one or more processors and a memory, wherein the memory stores a program or instructions, which, when executed by the processor, implement the training method for the three-dimensional content generation model as described in the first aspect.

[0008] Fourthly, this disclosure provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the training method for the three-dimensional content generation model as described in the first aspect.

[0009] Fifthly, this disclosure provides a computer program product, the computer program product including computer program information, which, when run by a processor, causes the processor to execute the training method for the three-dimensional content generation model as described in the first aspect.

[0010] The technical solution provided in this disclosure has at least the following advantages compared with the prior art: In this disclosure, a first sample, a second sample, and a third sample are obtained. The first sample is composed of pairs of data from three different modalities: text, image, and 3D geometry. The second sample includes an image with viewpoint information. The third sample is a sequence of text data, an image with viewpoint information, and 3D geometric data. In the first training phase, the initial model is trained using the first sample to learn visual appearance rules and geometric structure priors, resulting in a first model. In the second training phase, the first model is further trained using the first and second samples to learn the projection relationship between 3D geometry and viewpoint images, resulting in a second model. In the third training phase, the second model is trained using the third sample, resulting in a second model. The model learns the alignment relationships between text, images, and 3D geometry to obtain a trained 3D content generation model. Therefore, this scheme adopts a three-stage hierarchical progressive training paradigm, decomposing the complex unified cross-modal generation task into three decoupled progressive stages: intermodal basic alignment, spatial enhancement, and closed-loop integration. This achieves step-by-step learning of cross-modal basic alignment, multi-view spatial perception, and closed-loop consistency constraints, enabling the model to maximize the use of massive 2D data to learn geometric structure and visual appearance priors under limited 3D data conditions. This effectively avoids gradient conflicts caused by the large differences in the scale of heterogeneous modal data and the mutual interference between appearance and geometric tasks, and significantly improves the multi-view geometric consistency and 3D structural fidelity of the 3D content generated by the 3D content generation model. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a training method for a three-dimensional content generation model provided as an exemplary embodiment of the present disclosure.

[0014] Figure 2 A flowchart illustrating a method for training a three-dimensional content generation model provided as another exemplary embodiment of this disclosure.

[0015] Figure 3 This is a schematic diagram of the structure of a training device for a three-dimensional content generation model provided in an embodiment of this disclosure. Detailed Implementation

[0016] To better understand the above-described objectives, features, and advantages of this disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the described embodiments are only some, not all, of the embodiments of this disclosure. The specific embodiments described herein are merely for explaining this disclosure and not for limiting it. Unless otherwise specified, the embodiments of this disclosure and the features within them can be combined with each other. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure.

[0017] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0018] Currently, text-driven 3D generation technologies still face numerous insurmountable technical bottlenecks. For example, cascaded training methods rely on intermediate image proxies for dimensionality enhancement, which easily leads to geometric distortion and inconsistencies across multiple perspectives. While native training methods can directly map, they suffer from weak generalization ability and insufficient generation stability due to the scarcity of high-quality 3D data and the mutual interference between appearance generation and geometry generation tasks. Furthermore, in terms of training paradigms, existing methods mostly employ single-stage or simple two-stage training approaches, failing to implement hierarchical and progressive learning for cross-modal basic alignment, 3D spatial perception, and closed-loop consistency constraints. This makes it impossible to fully leverage the visual and geometric priors of massive 2D data under limited 3D data conditions, resulting in 3D generation geometric fidelity and robustness that fail to meet practical application requirements.

[0019] To address the problems in related technologies, such as unreasonable training paradigms for 3D generative models, scarcity of 3D data, strong intermodal task interference, and poor multi-view geometric consistency, making it difficult to efficiently utilize 2D data to achieve hierarchical alignment and closed-loop learning within a unified framework to improve the quality of 3D content generation, this disclosure proposes a three-stage progressive training process for the 3D content generation model: pre-training, continuous training, and supervised fine-tuning. In the pre-training stage, the model is trained using first samples composed of pairwise combinations of text, image, and 3D geometry data from three different modalities. This establishes basic alignment relationships between modalities, fully absorbing prior visual appearance and geometric structure from massive amounts of 2D data while avoiding interference between appearance and geometric tasks, enabling the model to initially possess the ability to directly generate 3D content from text. In the continuous training stage… This approach introduces perspective information on top of pre-trained weights, retains all pre-training tasks, and adds a new perspective-conditional generation task. Utilizing multi-view rendered images and 3D data, it enhances the model's 3D spatial perception and multi-view projection consistency, enabling the model to grasp the geometric and appearance correspondences of objects under different perspectives, thus significantly improving the stability of the generated 3D structures. In the supervised fine-tuning stage, an interleaved sequence of aligned text data, images with perspective information, and 3D geometric data is used as the third sample. Based on continuous training, a closed-loop generation task is executed in an end-to-end autoregressive manner, forcing the model to follow the semantic-visual-geometric cyclic constraint. This ensures high alignment of text, images, and 3D geometry throughout the process, ultimately integrating into a stable, unified, and high-fidelity text-driven 3D generation and editing capability. Compared to the aforementioned cascaded and single-stage native methods, this approach achieves significant breakthroughs in generation stability, generalization ability, and semantic alignment accuracy.

[0020] The following detailed explanation, with reference to the accompanying drawings, details the specific implementation methods of the training method, electronic device, and storage medium for the three-dimensional content generation model disclosed herein.

[0021] Figure 1 This is a flowchart illustrating a training method for a three-dimensional content generation model provided in an exemplary embodiment of the present disclosure. The method can be executed by a training device for the three-dimensional content generation model provided in the embodiments of the present disclosure. The training device for the three-dimensional content generation model can be implemented in software and / or hardware and can be integrated into an electronic device.

[0022] like Figure 1 As shown, the training method for this 3D content generation model may include the following steps: Step 101: Obtain the first sample, the second sample, and the third sample. The first sample is composed of pairs of data from different modalities. The second sample includes an image with viewpoint information. The third sample is a sequence of text data, an image with viewpoint information, and three-dimensional geometric data. The different modalities include text, images, and three-dimensional geometry.

[0023] Step 102: Use the first sample to train the initial model, so that the initial model learns the visual appearance rules and geometric structure priors, and obtains the first model.

[0024] Step 103: Train the first model using the first and second samples, so that the first model learns the projection relationship between the three-dimensional geometry and the viewpoint image, and obtains the second model.

[0025] Step 104: Use the third sample to train the second model, so that the second model learns the alignment relationship between text, image and 3D geometry, and obtains a 3D content generation model.

[0026] In step 101, three types of training samples can be collected from a multi-source data warehouse, annotation platform, or offline asset repository. The first sample can be at least two of the following: text-image pairing samples, text-3D geometry pairing samples, image-3D geometry pairing samples, and 3D geometry-image pairing samples. Accordingly, the pairwise combinations in the first sample are used to support cross-modal basic alignment learning. The second sample can include images with viewpoint information. For example, the second sample can be at least two of the following: text-image configuration pairing samples, image-image pairing samples, and 3D geometry-image pairing samples. The viewpoint information can be represented by learnable viewpoint tokens. Viewpoint tokens can, for example, correspond to multiple standard viewpoints. Exemplarily, they can include learnable viewpoint tokens corresponding to six standard viewpoints: front, back, left, right, top, and bottom. This allows the model to explicitly perceive the mapping relationship between the observation direction and the 2D projected appearance during training, providing conditions for the model to learn the projection relationship between 3D geometry and multi-view images, ensuring that the generated 3D model maintains structural stability and reasonable projection under any viewpoint. The third sample can be an interleaved sequence of data organized in a predetermined order. This interleaved sequence can simultaneously contain text data, images with viewpoint information, and 3D geometric data to facilitate subsequent end-to-end closed-loop supervised training. The third sample can consist of artificially synthesized and fully aligned data. For example, a text description of a 3D target object can be generated first, then multi-view images (i.e., images with different viewpoint information) can be rendered based on the target object, and the corresponding text tokens, viewpoint tokens, image tokens, and 3D geometry tokens can be sequentially concatenated into a unidirectional long sequence to obtain a third sample, which is used to perform end-to-end autoregressive generation.

[0027] For example, text data can be natural language descriptions, attribute labels, style phrases, scene descriptions, or object constraint statements; image data can be a single two-dimensional image, a multi-view image set, a rendered image, or an annotated image; three-dimensional geometric data can be a three-dimensional mesh, point cloud, topological model, voxel representation, implicit geometric representation, or its discretization result.

[0028] In step 102, the initial model can be a neural network architecture integrating natural language processing, computer vision, and 3D geometry processing technologies. For example, it may include an encoder, a fusion module, and a decoder. The encoder is used to uniformly map data from three heterogeneous modalities—text, image, and 3D geometry—to a shared discrete token space, thereby progressively enhancing the model's capabilities through hierarchical training within this unified discrete token space. The fusion module is used to align features from different modalities, and the decoder is used to perform predictive learning based on the features output by the fusion module.

[0029] As an example, the encoder can combine text encoding using two different methods (e.g., CLIP and Qwen3) to convert the text into a fixed-length semantic token. It can also compress an image into a one-dimensional discrete visual token using a two-stage image encoder, and convert 3D geometric data into a one-dimensional structural token using a geometric encoder (e.g., Cube3D). These three tokens are compatible in length, format, and feature space, and can be directly concatenated into a single sequence for autoregressive modeling. By mapping different modalities to a shared discrete token space, text, images, and 3D geometry can form compatible representations in terms of length, format, and feature space. This allows for autoregressive modeling within the same framework, eliminating representation incompatibility issues between heterogeneous modalities and providing a unified foundation for subsequent hierarchical training.

[0030] When training the initial model using the first sample, the initial model uses heterogeneous paired data as supervision signals. Based on at least two types of paired samples from text-image, text-3D geometry, image-3D geometry, and 3D geometry-image, it performs independent cross-modal generation tasks without modal concatenation to avoid gradient interference between different tasks. For example, it can perform at least two of the following tasks: text-to-image generation, text-to-3D geometry generation, image-to-3D geometry generation, and 3D geometry-to-image generation. By training the initial model using independent cross-modal generation tasks, the model can simultaneously establish basic alignment relationships between text semantics and visual appearance, text semantics and 3D geometry, and visual appearance and 3D geometry. This reduces gradient interference between appearance generation tasks and geometry generation tasks, enabling the model to initially possess the basic ability to directly generate 3D models from text.

[0031] In some embodiments, a temperature-weighted sampling strategy can be employed during training to balance the distribution of data of different scales, increasing the sampling probability of small sample data (such as text-3D geometry pairing data). This allows the model to learn visual appearance rules under the supervision of massive 2D data and geometric structure priors under the supervision of limited 3D data, even under conditions of uneven data distribution. This avoids model training biased towards large-scale 2D tasks while neglecting the establishment of 3D generation capabilities. Visual appearance rules reflect the statistical regularities and representational relationships related to the 2D visible representation of the generated object or scene, and may include, but are not limited to, color distribution, material texture, edge contours, lighting and darkness, local details, and overall composition. Geometric structure priors reflect prior knowledge related to the spatial morphology, structural topology, local connectivity, and overall scale relationships of 3D objects. These priors can originate from real 3D asset data or from learning pairing mappings between images and geometry. After this pre-training phase, the initial model learns the visual appearance rules and geometric structure priors, resulting in the first model.

[0032] In step 103, the first model is trained using the first and second samples. This involves retaining all training tasks from step 102 while adding at least one new generation task with viewpoint conditions as input. For example, at least one of the following could be added: text-to-viewpoint image generation, image-to-viewpoint image generation, and 3D geometry-to-viewpoint image generation. When the image with viewpoint information from the second sample is input into the first model, the observation direction condition can be explicitly injected using the corresponding viewpoint token, enabling the model to establish a correlation between image appearance and viewpoint changes. By adding viewpoint information as an explicit conditional injection into the generation sequence on top of the existing cross-modal tasks, the consistency of 3D spatial perception and multi-view projection is strengthened. The model can further learn the 2D projection differences of the same 3D object under different observation directions, and the projection correspondence between 3D geometry and multi-view images, while retaining the learned cross-modal alignment capabilities.

[0033] In some embodiments, learning the projection relationship between 3D geometry and viewpoint images can specifically manifest as learning the 2D mapping rules of geometric surfaces, local convexity / concaveness, spatial occlusion, material representation, and lighting response under different viewpoints. Here, the viewpoint image is an image containing viewpoint information, which can be an actual photographed image, an image rendered from 3D data, or a multi-view image output by a simulator or content creation system. The projection relationship can refer to the correspondence between a 3D geometric representation and a 2D image representation under given observation conditions. This correspondence can be an explicit geometric projection relationship or a conditional mapping relationship implicitly learned by the model. Projection relationships can be established through various methods such as orthogonal projection, perspective projection, differentiable rendering mapping, or viewpoint conditional autoregressive generation.

[0034] Optionally, during the continuous training process in step 103, only the viewpoint embedding vector can be initialized, while other parameters are frozen and fine-tuned. This low learning rate ensures stable optimization, allowing the generated 3D geometric data to maintain structural stability and reasonable projection from any viewpoint, thus enhancing the reliability of the generated 3D content from a spatial perception perspective. The low learning rate can be understood as a smaller update rate compared to the basic training learning rate in step 102. For example, it can be less than the basic training learning rate, or it can be set to 0.1 to 0.5 times the basic training learning rate. This range is merely an example and does not constitute a limitation. By initializing only the viewpoint embedding and optimizing with a low learning rate, the model can master the geometric and appearance correspondences of objects under different viewpoints. It efficiently learns viewpoint-related spatial perception capabilities while stably retaining existing abilities, significantly improving the stability of the generated 3D structure. After this continuous training phase, the first model learns the projection relationship between the 3D geometry and the viewpoint image, resulting in the second model.

[0035] In step 104, the third sample is a sequence comprising text data, images with viewpoint information, and 3D geometric data. This sequence can be, for example, a long, unidirectional sequence formed by sequentially concatenating text tokens, viewpoint tokens, image tokens, and 3D geometry tokens. This allows the second model to perform a closed-loop generation task in an end-to-end autoregressive manner. The second model can perform token-by-token prediction according to the generation chain from text to multi-view images to 3D geometric data, forcing the model to complete coherent reasoning of semantic understanding, image generation, 3D reconstruction, and multi-view rendering within a unified process, ensuring high alignment of text, images, and 3D geometry throughout. The text portion can describe object categories, structural features, material properties, emotional style, or scene semantics; the image portion can consist of multi-view rendered images with corresponding viewpoint identifiers; and the 3D geometry portion can be a 3D mesh or discrete geometry token sequence that strictly corresponds to the text and images.

[0036] In this embodiment, the alignment relationship between text, images, and 3D geometry can refer to the correspondence between different modalities of data in terms of semantic theme, visual appearance, spatial structure, emotional tone, style expression, or entity consistency. Specifically, the alignment between text and images can be reflected in the correspondence between the described object and the image content; the alignment between text and 3D geometry can be reflected in the correspondence between the text description structure and the geometric shape; and the alignment between images and 3D geometry can be reflected in the consistency between the view appearance and the 3D structure under different viewing angles.

[0037] When training the second model using third samples, cross-entropy loss can be applied to supervise the target generated tokens. Through closed-loop supervision, semantics, vision, and geometry are aligned throughout the process, integrating scattered cross-modal capabilities into a stable, unified, high-fidelity, end-to-end text-driven 3D generation and editing capability, thus obtaining a stable 3D content generation model. In practical applications, inputting text descriptions into the trained 3D content generation model can yield high-fidelity 3D content.

[0038] In the above training process, the pre-training stage in step 102 provides basic semantic-visual-geometric alignment capabilities, the continuous training stage in step 103 provides spatial mapping capabilities from 3D structures to multi-view images, and the supervised training stage in step 104 ensures high alignment of text, images, and 3D geometry throughout the entire process. Through the above three-stage hierarchical and progressive unified training paradigm, the unified cross-modal generation of text, images, and 3D is modeled as a progressive learning process of unified representation, decoupled learning, spatial enhancement, and closed-loop constraints. This decomposes the complex cross-modal generation task into hierarchical and progressive sub-tasks, gradually strengthening the model's capabilities within a unified discrete token space. This improves the geometric fidelity and cross-modal consistency of the model under conditions of relatively limited 3D data.

[0039] The training method for a 3D content generation model according to embodiments of this disclosure acquires a first sample, a second sample, and a third sample. The first sample is composed of pairs of data from three different modalities: text, image, and 3D geometry. The second sample includes an image with viewpoint information. The third sample is a sequence of text data, an image with viewpoint information, and 3D geometry data. In the first training stage, the initial model is trained using the first sample to learn visual appearance rules and geometric structure priors, thus obtaining a first model. In the second training stage, the first model is further trained using the first and second samples to learn the projection relationship between 3D geometry and the viewpoint image, thus obtaining a second model. In the third training stage, the third sample is used to train the second model. The second model is trained to learn the alignment relationships between text, images, and 3D geometry, resulting in a trained 3D content generation model. Therefore, this scheme adopts a three-stage hierarchical progressive training paradigm, decomposing the complex unified cross-modal generation task into three decoupled progressive stages: intermodal basic alignment, spatial enhancement, and closed-loop integration. This achieves step-by-step learning of cross-modal basic alignment, multi-view spatial perception, and closed-loop consistency constraints, enabling the model to maximize the use of massive 2D data to learn geometric structure and visual appearance priors under limited 3D data conditions. This effectively avoids gradient conflicts caused by significant differences in the scale of heterogeneous modal data and interference between appearance and geometric tasks, greatly improving the multi-view geometric consistency and 3D structural fidelity of 3D content generated using the 3D content generation model.

[0040] Currently, in related text-driven 3D content generation technologies, existing methods generally use autoregressive cross-entropy, reconstruction loss, and geometric distance loss as core optimization objectives in terms of supervision mechanisms. These supervision mechanisms only focus on surface sequence or structural matching between modalities, lacking consistency constraints on deep semantics, emotion, and style. This easily leads to problems such as misalignment between text semantics and 3D form, conflicting emotional tones, and inconsistent styles. Based on this, the scheme disclosed in this paper introduces semantic resonance loss during the supervised training phase. Designed based on human cross-sensory cognitive mechanisms, it relies on the Qwen-VL general large-scale visual language model to construct cross-modal semantic and emotional feature extraction capabilities. This achieves deep semantic and emotional unity across text, image, and 3D geometry modalities, breaking through the limitations of traditional token-level supervision that only focuses on local sequence matching. It solves pain points such as semantic misalignment, emotional conflict, and discrepancies between entity form and text description in multimodal generation. Therefore, in one optional implementation of this disclosure, such as... Figure 2 As shown, based on the foregoing embodiments, step 104 may include the following sub-steps: Step 201: Input the text data of the third sample and the image with viewpoint information into the second model to obtain the three-dimensional prediction data output by the second model based on the text data and the image with viewpoint information.

[0041] Step 202: Determine the structural loss value based on the three-dimensional prediction data and the three-dimensional geometric data of the third sample.

[0042] Step 203: Determine the semantic loss value based on the text data of the third sample, the image with viewpoint information, and the 3D geometric data.

[0043] Step 204: Determine the total loss value based on the structural loss value and the semantic loss value.

[0044] Step 205: If the total loss value is greater than the preset value, adjust the model parameters of the second model and perform iterative training until the total loss value is less than or equal to the preset value, and obtain the trained 3D content generation model.

[0045] In step 201, the text data and image with viewpoint information of the third sample are input into the second model to obtain the 3D prediction data output by the second model based on the text data and viewpoint information. This 3D prediction data can be, for example, a sequence of 3D geometric tokens output by the model's autoregressive prediction. In step 202, after obtaining the 3D prediction data output by the second model, it can be compared with the 3D geometric data in the third sample to calculate the loss and obtain the structural loss value. The loss function can be the cross-entropy loss function, or other sequence supervision methods that can measure the difference between the predicted discrete tokens and the target discrete tokens; this disclosure does not impose any limitations on this. As a specific implementation, the 3D geometric data can be regarded as the target token sequence, and the predicted distribution output by the model can be compared with the target tokens position by position to obtain the cross-entropy loss value. The cross-entropy loss value is used to constrain the consistency between the predicted sequence and the real geometric sequence, enabling the model to accurately predict the target 3D geometric sequence at the token level, ensuring that the generated result maintains a high degree of matching with the real 3D geometric data at the structural level, and providing accurate supervision signals for surface sequence matching.

[0046] In step 203, a semantic loss value is determined based on the text data, image with viewpoint information, and 3D geometric data of the third sample. This semantic loss value measures the consistency of text, image, and 3D geometry within a shared semantic sentiment space. For example, the semantic loss value can be represented by the cosine similarity between features of the three modalities. The shared semantic sentiment space can be established through a pre-trained visual language model or by mapping features from different modalities to the same dimension using a unified projection layer. By calculating the semantic loss value based on data from different modalities within the same sample, the training objective can be extended from surface token matching to deep semantic, sentiment, and style consistency.

[0047] In some embodiments, determining the semantic loss value based on each modal data in the third sample may include: encoding the text data, the image with viewpoint information, and the three-dimensional geometric data of the third sample respectively to obtain corresponding text features, image features, and three-dimensional features; determining the text-image alignment based on the text features and image features; determining the text-three-dimensional alignment based on the text features and three-dimensional features; and determining the semantic loss value based on the text-image alignment and text-three-dimensional alignment.

[0048] In encoding text data, a text encoder can be used to extract semantic, sentiment, and stylistic features, resulting in text features. Similarly, an image encoder can be used to encode images with perspective information, extracting perspective features, visual semantics, compositional texture, and lighting characteristics, resulting in image features. For encoding 3D geometric data, deep learning networks can be used to directly extract features, or the 3D geometric data can be mapped into multiple 2D slices, and then an image encoder can be used to extract features from these slices, which are then aggregated to obtain the final 3D features.

[0049] In some embodiments, to avoid the problems of feature space mismatch and semantic tone fragmentation caused by multi-model fusion, when encoding data of different modalities, a single pre-trained Vision-Language Model (VLM) can be used as the sole universal semantic base. A modality normalization preprocessing strategy can be employed to achieve unified feature encoding for the three heterogeneous modalities. Therefore, in this embodiment, when acquiring text features, image features, and 3D features, the visual language model can be used to extract semantic features from the text data of the third sample to obtain text features; the visual language model can be used to extract visual features from images with viewpoint information to obtain image features; and the 3D geometric data can be converted into multiple 2D images, and the visual language model can be used to extract visual features from the multiple 2D images to obtain 3D features.

[0050] The visual language model can employ the Qwen-VL model or other visual language models capable of handling both text and image inputs. Taking the Qwen-VL model as an example, for text and 2D image modalities, Qwen-VL's native pre-trained encoding capabilities can be directly reused: the text encoding branch deeply mines the semantic themes, sentiment tendencies, and stylistic features of the input text, accurately capturing the core semantic connotations and emotional orientations of the text; the image encoding branch relies on Qwen-VL's powerful fine-grained visual perception capabilities to accurately decouple underlying visual elements such as image color, texture, composition, and lighting, extracting corresponding high-order emotional representations and stylistic features. For the 3D model modal, the approach of building an independent 3D encoder is abandoned, and a 3D model 2D slicing normalization strategy is adopted to complete modal adaptation: multi-angle, full-coverage 2D orthogonal slicing and perspective projection are performed on the 3D mesh and topological model, generating batch 2D slice image sets containing the overall structure of the model, local details, material textures, and spatial topological relationships. The core features such as 3D geometry, material, and spatial relationships are completely preserved in the 2D slices, achieving lossless conversion from 3D modality to 2D visual modality. The preprocessed 2D slice images can be directly accessed by the visual encoding branch of Qwen-VL. Through the model's native visual feature extraction logic, the structural semantics, material features, topological relationships, and visual sentiment representations corresponding to the 3D geometric data are captured simultaneously to obtain 3D features.

[0051] In this embodiment of the disclosure, by using the same visual language model to extract features from the data of three modalities—text, image, and 3D geometry—a unified model, unified feature dimension, and unified semantic logic encoding for the three modalities are achieved. This ensures that the 3D features, text features, and image features are in the same feature space, eliminates the representation differences between modalities, and avoids the feature space mismatch problem caused by introducing an additional encoder.

[0052] After obtaining text features, image features, and 3D features, text-image alignment can be determined based on text and image features, and text-3D alignment can be determined based on text and 3D features. Alignment characterizes the proximity of two or more modal features and can be determined through at least one of similarity calculation, distance calculation, joint scoring, mapping error, or other differentiable evaluation functions. Text-image alignment measures the consistency between text semantics and image visual content, constraining the image to maintain consistency with the input text in semantic connotation, emotional tone, and stylistic expression, avoiding semantic misalignment or emotional conflict between the generated image and the text description. Text-3D alignment measures the consistency between text semantics and 3D geometric content, constraining the generated 3D geometry to maintain consistency with the input text in structural semantics, material features, and entity form, avoiding discrepancies between the 3D model's form and the text description.

[0053] In some embodiments, determining text-image alignment based on text features and image features includes: determining the cosine similarity between text features and image features; determining the Euclidean distance between text features and image features; and determining the text-image alignment based on cosine similarity and Euclidean distance. Cosine similarity measures the degree of alignment between two feature vectors in direction, capturing the degree of matching between text and image in semantic theme and sentiment, and is robust to scale changes in feature vectors. When the object attributes and style of the text semantics are relatively consistent with the visual content of the image, their vector directions are usually closer, resulting in a higher cosine similarity. Euclidean distance measures the absolute distance between two feature vectors in the feature space, capturing the deviation between text and image in feature strength and fine-grained semantic details, complementing cosine similarity. When image content deviates from the text description, the spacing between feature points in the shared space may increase, thus increasing the Euclidean distance accordingly. Furthermore, the text-image alignment is determined based on cosine similarity and Euclidean distance. For example, the text-image alignment can be expressed as: text-image alignment = a × cosine similarity + b × (1 - Euclidean distance), so that when the difference between text features and image features is large, a smaller text-image alignment is obtained, where the values ​​of a and b are greater than 0 and not greater than 1.

[0054] It should be noted that, in this embodiment, after completing the multimodal feature encoding, textual semantic features (i.e., text features), native image visual sentiment features (i.e., image features), and 3D slice-derived visual features (i.e., 3D features) can be mapped to the same shared semantic sentiment space through a unified projection layer, aligning the feature dimensions of the three different modalities to facilitate subsequent calculations of cosine similarity and Euclidean distance. Furthermore, cosine similarity and Euclidean distance are only used as examples to illustrate this disclosure and should not be considered as limitations on this disclosure. In addition to the two methods mentioned above, Manhattan distance, Mahalanobis distance, kernel function distance, dual-tower matching scores, or other similarity evaluation methods can be used as supplements or alternatives.

[0055] In this embodiment of the disclosure, by determining the cosine similarity and Euclidean distance between text features and image features, the text-image alignment is determined based on the cosine similarity and Euclidean distance. This enables the text-image alignment to fully reflect the matching status of the two modalities in the semantic space, avoids alignment deviations that may be missed by a single measurement method, accurately calculates the semantic consistency and sentiment alignment between the two modalities, and improves the accuracy of the alignment results.

[0056] It is understood that in this disclosure, the three-dimensional alignment of text can be determined based on text features and three-dimensional features in the same way as the method for determining text image alignment. The specific process for determining the three-dimensional alignment of text will not be described in detail in this disclosure.

[0057] Next, after determining the text-image alignment and text 3D alignment, the semantic loss value can be determined based on these two values. If both text-image alignment and text 3D alignment are high, the semantic loss value can be low; if either alignment is low, the semantic loss value can be increased to drive the model to correct generation biases in subsequent training. In other words, when there are deviations in the deep semantic, sentiment, and style features of text, images, and 3D geometry, the semantic loss value will dynamically increase, inversely forcing the model to optimize its generation strategy. This continuously constrains the high degree of consistency among the three modalities in semantic connotation, sentiment expression, style tone, and entity morphology features, fundamentally avoiding various misalignments and conflicts in the multimodal generation process.

[0058] In this embodiment, text, image, and 3D geometry data are encoded to obtain corresponding text features, image features, and 3D features. The text-image alignment is determined based on the text features and image features, and the text-3D alignment is determined based on the text features and 3D features. Then, the semantic loss value is determined based on the text-image alignment and text-3D alignment. This achieves global semantic unification of the three modalities by using text features as a common anchor point and constraining the alignment relationship between images and 3D geometry and text, ensuring semantic consistency between images and 3D geometry, and effectively improving the consistency and rationality of multimodal generated content.

[0059] In step 204, the obtained semantic loss value can be fused with the structural loss value to form a joint loss, resulting in a total loss value. This total loss value can serve as the joint optimization objective for the current training epoch. For example, when determining the total loss value, the sum or weighted sum of the semantic and structural loss values ​​can be used as the total loss value. The total loss value integrates surface sequence matching constraints and deep semantic consistency constraints. By fusing structural and semantic losses to construct a joint loss, the model is simultaneously subject to the dual constraints of sequence accuracy and semantic consistency. During the optimization process, it considers both the structural accuracy and semantic rationality of the generated results, as well as the consistency of multimodal content.

[0060] In step 205, if the total loss value is greater than a preset value, the model parameters of the second model are adjusted and iterative training is performed until the total loss value is less than or equal to the preset value, resulting in a trained 3D content generation model. The preset value can be pre-set based on actual needs or experience and can be dynamically updated based on validation set performance. When the total loss value is greater than the preset value, the model parameters are adjusted with the goal of minimizing the total loss value. For example, gradient descent, momentum update, adaptive optimization, or a combination thereof are used to update the encoding parameters, projection parameters, viewpoint embedding parameters, or decoding parameters in the model. Next, the model with adjusted parameters is trained again using new third samples, the total loss value is recalculated, and the model parameters are updated again when the total loss value is greater than the preset value. This process is repeated until the determined total loss value is less than or equal to the preset value, completing the training and resulting in a trained 3D content generation model.

[0061] In this embodiment of the disclosure, when training the second model using the third sample, the third-dimensional prediction data of the second model based on text data and image output with viewpoint information is obtained. The structural loss value is determined based on the three-dimensional prediction data and the three-dimensional geometric data of the third sample, and the semantic loss value is determined based on the text data, image with viewpoint information and three-dimensional geometric data of the third sample. The total loss value is then determined based on the structural loss value and the semantic loss value. The model is iteratively trained until the total loss value converges to below a preset value, thus obtaining a three-dimensional content generation model. Therefore, the model is directly supervised by the three-dimensional geometric truth value in each training session, and is also constrained by the deep semantic consistency between the text, image and three-dimensional geometry. This helps to solve the problems of semantic misalignment, sentiment tone shift and style inconsistency that are easy to occur when relying solely on sequence prediction, thereby further improving the reliability and usability of the trained three-dimensional content generation model.

[0062] In some embodiments, when determining the total loss value based on the structural loss value and the semantic loss value, the loss coefficient corresponding to the semantic loss term can be obtained first. This loss coefficient is used to control the weight of the semantic loss value in the total loss value. The product of the loss coefficient and the semantic loss value is then added to the structural loss value to obtain the total loss value. In other words, the total loss value can be expressed as a combination of the structural loss value and the weighted semantic loss value, i.e., total loss value = loss coefficient × semantic loss value + structural loss value. The loss coefficient can be a pre-set fixed value or dynamically adjusted according to the quality of the generated results during training.

[0063] In this embodiment of the disclosure, by obtaining the loss coefficient corresponding to the semantic loss term, and adding the product of the loss coefficient and the semantic loss value to the structural loss value to obtain the total loss value, the contribution of semantic loss is controlled by independent loss coefficients. Furthermore, by fusing the two losses in an additive manner, the training process can improve the deep consistency of text, image, and 3D geometry while maintaining the accuracy of geometric prediction.

[0064] In some embodiments, the loss coefficient can be a single scalar, or it can be a piecewise parameter, a dynamically mapped parameter, or a time-varying parameter that changes with the training epochs. For example, in the early stages of training, to ensure that the basic geometric prediction capability converges first, the loss coefficient can be taken as a small value; in the later stages of training, to more strongly constrain deep semantic consistency, the loss coefficient can be gradually increased.

[0065] In some embodiments, the loss coefficient can be dynamically determined based on the quality of the current generated results from the model. Therefore, in this embodiment, obtaining the loss coefficient corresponding to the semantic loss term includes: converting the three-dimensional prediction data into a two-dimensional prediction image; performing common-sense logic verification on the two-dimensional prediction image using a visual model to obtain the anomaly level output by the visual model; and determining the loss coefficient based on the anomaly level, where a higher anomaly level corresponds to a larger loss coefficient.

[0066] In this embodiment, when converting 3D prediction data into 2D prediction images, the 3D prediction data can be sliced ​​into 2D projections from multiple angles and directions to transform the 3D structure, material, lighting, and spatial relationships into recognizable 2D image data (i.e., 2D prediction images). Next, the obtained 2D image data is input into a visual model for common-sense logic verification. The visual model performs multi-dimensional compliance verification on the input 2D image data and outputs anomaly levels based on the number of identified flaws, defect areas, and severity. The visual model can be, for example, a pre-trained large-scale visual model. Common-sense logic verification refers to the process of identifying counterintuitive aspects of the generated content based on preset physical laws, human cognitive rules, or structural rationality rules. The anomaly level indicates the degree to which the generated result deviates from common-sense logic; a higher anomaly level indicates a greater deviation from common-sense logic and a less consistent interpretation of common-sense rules. For example, the anomaly levels, from low to high, are excellent, good, minor flaws, moderate anomalies, and serious violations.

[0067] In some embodiments, the dimensions of common-sense logic verification include at least one of physical common sense, material properties, structural logic, light and shadow patterns, and spatial relationships. After inputting two-dimensional image data into the visual model, the visual model accurately screens for anomalies from at least one of the five dimensions: physical common sense, material properties, structural logic, light and shadow patterns, and spatial relationships. Specifically, the physical common sense dimension can verify physical rules such as gravity support and force balance, identifying problems such as suspension, interweaving, and disproportionate ratios; the material properties dimension can verify the texture, reflectivity, and translucency of various materials, preventing material distortion; the structural logic dimension can detect the connection and proportional logic of object structures, identifying structural deformities, breaks, and interweaving; the light and shadow patterns dimension can verify the rationality of light and shadow projection and changes in brightness by combining light source parameters; and the spatial relationships dimension can verify spatial relationships such as object occlusion, hierarchy, and spacing, correcting spatial logic errors.

[0068] In this embodiment, the dimensions for common-sense logic verification include at least one of the following: physical common sense, material properties, structural logic, lighting and shadow patterns, and spatial relationships. Specifically, verification of the physical common sense dimension ensures that the generated 3D content follows basic physical laws, improving the physical plausibility of the generated result; verification of the material properties dimension ensures that the material representation of objects in the generated 3D content conforms to the material characteristics of the real world, improving the visual realism of the generated result; verification of the structural logic dimension ensures that the connection relationships between different parts of the generated 3D content are reasonable and proportionally coordinated, improving the structural integrity of the generated result; verification of the lighting and shadow patterns dimension ensures that the lighting effects in the generated 3D content match the light source position, object shape, and material properties, improving the visual consistency of the generated result; and verification of the spatial relationships dimension ensures that the spatial relationships such as occlusion, front-back hierarchy, and relative spacing between objects in the generated 3D content conform to the spatial logic of the real world, improving the spatial plausibility of the generated result. By performing common-sense logic verification on the 2D prediction image of the 3D prediction data in at least one of the above dimensions, the visual model can accurately identify anomalies in the 3D prediction data that violate common-sense laws, helping to ensure that the generated 3D content conforms to objective real-world laws.

[0069] For example, when using a visual model to perform counterintuitive filtering on a 2D predicted image corresponding to 3D predicted data, a preset filtering prompt text can be obtained first. This text includes at least one verification dimension from physical common sense, material properties, structural logic, light and shadow patterns, and spatial relationships, along with a description of the content to be verified for each dimension. It also includes rules for classifying different anomaly levels based on the number and severity of identified defects, or scoring rules and score ranges for each verification dimension, as well as the calculation method for determining the anomaly score based on the scores of each verification dimension, and the correspondence between the anomaly score and the anomaly level. Then, the 2D predicted image and the filtering prompt text are input into the visual model. The visual model verifies the 2D predicted image according to the verification dimensions and content recorded in the filtering prompt text, determines the anomaly level based on the verification results, or first determines the overall anomaly score based on the scores of different verification dimensions and then determines the anomaly level based on the anomaly score, outputting the anomaly level.

[0070] It should be noted that the method of outputting anomaly levels by the visual model is only an example and should not be regarded as a limitation of this disclosure. In some embodiments, the visual model may output the verification results corresponding to different verification dimensions, and the electronic device may determine the corresponding anomaly level based on the verification results.

[0071] In this embodiment, after determining the anomaly level, the corresponding loss coefficient can be determined based on the anomaly level. For example, the loss coefficients for the five anomaly levels—excellent, good, minor flaw, moderate anomaly, and serious violation—are 0.0, 0.2, 0.4, 0.7, and 1.0, respectively. Therefore, based on the determined anomaly level and the correspondence between the anomaly level and the loss coefficient, the current loss coefficient can be determined. A higher anomaly level indicates a more severe violation of common sense logic in the generated result. In this case, increasing the loss coefficient strengthens the semantic constraints, forcing the model to significantly correct its generation strategy. A lower anomaly level results in a smaller loss coefficient, avoiding excessive constraints that could affect the model's normal learning and achieving adaptive, refined optimization. It should be noted that the number of anomaly levels and the corresponding loss coefficients can be adjusted according to the application scenario and are not limited to five levels.

[0072] Existing 3D generation models lack external common-sense logic verification mechanisms, often resulting in contradictory material properties, anomalous structural forms, and violations of common-sense physical laws. This leads to significant deficiencies in the realism, plausibility, and practicality of 3D content, failing to meet the demands of industrial-grade creation and immersive scene construction. The proposed solution introduces this anti-common-sense filtering mechanism during the supervised training phase. This mechanism employs an automated closed-loop workflow combining 3D global 2D slicing sampling, multi-dimensional verification using a large visual model, and hierarchical loss coefficient optimization. Instead of directly outputting the result after 3D content generation, it first performs multi-angle, multi-directional 2D projection slicing of the 3D content, converting the three-dimensional structure, materials, lighting, and spatial relationships into recognizable 2D image data. This data is then fed into a large visual model for multi-dimensional compliance verification. Based on the anomaly level output by the large visual model, differentiated dynamic loss coefficients are matched for iterative optimization of the model. This mechanism overcomes the shortcomings of traditional 3D generation models, such as weak self-constraint, high detection miss rate, and crude error correction. It can take into account both minor visual flaws and core logical errors, and effectively ensure that the 3D content conforms to the objective laws of reality while ensuring generation efficiency.

[0073] In this embodiment of the disclosure, after converting the three-dimensional prediction data into a two-dimensional prediction image, a visual model is used to perform common sense logic verification. The current loss coefficient is dynamically determined based on the anomaly level obtained from the verification. This realizes the dynamic adjustment of the loss coefficient of the semantic loss term based on the three-dimensional content generated by the model. This allows the semantic constraints to be automatically enhanced when the generated three-dimensional prediction data has serious common sense violations, and the semantic constraints to be weakened when the generated result is reasonable. This realizes an adaptive training optimization strategy.

[0074] To implement the above embodiments, this disclosure also provides a training device for a three-dimensional content generation model, which can be implemented in software and / or hardware and can be integrated into an electronic device.

[0075] Figure 3 This is a schematic diagram of the structure of a training device for a three-dimensional content generation model provided in an embodiment of this disclosure, as shown below. Figure 3 As shown, the training device 30 for the 3D content generation model may include: a sample acquisition module 310, a pre-training module 320, a continuous training module 330, and a supervised training module 340.

[0076] The sample acquisition module 310 is used to acquire a first sample, a second sample and a third sample. The first sample is composed of data from different modalities combined in pairs. The second sample includes an image with viewpoint information. The third sample is a sequence of text data, an image with viewpoint information and three-dimensional geometric data. The different modalities include text, images and three-dimensional geometry. The pre-training module 320 is used to train the initial model using the first sample, so that the initial model learns the visual appearance rules and geometric structure priors to obtain the first model; The continuous training module 330 is used to train the first model using the first sample and the second sample, so that the first model learns the projection relationship between the three-dimensional geometry and the view image, and obtains the second model. The supervised training module 340 is used to train the second model using third samples, enabling the second model to learn the alignment relationships between text, images, and 3D geometry, thereby obtaining a 3D content generation model.

[0077] The training apparatus for a 3D content generation model applicable to electronic devices provided in this disclosure can execute the training method for the 3D content generation model provided in this disclosure, and has the corresponding functional modules and beneficial effects of the method execution. Content not described in detail in the apparatus embodiments of this disclosure can be referred to the description in any method embodiment of this disclosure.

[0078] This disclosure also provides an electronic device, including one or more processors and a memory, wherein the memory stores a program or instructions, and when the program or instructions are executed by the processor, they implement the steps of the training method for the three-dimensional content generation model as described in the foregoing embodiments. To avoid repetition, these steps will not be repeated here.

[0079] This disclosure also provides a computer-readable storage medium that is non-transitory. The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the training method for the three-dimensional content generation model as described in the foregoing embodiments. To avoid repetition, these steps will not be repeated here.

[0080] This disclosure also provides a computer program product, which includes computer program information. When the computer program information is run by a processor, the processor performs the steps of the training method for the three-dimensional content generation model as described in the foregoing embodiments.

[0081] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0082] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a three-dimensional content generation model, characterized in that, The method includes: Acquire a first sample, a second sample, and a third sample. The first sample is composed of pairs of data from different modalities. The second sample includes an image with viewpoint information. The third sample is a sequence of text data, an image with viewpoint information, and three-dimensional geometric data. The different modalities include text, images, and three-dimensional geometry. The initial model is trained using the first sample, so that the initial model learns the visual appearance rules and geometric structure priors, and a first model is obtained. The first model is trained using the first sample and the second sample, so that the first model learns the projection relationship between three-dimensional geometry and viewpoint image, and a second model is obtained. The second model is trained using the third sample, enabling it to learn the alignment relationships between text, images, and 3D geometry, thus obtaining a 3D content generation model.

2. The method according to claim 1, characterized in that, The step of training the second model using the third sample to learn the alignment relationships between text, images, and 3D geometry to obtain a 3D content generation model includes: The text data and the image with viewpoint information of the third sample are input into the second model to obtain the three-dimensional prediction data output by the second model based on the text data and the image with viewpoint information; Based on the three-dimensional prediction data and the three-dimensional geometric data of the third sample, the structural loss value is determined; Based on the text data of the third sample, the image with viewpoint information, and the three-dimensional geometric data, a semantic loss value is determined; Based on the structural loss value and the semantic loss value, the total loss value is determined; If the total loss value is greater than the preset value, the model parameters of the second model are adjusted and iterative training is performed until the total loss value is less than or equal to the preset value, thus obtaining the trained 3D content generation model.

3. The method according to claim 2, characterized in that, The determination of semantic loss values ​​based on the text data, the image with viewpoint information, and the three-dimensional geometric data of the third sample includes: The text data, the image with viewpoint information, and the three-dimensional geometric data of the third sample are encoded respectively to obtain corresponding text features, image features, and three-dimensional features; Based on the text features and the image features, determine the text-image alignment. Based on the text features and the three-dimensional features, the three-dimensional alignment of the text is determined; The semantic loss value is determined based on the text image alignment and the text 3D alignment.

4. The method according to claim 3, characterized in that, The text data, the image with viewpoint information, and the three-dimensional geometric data of the third sample are encoded respectively to obtain corresponding text features, image features, and three-dimensional features, including: The text features are obtained by extracting semantic features from the text data of the third sample using a visual language model; The visual language model is used to extract visual features from the image with viewpoint information to obtain the image features; The three-dimensional geometric data is converted into multiple two-dimensional images, and the visual language model is used to extract visual features from the multiple two-dimensional images to obtain the three-dimensional features.

5. The method according to claim 3, characterized in that, Determining the text-image alignment based on the text features and the image features includes: Determine the cosine similarity between the text features and the image features; Determine the Euclidean distance between the text features and the image features; The text image alignment is determined based on the cosine similarity and the Euclidean distance.

6. The method according to claim 2, characterized in that, The determination of the total loss value based on the structural loss value and the semantic loss value includes: Obtain the loss coefficients corresponding to the semantic loss terms; The total loss value is obtained by adding the product of the loss coefficient and the semantic loss value to the structural loss value.

7. The method according to claim 6, characterized in that, The process of obtaining the loss coefficients corresponding to the semantic loss term includes: The three-dimensional prediction data is converted into a two-dimensional prediction image; The two-dimensional predicted image is subjected to common sense logic verification using a visual model to obtain the anomaly level output by the visual model. Based on the anomaly level, the loss coefficient is determined, and the higher the anomaly level, the larger the corresponding loss coefficient.

8. The method according to claim 7, characterized in that, The dimensions of the common sense logic verification include at least one of the following: physical common sense, material properties, structural logic, light and shadow patterns, and spatial relationships.

9. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores a program or instructions that, when executed by the processor, implement the training method for the three-dimensional content generation model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the training method for the three-dimensional content generation model as described in any one of claims 1 to 8.

Images