Method and system for generating complex text to 3D model based on disturbance coding, medium and equipment

Abstract

The invention relates to the field of Text-3D generation, and discloses a method, a system, a medium and equipment for generating a 3D model from a complex text based on disturbance coding, and the method comprises the steps: initializing a grid nerve radiation field, and giving a text prompt; a view angle is sampled randomly, and after a nerve radiation field output code based on Grid is disturbed, an image z0 rendered and disturbed according to the angle is obtained; setting random time t, and adding random noise into the image z0 to obtain an image zt; inputting the image zt, the text y and the random time t into an image-text large model, calculating a loss gradient according to the SDS, and taking the loss gradient as the loss of the radiation field network; and taking the calculated loss gradient as an epoch, repeating to a set number of times, completing 3D neural radiation field training, outputting a grid neural radiation field model, and completing 3D modeling. According to the method, the robustness and the calculation efficiency of the algorithm are improved, and the consistency of the generated result and the complex text prompt is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of Text-3D generation technology, and in particular to a method, system, medium and device for generating a complex text into a 3D model based on perturbation coding. Background Art

[0002] Fractional distillation sampling (SDS) provides a fundamental approach to addressing the challenge of text-to-3D generation. However, methods derived from fractional distillation often suffer from several limitations, including unstable training, overly simple output structures, and insufficient adherence to textual cues. Traditional distillation sampling methods suffer from polyhedral problems and low consistency of complex texts:

[0003] The polyhedron problem refers to the fact that 3D models generated from pure text often have multiple ambiguous faces. This problem is caused by the limitations of prior knowledge in large multimodal models, significantly impacting the accuracy of model generation. The problem of low consistency in complex text refers to the fact that when processing complex text descriptions, the generated 3D assets often fail to accurately reflect the text content, resulting in inconsistent results. These two issues not only increase training and generation costs but also affect the quality and practicality of the generated 3D assets. Therefore, addressing these issues is crucial to improving the efficiency and effectiveness of 3D asset generation. Summary of the Invention

[0004] In response to the above problems, the purpose of the present invention is to provide a method, system, medium and device for generating complex text into 3D models based on perturbation coding, which introduces random perturbation coding, improves the robustness and computational efficiency of the algorithm, and significantly improves the consistency between the generated results and the complex text prompts.

[0005] To achieve the above objectives, in the first aspect, the technical solution adopted by the present invention is: a method for generating a complex text to a 3D model based on perturbation coding, which includes: initializing a grid neural radiation field, giving a text prompt prompt y; randomly sampling the view angle, and perturbing the output encoding of the grid-based neural radiation field to obtain an image z0 perturbed by the angle; setting a random time t, adding random noise to the image z0, and obtaining an image z t ; Image z t , text y and random time t are input into the large image and text model, and the loss gradient is calculated according to SDS as the loss of the radiation field network; the calculation of the loss gradient once is regarded as an epoch, which is repeated to the set number of times to complete the 3D neural radiation field training, output the grid neural radiation field model, and complete the 3D modeling.

[0006] Furthermore, the output encoding of the neural radiation field based on the grid is perturbed to obtain the image z0 perturbed by the angle rendering, including:

[0007] Set up a perturbation encoder. The perturbation encoder is a position encoding module. The perturbation encoding is used to enhance the position encoding parameters to prevent the model from obtaining incorrect solutions or falling into the local minimum neighborhood during training, and output the results of random noise perturbation.

[0008] Furthermore, the result of outputting random noise disturbance is:

[0009] (τ+Δτ,ρ+Δρ)=MLP(μ+Δμ;θ)

[0010] Where θ is the parameter of the MLP; Δμ is the perturbation code, which introduces a small increment to the original code μ. After passing through the MLP network, this increment will introduce a small perturbation to the values of the material density τ and transmittance ρ, which are Δτ and Δρ, respectively.

[0011] Furthermore, we set a random time t and add random noise ∈ sampled from the normal distribution to the image z0 generated by NeRF to obtain the image z t , the calculation process is:

[0012]

[0013] Where z t is the result of adding noise to the image z0 generated by NeRE according to time t; a t is a function of t β t is a linear interpolation from 0.0001 to 0.02, and the total step size of the interpolation is equal to the maximum value of the random time t max =1000.

[0014] Furthermore, the loss gradient is calculated according to the fractional distillation sampling algorithm SDS:

[0015]

[0016] Where, To find the partial derivative of the formula with the generator parameter θ as the variable; The loss Loss calculated by the fractional distillation sampling algorithm; represents the expected function with the time step t and the randomly sampled noise ∈ as variables; ω is a hyperparameter that affects the control of the text on the generated results; ∈ φ is the text-image module of the pre-trained multimodal model with φ as the parameter.

[0017] Furthermore, after the Grid-based neural radiation field output encoding is disturbed, it also includes a step for overcoming the error introduced by the disturbance.

[0018] Furthermore, the steps for overcoming the error introduced by the disturbance include adopting a two-stage series modeling method, specifically:

[0019] The first stage is the initial coarse model generation stage, which uses the perturbation coded generation space to create an initial coarse model;

[0020] The second stage is the texture mesh fine-tuning stage, in which the DMTet algorithm is applied to fine-tune the texture mesh.

[0021] In the second aspect, the technical solution adopted by the present invention is: a complex text to 3D model generation system based on perturbation coding, which includes: an initialization module, which initializes the grid neural radiation field, given a text prompt prompt, represented as y; a perturbation module, which randomly samples the view angle, and obtains the image z0 after perturbation based on the output encoding of the grid neural radiation field; a noise module, which sets a random time t, adds random noise to the image z0, and obtains the image z t ; Loss calculation module, the image z t , text y and random time t are input into the large image and text model, and the loss gradient is calculated according to SDS as the loss of the radiation field network; the 3D model generation module calculates the loss gradient once as an epoch, repeats it to the set number of times, completes the 3D neural radiation field training, outputs the grid neural radiation field model, and completes the 3D modeling.

[0022] In a third aspect, the technical solution adopted by the present invention is: a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the above methods.

[0023] In a fourth aspect, the technical solution adopted by the present invention is: a computing device, comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0024] The present invention has the following advantages due to the adoption of the above technical solution:

[0025] The present invention introduces random perturbation coding, successfully improving the robustness and computational efficiency of the algorithm and significantly improving the consistency of the generated results with complex text prompts. The effectiveness of the method of the present invention is demonstrated through extensive experimental results and comparisons. For example, when processing complex text prompts, the present invention can more accurately reflect the various elements in the prompt word, and the generated effect is closer to the description of the prompt word. This not only verifies the superiority of the present invention, but also demonstrates its potential and advantages in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1This is an overall flow chart of a method for generating a complex text into a 3D model based on perturbation coding in an embodiment of the present invention;

[0027] Figure 2 This is a detailed flow chart of a method for generating a complex text into a 3D model based on perturbation coding in an embodiment of the present invention;

[0028] Figure 3 This is a multi-view example diagram of a complex text-3D model generated by perturbation coding in an embodiment of the present invention;

[0029] Figure 4 This is a comparison chart of the perturbation coding algorithm in an embodiment of the present invention and other Text-3D algorithms based on SDS;

[0030] Figure 5 This is a comparison graph generated for “A lark is singing on a branch adorned with flowers.” in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The loss constraint method for federated learning disclosed in the existing literature adopts a perturbation coding technology similar to that of the present invention; however, in this document, when the server receives the model parameter data generated by each participant, it performs random perturbation coding on the model parameter data to obtain perturbation model parameter data, summarizes and decodes it, and performs verification and regression statistics on the decoded perturbation model parameter data according to the loss constraint mechanism to obtain target model parameter data. The target model parameter data is passed back to each participant so that each participant can iteratively update the model according to the target model parameter data, and the steps are repeated until the model converges or the maximum number of iterations is reached. In this way, the efficiency of joint modeling can be improved while ensuring data security. In contrast, the perturbation coding proposed in the present invention is applied in the field of Text-3D generation, and perturbation coding is used to improve the generation consistency of complex texts and scenes.

[0032] The method for reconstructing three-dimensional scenes based on variational fractional distillation and EEG encoder disclosed in existing literature is similar to the present invention; the literature includes: S1, obtaining EEG signals and creating a data set, and preprocessing the data set; S2, constructing a 3D-Telepathy neural network model for reconstructing corresponding three-dimensional scenes through EEG signals, the 3D-Telepathy neural network model includes a double-mask EEG signal autoencoder and a 3D generation module, the 3D generation module includes a stable diffusion model LDM, a low-rank adaptation matrix LoRA and a neural radiation field NeRF; S3, using the training set to train the 3D-Telepathy neural network, combining the characteristics of EEG signals, and tuning the parameters of the 3D-Telepathy neural network model to optimize the model generation ability; S4, using the test set as input, using the trained double-mask EEG signal autoencoder to encode the EEG signal into a latent space embedding vector, and inputting the embedding vector as a control condition into the 3D generation module to guide NeRF to generate a 3D scene, ultimately achieving the purpose of reconstructing the three-dimensional vision of the human brain. In contrast, the present invention only requires text and can generate 3D assets and models directly from a pre-trained multimodal large model. Through perturbation coding, high-precision modeling effects can be achieved, which improves the production efficiency of 3D models and enhances the user experience.

[0033] After in-depth analysis, the present inventors determined that the fundamental reason for the lack of consistency between the generated results and complex text prompts lies in the fact that the distillation sampling algorithm is a multi-view, multi-step optimization algorithm. 1) In the specific implementation of multi-view sampling, it is first necessary to design an efficient view distribution strategy in 3D space. This is typically achieved through a spherical uniform sampling algorithm, ensuring that the target object is observed from different elevation and azimuth angles. The key to this strategy is to cover the geometric features of the object as comprehensively as possible within the limited number of samples, avoiding information loss due to view limitations. Through carefully designed camera pose randomization and angle control, the generative model's ability to understand spatial structure can be significantly enhanced. 2) Multi-step optimization is a key technical path to improving generation quality. The optimization process typically begins with constructing an initial neural implicit representation, setting basic geometric constraints, and initializing the parameter space. In each iteration, the model renders the current 3D representation from multiple pre-sampled views, generating a series of 2D projection images. These images are then used to calculate the fractional distillation loss, which aggregates gradient information from different views to comprehensively evaluate the consistency and quality of the generated content. Compared with single-view optimization, the multi-view method can effectively suppress local overfitting and provide more comprehensive and stable gradient signals.

[0034] Therefore, the present invention introduces a perturbation coding method to enhance the robustness of voxel-based expression models. Existing methods typically use neural radiation fields or Gaussian sputtering as 3D expression models, which require training data with extremely high 3D consistency; otherwise, artifacts, over-smoothing and other problems may occur, which will damage the rendering results. However, the pre-trained stable diffusion model (SDM) relies on adding noise to the samples to provide a score loss for the distillation sampling, which makes it challenging to generate results with strict 3D consistency. Therefore, the 3D output of the pre-trained SDM distillation sampling is often conservative and too smooth, making it difficult to maintain high fidelity for complex text cues. The present invention proposes perturbation coding, which incorporates tiny random perturbations into the training process of the 3D representation model, and applies random perturbations to randomly perturb the position code to create more complex 3D generation and stronger text consistency. The present invention will demonstrate that these carefully calibrated perturbations help generate complex details that are more accurately consistent with complex input clues, thereby improving the model's fidelity for complex cues. As Figure 3 As shown in the figure, it demonstrates the excellent generation effects of various 3D models under multiple perspectives.

[0035] The proposed method demonstrates strong performance in generating highly consistent results for complex prompts without fine-tuning pre-trained models. Furthermore, the minimal architectural modifications required enable easy integration with various SDS derivatives.

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] In one embodiment of the present invention, a method for generating a complex text to a 3D model based on perturbation coding is provided, wherein the perturbation coding technique is used to enhance the position coding parameters to prevent the model from obtaining an incorrect solution or falling into a local minimum during the training process. In this embodiment, Figure 1 、 Figure 2 As shown, the method includes the following steps:

[0039] 1) Initialize the grid neural radiance field, given a text prompt, denoted as y.

[0040] 2) Randomly sample the view angle, and the grid-based neural radiance field (NeRF) output encoding is perturbed to obtain the image z0 perturbed by the angle rendering.

[0041] 3) Set the random time t, add random noise v sampled from the normal distribution to the image z0, and get the image z t ;

[0042] The specific process is as follows:

[0043]

[0044] Where z t is the result of adding noise to the image z0 generated by NeRE according to time t; a t is a function of t β t is a linear interpolation from 0.0001 to 0.02, and the total step size of the interpolation is equal to the maximum value of the random time t max =1000.

[0045] 4) Image z t , text y and random time t are input into the unet module of the large graph and text model (Stablediffsion), and the loss gradient is calculated according to SDS as the loss of the radiation field network.

[0046] 5) Repeat steps 2) to 4) to calculate the loss gradient once as an epoch, repeat to a set number of times (for example, repeat at least 10,000 times), complete the 3D neural radiation field training, output the mesh neural radiation field model, and complete the 3D modeling.

[0047] In the above step 2), the Grid-based neural radiation field output encoding is perturbed to obtain an image z0 perturbed according to the angle rendering, including: setting a perturbation encoder, the perturbation encoder is a position encoding module, and the position encoding parameters are enhanced by perturbation encoding to avoid the model obtaining an incorrect solution or falling into the local minimum neighborhood during the training process, and outputting the result of random noise perturbation.

[0048] like Figure 2 As shown, perturbation coding is perturbation coding. Figure 2 In the figure, the output encoding of the Grid-based neural radiation field is perturbed to obtain the rendered image z0. The rendered image z0 is calculated through the denoising and denoising process of the large image and text model to provide training gradients for the Grid-based neural radiation field.

[0049] like Figure 2 As shown in Figure 1, the perturbation encoder is a position encoding module whose output is the result of random noise perturbation. Given that the density τ and material reflectivity ρ are calculated based on the three-dimensional coordinates μ: (τ, ρ) = MLP(μ; θ), the result of combining the perturbation encoder can be expressed as:

[0050] (τ+Δτ,ρ+Δρ)=MLP(μ+Δμ;θ)

[0051] Where θ is the parameter of the MLP; Δμ is the perturbation code, which introduces a small increment to the original code μ. After passing through the MLP network, this increment will introduce a small perturbation to the values of the material density τ and transmittance ρ, which are Δτ and Δρ, respectively.

[0052] Then the color c of each point is rendered as a function of the increment as c Δ :

[0053]

[0054] Among them, l,l ρ ,l a are the three-dimensional coordinates of the point light source, the color of the point light source, and the color of the ambient light; n Δ is the surface normal vector, which is calculated as the unit vector of the partial derivative of the material density increment with respect to the perturbed encoded value:

[0055] Let the color and normal vector of a point light source that is not randomly perturbed be denoted by c and n respectively. We can analyze two possible situations:

[0056] When Δτ ≈ 0: the density change is minimal. In this case, the area around the point exhibits similar density and reflectivity, resulting in c Δ =c.

[0057] When |Δτ|>>0: the density changes significantly. In this case, point n Δ The normal of n is likely to be different from n. Regardless of whether the reflectivity change Δρ is significant, according to the formula above c Δ The calculation formula for each n Δ >0 for the point c Δ Both are expected to be different from c.

[0058] Therefore, after introducing random perturbations, the positional encodings of the perturbation encoder will randomly fluctuate around their current optimal value during training. This random fluctuation causes color changes in areas where density changes significantly. These areas typically correspond to points on the modeled surface. Therefore, this fluctuation effectively introduces small, random bumps to the otherwise smooth modeled surface. These subtle changes may promote more complex structure at the modeled edges, making the generated model more likely to match certain descriptions in the complex cues. Essentially, the random encoding perturbations introduce a degree of randomness to the surface structure of the 3D representation, effectively alleviating the inherent limitations of stochastic gradient descent optimization. By introducing this randomness, the network is better able to escape local optima in the parameter space, thus avoiding optimization traps. This mechanism significantly enhances the network's ability to learn and effectively improve from complex text cues, enabling it to more accurately capture and reflect the details and complexity of the cues. Therefore, this approach not only improves the model's expressiveness and ability to capture detail, but also enhances its adaptability and robustness to complex cues. In this way, the network can more effectively learn and generate 3D models that are consistent with complex text cues.

[0059] In step 3) above, the loss gradient is calculated according to the fractional distillation sampling algorithm SDS:

[0060]

[0061] Where, To find the partial derivative of the formula with the generator parameter θ as the variable; The loss Loss calculated by the fractional distillation sampling algorithm; represents the expected function with the time step t and the randomly sampled noise ∈ as variables; ω is a hyperparameter that affects the control of the text on the generated results, and is preferably 7.5 in this invention; ∈ φ In the above embodiments, the text-image module of the pre-trained multimodal model with φ as the parameter (for example, the UNet module in the StableDiffusion algorithm) further includes a step for overcoming the error introduced by the disturbance after executing step 2) the Grid-based neural radiation field output encoding is disturbed.

[0062] Under normal circumstances, accurate and smooth surface modeling can be achieved by relying solely on fixed coding. The advantage of this method lies in its stability and precise capture of details. However, if only perturbation coding is used, unnecessary interference may be introduced, resulting in surface irregularities and possibly hindering the convergence of the model. The two methods are almost opposite in results: fixed coding tends to produce smooth but possibly lacking details, while perturbation coding may cause the surface to be too rough and difficult to control. In order to overcome the respective limitations of these methods, the present invention proposes a modeling scheme using a two-stage series connection to overcome the errors introduced by the perturbation.

[0063] Specifically, the steps for overcoming the error introduced by the disturbance include adopting a two-stage series modeling method:

[0064] The first stage is the initial coarse model generation stage, which uses the generation space of perturbation coding to create an initial coarse model; the goal of this stage is to take advantage of the perturbation coding, that is, to introduce randomness to enhance the model's generalization ability and detail capture ability.

[0065] The second stage is the texture mesh fine-tuning stage, which uses the DMTet (Differentiable Marching Tetrahedron) algorithm to fine-tune the texture mesh. The goal of this stage is to further improve the model's resolution and detail accuracy while maintaining the stability of the overall structure.

[0066] Through this continuous two-stage process, the present invention achieves high resolution and precise detail representation in the final model. This approach not only combines the advantages of fixed and perturbation encoding, but also ensures the quality and accuracy of the model through a meticulous adjustment process, making it better suited to the needs of complex modeling tasks.

[0067] In order to comprehensively evaluate the consistency of complex texts, this paper selects two advanced indicators: Clip Score and VQAScore, as shown in Table 1.

[0068] Table 1 Comparison of Clip Score and VQAScore

[0069]

[0070] Clip Score measures consistency by calculating the semantic similarity between the image and the text description. Complex text requires that the text description contains more than two objects. Specifically, the CLIP model is used to calculate similarity, which is mathematically expressed as:

[0071] Clip Score=cos(CLIP(I),CLIP(y))

[0072] Here, I represents the image and y represents the text description. In the experiment, the clip-vit-base-patch16 model was used, and the 3D generation result was subjected to perspective sampling as the input image. VQAScore focuses on evaluating the performance of visual question answering models by calculating the accuracy of the model's predicted answers:

[0073]

[0074] In this formula, N represents the total number of samples, and the division by 3 is used to normalize the score. In this example, the ViLBERT model is selected, and the question is set as "Does the image match the text description?" By randomly sampling 100 views, the VQAScore is finally obtained to comprehensively evaluate the text prompt consistency of the generated results. The results are shown in Table 1, which achieves the current state-of-the-art (SOTA) results in terms of complex text consistency. Figure 4 In order to further verify the method of the present invention, Figure 5 A comparative case under complex text conditions is shown in . In this case, the prompt word "A lark issinging on a branch adorned with flowers." is used. The prompt word contains three main elements: "lark", "branch" and "flowers". By comparing the generated images, it can be clearly seen that the method of the present invention can more accurately reflect the various elements in the prompt word when processing complex text, and the generated effect is closer to the description of the prompt word. This ability not only demonstrates the superiority of the method of the present invention under complex text conditions, but also shows that the method of the present invention has higher accuracy and consistency when processing multi-element prompt words. These results further demonstrate the potential and advantages of the method of the present invention in practical applications.

[0075] In one embodiment of the present invention, a system for generating a complex text into a 3D model based on perturbation coding is provided, which includes:

[0076] Initialization module, initializes the grid neural radiation field, given a text prompt, represented as y;

[0077] The perturbation module randomly samples the view angle, and the grid-based neural radiation field output encoding is perturbed to obtain the image z0 perturbed by the angle rendering;

[0078] Noise module, set random time t, add random noise to image z0, and get image z t ;

[0079] Loss calculation module, the image z t, text y and random time t are input into the large graph and text model, and the loss gradient is calculated according to SDS as the loss of the radiation field network;

[0080] The 3D model generation module calculates the loss gradient once as an epoch, repeats it a set number of times, completes the 3D neural radiation field training, outputs the mesh neural radiation field model, and completes the 3D modeling.

[0081] In the above embodiment, after the Grid-based neural radiation field output encoding is perturbed, the image z0 perturbed according to the angle rendering is obtained, including:

[0082] Set up a perturbation encoder. The perturbation encoder is a position encoding module. The perturbation encoding is used to enhance the position encoding parameters to prevent the model from obtaining incorrect solutions or falling into the local minimum neighborhood during training, and output the results of random noise perturbation.

[0083] In this embodiment, the result of outputting random noise disturbance is:

[0084] (τ+Δτ,ρ+Δρ)=MLP(μ+Δμ;θ)

[0085] Where Δμ is the small coordinate increment introduced by the perturbation code; θ is the parameter of the MLP, and Δμ is the perturbation code, which introduces a small increment on the original code μ. After passing through the MLP network, this increment will introduce a small perturbation to the values of the material density τ and transmittance ρ, which are Δτ and Δρ, respectively.

[0086] In the above embodiment, a random time step t is set, random noise is added to image z0, and image z is obtained. t for:

[0087]

[0088] Where z t is the result of adding noise to the image z0 generated by NeRE according to time t; a t is a function of t β t is a linear interpolation from 0.0001 to 0.02, and the total step size of the interpolation is equal to the maximum value of the random time t max =1000.

[0089] In the above embodiment, the loss gradient is calculated according to SDS:

[0090]

[0091] Where, To find the partial derivative of the formula with the generator parameter θ as the variable; The loss Loss calculated by the fractional distillation sampling algorithm; represents the expected function with the time step t and the randomly sampled noise ∈ as variables; ω is a hyperparameter that affects the control of the text on the generated results. In this invention, it is recommended to take 7.5; ∈ φ is the text-image module of a pre-trained multimodal model with φ as a parameter (for example, the unet module in the StableDiffusion algorithm).

[0092] In the above embodiment, after the Grid-based neural radiation field output encoding is disturbed, a step for overcoming the error introduced by the disturbance is also included.

[0093] In this embodiment, the steps for overcoming the error introduced by the disturbance include adopting a two-stage series modeling method, specifically:

[0094] The first stage is the initial coarse model generation stage, which uses the perturbation coded generation space to create an initial coarse model;

[0095] The second stage is the texture mesh fine-tuning stage, in which the DMTet algorithm is applied to fine-tune the texture mesh.

[0096] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0097] A computing device provided in one embodiment of the present invention may be a terminal and may include: a processor, a communications interface, a memory, a display screen, and an input device. The processor, communications interface, and memory communicate with each other via a communications bus. The processor is configured to provide computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. When executed by the processor, the computer program implements the methods described in the aforementioned embodiments. The internal memory provides an environment for the operating system and computer program in the non-volatile storage medium to run. The communications interface is configured to communicate with an external terminal via wired or wireless communication, where wireless communication may be achieved via Wi-Fi, a network management service provider, NFC (near-field communication), or other technologies. The display screen may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen layer covering the display screen, or may be buttons, a trackball, or a touchpad provided on the computing device housing, or may be an external keyboard, touchpad, or mouse. The processor may invoke logic instructions stored in the memory.

[0098] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0099] In one embodiment of the present invention, a computer program product is provided, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0100] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores server instructions. The computer instructions enable a computer to execute the methods provided in the above embodiments.

[0101] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effects are similar to those of the above method embodiment, and will not be repeated here.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating complex text into 3D models based on perturbation coding, characterized in that: include: Initialize the grid neural radiance field, given a text prompt, denoted as y; Randomly sample the view angle, and after perturbation, the grid-based neural radiation field output encoding is perturbed to obtain the image z0 perturbed by the angle rendering; Set random time t, add random noise to image z0, and get image z t ; The image z t , text y and random time t are input into the large graph and text model, and the loss gradient is calculated according to SDS as the loss of the radiation field network; The calculation of the loss gradient once is regarded as an epoch, which is repeated a set number of times to complete the 3D neural radiation field training, output the mesh neural radiation field model, and complete the 3D modeling.

2. The method for generating a complex text to a 3D model based on perturbation coding according to claim 1, wherein: After the grid-based neural radiation field output encoding is perturbed, the image z0 perturbed by the angle rendering is obtained, including: Set up a perturbation encoder. The perturbation encoder is a position encoding module. The perturbation encoding is used to enhance the position encoding parameters to prevent the model from obtaining incorrect solutions or falling into the local minimum neighborhood during training, and output the results of random noise perturbation.

3. The method for generating a complex text to a 3D model based on perturbation coding as claimed in claim 2, characterized in that: The result of outputting random noise disturbance is: (τ+Δτ,ρ+Δρ)=MLP(μ+Δμ;θ) Where θ is the parameter of the MLP; Δμ is the perturbation code, which introduces a small increment to the original code μ. After passing through the MLP network, this increment will introduce a small perturbation to the values of the material density τ and transmittance ρ, which are Δτ and Δρ, respectively.

4. The method for generating a complex text to a 3D model based on perturbation coding according to claim 1, wherein: Set a random time t, add random noise ∈ sampled from the normal distribution to the image z0 generated by NeRF, and get the image z t , the calculation process is: Where z t is the result of adding noise to the image z0 generated by NeRE according to time t; a t is a function of t β t is a linear interpolation from 0.0001 to 0.02, and the total step size of the interpolation is equal to the maximum value of the random time t max =1000.

5. The method for generating a complex text to a 3D model based on perturbation coding as claimed in claim 1, wherein: The loss gradient is calculated according to the fractional distillation sampling algorithm SDS: Where, To find the partial derivative of the formula with the generator parameter θ as the variable; The loss Loss calculated by the fractional distillation sampling algorithm; represents the expected function with the time step t and the randomly sampled noise ∈ as variables; ω is a hyperparameter that affects the control of the text on the generated results; ∈ φ is the text-image module of the pre-trained multimodal model with φ as the parameter.

6. The method for generating a complex text to a 3D model based on perturbation coding as claimed in claim 1, wherein: After the Grid-based neural radiation field output encoding is perturbed, it also includes steps for overcoming the errors introduced by the perturbation.

7. The method for generating a complex text to a 3D model based on perturbation coding according to claim 6, characterized in that: The steps for overcoming the error introduced by the disturbance include adopting a two-stage series modeling method, specifically: The first stage is the initial coarse model generation stage, which uses the perturbation coded generation space to create an initial coarse model; The second stage is the texture mesh fine-tuning stage, in which the DMTet algorithm is applied to fine-tune the texture mesh.

8. A complex text to 3D model generation system based on perturbation coding, characterized in that: include: Initialization module, initializes the grid neural radiation field, given a text prompt, represented as y; The perturbation module randomly samples the view angle, and the grid-based neural radiation field output encoding is perturbed to obtain the image z0 perturbed by the angle rendering; Noise module, set random time t, add random noise to image z0, and get image z t ; Loss calculation module, the image z t , text y and random time t are input into the large graph and text model, and the loss gradient is calculated according to SDS as the loss of the radiation field network; The 3D model generation module calculates the loss gradient once as an epoch, repeats it a set number of times, completes the 3D neural radiation field training, outputs the mesh neural radiation field model, and completes the 3D modeling.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 7 .

10. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods according to claims 1 to 7.

Images