Methods for building real-time rendering engines for generative AI models and related equipment

Abstract

This invention discloses a method for constructing a real-time rendering engine for generative AI models and related equipment, relating to the field of human-computer interaction technology. The method includes acquiring multimodal semantic information from user input and converting it into latent vectors and constraint parameters; constructing a lightweight generative model and directly generating renderable intermediate representations using this model; building a rendering pipeline based on the generated intermediate representations to convert them into visualized images; capturing user operations in real-time and dynamically updating the constraints of the generative model to trigger incremental generation; optimizing system performance using model lightweighting, hardware acceleration, and hybrid rendering strategies; and ensuring temporal and spatial consistency of generated content through temporal filtering and spatial constraint techniques. This invention solves the problems of slow generation speed and low rendering efficiency in the fusion of generative AI models and real-time rendering engines, improving the efficiency and quality of visualized image generation and rendering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, more specifically, the present application relates to a generative AI model real-time rendering engine construction method and related equipment thereof. BACKGROUND

[0002] A generative AI model refers to a model that automatically generates content through artificial intelligence algorithms, rather than simply editing existing content. In a rendering scene, it can automatically generate new visual content (such as 3D models, textures, lighting effects, etc.) based on text descriptions, sketches, a small number of images or parameters, for example, inputting "a cyberpunk-style future city", the model can directly generate 3D scene details that meet the description. A rendering engine is a core system that converts digital models (3D geometry, materials, lighting, etc.) into visual images. "Real-time" means that the rendering speed needs to match human eye perception (usually ≥ 30fps, i.e. generating more than 30 frames of images per second), and there is no obvious delay when supporting interactive operations (such as view rotation, scene editing). Traditional real-time rendering relies on pre-defined geometry and materials, but with the combination of generative AI, the engine can dynamically generate content, breaking the limitations of fixed materials.

[0003] Problems existing in the prior art:

[0004] In traditional technology, after the generative AI model generates 3D content, it needs to be exported to an independent rendering engine for processing, which has a serial process of "first generation and then rendering", resulting in a long overall time consumption; the model has randomness in generating 3D content (such as color deviation, shape distortion), and it is difficult to respond to real-time parameter adjustment; AI-generated content is prone to inter-frame flickering (poor temporal consistency) or object surface mutation (poor spatial consistency), affecting the visual experience.

[0005] In view of the above problems, the present application provides a solution. SUMMARY

[0006] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a generative AI model real-time rendering engine construction method and related equipment thereof, by constructing a system, the AI model automatically generates 3D content, and converts these contents into visual images, to solve the problems raised in the above background art.

[0007] To achieve the above purpose, the present application provides the following technical scheme:

[0008] The generative AI model real-time rendering engine construction method comprises the following steps:

[0009] Obtain the multi-modal semantic information input by the user, analyze the multi-modal semantic information through the CLIP model, and convert it into a latent vector and constraint parameters that can be understood by the generative AI model.

[0010] A lightweight generative model is constructed based on the latent vector and the constraint parameter, and a renderable intermediate representation is directly generated using the lightweight generative model, wherein the intermediate representation includes a 3D Gaussian point cloud, an implicit symbolic distance function, and a dynamic texture atlas;

[0011] A rendering pipeline is constructed based on the generated intermediate representation, and the intermediate representation is converted into a visual image by combining rasterization acceleration and ray marching and spatial hashing acceleration techniques;

[0012] Real-time user operations are captured and the constraint conditions of the generative model are dynamically updated to trigger incremental generation, while the system performance is optimized by using model lightweight, hardware acceleration, and hybrid rendering strategies;

[0013] Through time filtering and spatial constraint techniques, the generated content is consistent in time and space.

[0014] In a preferred embodiment, the specific steps of analyzing multi-modal semantic information through a CLIP model and converting it into a latent vector and constraint parameter that can be understood by a generative AI model are as follows:

[0015] The preprocessed text sequence is input into the text encoder of the CLIP model to obtain a text feature vector, wherein the text encoder of the CLIP model adopts a Transformer structure to extract semantic features of the text by deep encoding of the text sequence;

[0016] The preprocessed image is input into the image encoder of the CLIP model to obtain an image feature vector, wherein the image encoder of the CLIP model usually adopts a convolutional neural network to extract visual features of the image;

[0017] The text feature vector and the image feature vector are fused to obtain a fused multi-modal feature vector, and the formula is as follows:

[0018]

[0019] In the formula, is the multi-modal feature vector, is the aligned text feature vector, is the aligned image feature vector, is the text feature importance weight, and b is the image feature importance weight;

[0020] The fused multi-modal feature vector is mapped to the latent space of the generative AI model to obtain a latent vector;

[0021] From the multi-modal features extracted from the CLIP model and the user input parameters, the constraint parameters required by the generative AI model are extracted.

[0022] In a preferred embodiment, the fused multi-modal feature vector is mapped to the latent space of the generative AI model to obtain the latent vector, the process being as follows:

[0023] The latent space dimension of the generative AI model is obtained, and the fused feature vector dimension is adjusted according to the latent space dimension;

[0024] The adjusted fused feature vector is projected to the latent space of the generative model through a nonlinear mapping network to obtain the latent vector, ensuring that the vector distribution is consistent with the latent space during model training.

[0025] In a preferred embodiment, the process of projecting the adjusted fused feature to the latent space of the generative model through a nonlinear mapping network to obtain the latent vector is as follows:

[0026] Batch normalization is performed on the fused multi-modal feature vector to stabilize the distribution range;

[0027] The batch-normalized multi-modal feature vector is input into a multi-layer perceptron (MLP) for nonlinear transformation; the output of the multi-layer perceptron (MLP) is regularized to ensure compliance with the latent space prior distribution, and thus the latent vector is obtained.

[0028] In a preferred embodiment, the process of directly generating a renderable intermediate representation using the lightweight generative model is as follows:

[0029] With the latent vector as input, the generator outputs an initial point set and Gaussian parameters, and imposes constraints on the Gaussian parameters. Similar Gaussian points are merged through clustering to generate a 3D Gaussian point cloud;

[0030] A micro-MLP is used to generate an implicit signed distance function, with 3D coordinates, latent vectors, and constraint parameters as input, and the signed distance from the coordinates to the object surface as output. Shape constraints are converted into a spatial mask, and coordinates exceeding the boundary are forced to output a positive distance;

[0031] Based on the latent vector, a lightweight GAN generator is used to generate a basic texture feature map, and material parameters are converted into texture modulation factors. The texture resolution is dynamically adjusted according to the rendering requirements to obtain a dynamic texture atlas.

[0032] In a preferred embodiment, the process of real-time capturing user operations and dynamically updating the constraint conditions of the generative model is as follows:

[0033] User operations are captured in real time through a system API, with a sampling frequency of 60 Hz to ensure no delay;

[0034] The user operation is converted into an incremental change of the constraint parameter, and is synchronized to the generative model through an event callback mechanism to realize dynamic updating of the constraint condition of the generative model.

[0035] In a preferred embodiment, the incremental generation process is triggered as follows:

[0036] Only the region affected by the operation is triggered for generation, and if the operation involves the whole, an incremental update of the whole intermediate representation is generated;

[0037] If it is a local operation, the generation range is limited by a spatial mask.

[0038] In a preferred embodiment, the real-time rendering engine performance optimization process of the generative AI model is as follows:

[0039] According to the hardware performance, the model precision is automatically switched, FP16 precision is used on high-end GPU, and INT8 precision is automatically switched on mobile GPU;

[0040] A special acceleration unit of GPU is enabled for Gaussian point cloud matrix operation, a special rendering channel is enabled for mobile GPU, and rasterization and ray stepping tasks are distributed to different calculation units through a synchronization barrier for parallel processing;

[0041] The static region uses pre-rendering cache, the result is stored as a texture after the first rendering, and the subsequent frames are directly sampled, and the dynamic region uses real-time generation + low resolution rendering, and the details are restored through a super-resolution module to balance speed and quality.

[0042] In a preferred embodiment, the generated visual image is consistent in time and space through the time filtering and spatial constraint technology as follows:

[0043] The CLIP features of the current frame and the previous frame are extracted and the cosine similarity is calculated, if the cosine similarity is less than a threshold, the latent vector of the generative model is adjusted back to keep the semantics of the generated content consistent with the historical frames;

[0044] The picture quality score is calculated through a no-reference image quality assessment algorithm, it is detected whether there is blur caused by excessive time and space constraints, if the quality score is lower than a threshold, the smoothing coefficient is dynamically reduced or the spatial constraint threshold is increased, and the consistency and clarity are balanced.

[0045] An electronic device, the electronic device, comprising:

[0046] At least one processor; and,

[0047] The memory in communication connection with the at least one processor; wherein,

[0048] The memory stores a computer program that can be executed by the at least one processor,

[0049] The computer program is executed by the at least one processor,

[0050] So that the at least one processor can execute the generative AI model real-time rendering engine construction method as claimed in any one of claims 1 to 6.

[0051] The generative AI model real-time rendering engine construction method and related device of the present application have the following technical effects and advantages:

[0052] 1. The present application constructs a "generation-rendering" deep fusion architecture, allowing lightweight generation models to directly output renderable intermediate representations, and the rendering pipeline and generation module share a data interface, effectively solving the problem of generation and rendering separation, reducing end-to-end delay from seconds to milliseconds, meeting real-time requirements above 30fps, and greatly improving overall efficiency. Through constraint injection and incremental generation mechanism, feature modulation technology is used to embed hard constraints into the generation process, and incremental generation is triggered for user operations, which not only reduces the error rate of generated content and constraint parameters, but also improves the generation efficiency, breaking through the technical difficulty of poor controllability of AI generation.

[0053] 2. The present application uses multi-level lightweight strategy, conducts knowledge distillation, quantization and pruning at the model level, automatically adapts different GPU computing power at the hardware level, and combines hybrid rendering strategy to make up for the limitations of insufficient hardware adaptability, realizing full-platform compatibility from mobile terminal to high-end workstation. Through space-time constraint technology, exponential weighted smoothing and Gamma correction are used in the time dimension, and 3D Gaussian point cloud and SDF are processed in the space dimension, overcoming the quality defects of space-time consistency loss and improving the visual experience. The related device of the present application further improves the rendering efficiency and operation response speed through hardware-software collaborative optimization, strengthens the real-time interaction experience, and provides efficient 3D content generation and rendering solutions for multiple fields. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The present application is a generative AI model real-time rendering engine construction method structure diagram.

[0055] Figure 2 The present application is a generative AI model real-time rendering engine construction device structure diagram. DETAILED DESCRIPTION

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1, Figure 1 The present invention provides a method for constructing a real-time rendering engine for generative AI models.

[0058] The system acquires multimodal semantic information from user input, parses the multimodal semantic information using the CLIP model, and transforms it into latent vectors and constraint parameters that can be understood by the generative AI model.

[0059] The multimodal semantic information (text, images, parameters, etc.) input by users is a natural way for humans to interact, but generative AI models cannot directly understand this unstructured or heterogeneous information. For example, the text "red curved object" is a natural language description, the image is a pixel matrix, and the parameters are isolated values. AI models need structured feature vectors to perform generative calculations, so they must establish connections through parsing and transformation.

[0060] The process of obtaining multimodal semantic information from user input is as follows: receiving natural language descriptions from user input via a text input interface, acquiring reference images provided by the user via an image acquisition device (such as a camera) or an image upload interface, and receiving specific parameters set by the user through the interactive interface;

[0061] The input text is segmented, stop words are removed, and lemmatization is performed to obtain a standardized text sequence. The input image is resized, normalized, and denoised to unify the image format and quality, such as resizing the image to 224×224 pixels and normalizing the pixel values ​​to the range of [0,1]. The user-input parameters are converted into standardized numerical forms to facilitate subsequent processing, such as converting the hexadecimal values ​​of colors to RGB values, thereby obtaining multimodal semantic information.

[0062] The specific steps for parsing multimodal semantic information using the CLIP model and transforming it into latent vectors and constraint parameters that can be understood by generative AI models are as follows:

[0063] The preprocessed text sequence is input into the text encoder of the CLIP model to obtain the text feature vector. The CLIP text encoder adopts a Transformer structure and extracts the semantic features of the text through deep encoding of the text sequence.

[0064] The preprocessed image is input into the image encoder of the CLIP model to obtain an image feature vector, wherein the image encoder of the CLIP usually adopts a convolutional neural network (such as ResNet, VisionTransformer, etc.) to extract visual features of the image;

[0065] The text feature vector and the image feature vector are fused to obtain a fused multi-modal feature vector, and the process is as follows:

[0066] Since the dimensions of the feature vectors output by the text encoder and the image encoder of the CLIP model may be different, the dimensions of the text feature vector and the image feature vector are first aligned through linear transformation; the aligned feature vectors are weighted and summed according to the importance of the multi-modal input to obtain a multi-modal feature vector, and the formula is as follows:

[0067]

[0068] In the formula, is the multi-modal feature vector, is the aligned text feature vector, is the aligned image feature vector, is the text feature importance weight, and b is the image feature importance weight.

[0069] The fused multi-modal feature vector is mapped to the latent space of the generative AI model to obtain a latent vector, and the process is as follows:

[0070] The latent space dimension of the generative AI model (such as diffusion model, StyleGAN) is obtained, and the dimension of the fused feature vector is adjusted according to the latent space dimension; the adjusted fused feature vector is projected to the latent space of the generative model through a nonlinear mapping network to obtain a latent vector, ensuring that the vector distribution is consistent with the latent space during model training.

[0071] The process of projecting the adjusted fused feature to the latent space of the generative model through a nonlinear mapping network to obtain a latent vector is as follows: first, the fused multi-modal feature vector is batch normalized to stabilize the distribution range; the batch normalized multi-modal feature vector is input into a multi-layer perceptron (MLP) for nonlinear transformation; the output of the multi-layer perceptron (MLP) is regularized to ensure that it conforms to the prior distribution of the latent space, and then the latent vector is obtained. This process is realized through a linear mapping layer, which converts the fused feature vector into a vector matching the dimension of the latent space of the generative AI model.

[0072] The constraint parameter extraction process is as follows: from the multi-modal features extracted from the CLIP model and the parameters input by the user, the constraint parameters required by the generative AI model are extracted. For example, from the text "blue round table", the color constraint parameter "blue" (RGB value) and the shape constraint parameter "round" are extracted; the height, diameter and material constraint parameters of the table are extracted from the size parameters input by the user.

[0073] A lightweight generation model is constructed based on the latent vector and the constraint parameter, and a renderable intermediate representation is directly generated using the lightweight generation model, which includes a 3D Gaussian point cloud, an implicit symbolic distance function and a dynamic texture atlas.

[0074] The process of constructing a lightweight generation model based on a latent vector and a constraint parameter is as follows: the constraint parameters are embedded into the middle layer of the generator through feature modulation to ensure that the generated content meets the hard constraints; a large model (such as the original StableDiffusion3D) is used as a teacher model, and a lightweight model learns the generation distribution through distillation loss; the original floating point value is mapped to the integer domain through a scale factor, and the model weight is quantized to INT8 and the activation value is quantized to FP16; redundant channels are filtered through L1 regularization, and feature channels with high contribution to the generation result are retained; the weights of the first layer channel , if ( is a threshold value), the channel is pruned, and the threshold value is controlled by the validation set accuracy loss.

[0075] The process of directly generating a renderable intermediate representation using the lightweight generation model is as follows:

[0076] The latent vector is input, the initial point set and Gaussian parameters are output by the generator, and the Gaussian parameters are constrained; similar Gaussian points (points with an intersection-over-union ratio of 0.8 or more are merged) are merged through clustering to reduce the number of points to 60% of the original while maintaining the visual effect, wherein the Gaussian parameters include color parameters and shape parameters;

[0077] An 8-layer micro-MLP (64-128 channels per layer) is used, the input is a 3D coordinate, a latent vector and a constraint parameter, and the output is the signed distance from the coordinate to the surface of the object; the shape constraint is converted into a spatial mask, and the coordinates exceeding the boundary are forced to output a positive distance;

[0078] A basic texture feature map is generated by the generator of the lightweight GAN based on the latent vector; the material parameters are converted into texture modulation factors, the texture resolution is dynamically adjusted according to the rendering requirements, and a dynamic texture atlas is obtained.

[0079] A rendering pipeline is constructed based on the generated intermediate representation, and the intermediate representation is converted into a visualized image in combination with rasterization acceleration and ray marching and spatial hashing acceleration technologies;

[0080] The rendering pipeline adopts a three-stage pipeline of "input routing + parallel processing + result fusion", determines the type of the intermediate representation through a feature detector, and automatically switches the processing path according to the type of the intermediate representation (3D Gaussian point cloud / implicit SDF / dynamic texture);

[0081] If the type of the intermediate representation is a 3D Gaussian point cloud, the mean and covariance of the 3D Gaussian point are converted into screen space parameters, the 3D coordinates are projected to 2D screen coordinates through a camera projection matrix, and the scaling of the covariance in the screen space is calculated. All Gaussian points are processed in parallel by the CUDA core of the GPU. For each Gaussian point, its coverage range (such as a 2D elliptical region) on the screen is calculated based on the scaling of the mean and covariance in the screen space, and the pixel color contribution is quickly generated through a precomputed Gaussian weight table. The overlapping pixels are sorted according to the z coordinates after projection, and the occlusion conflict is eliminated by using Alpha blending;

[0082] If the type of the intermediate representation is an implicit SDF, the 3D space is divided into a voxel grid, a hash table is constructed through a hash function, and the SDF sampling points in each voxel are stored to accelerate the coarse positioning of the intersection of a ray and an object. A ray is emitted for each pixel on the screen, the voxel that the ray may pass through is located through a spatial hash table, and the object surface is quickly approached by adaptive stepping in the voxel. The normal vector is calculated at the intersection point by the central difference method, and the pixel color is calculated in combination with the dynamic texture atlas;

[0083] If the type of the intermediate representation is a dynamic texture atlas, the dynamic texture T(x, y, t) is bound to the GPU texture unit, is mapped to the surface of a 3D model (such as a Gaussian point cloud or an SDF surface) through UV coordinates, and only the changed area of the dynamic texture is updated to reduce the GPU data transmission amount.

[0084] Real-time capture of user operations and dynamic updating of constraint conditions of the generated model trigger incremental generation, while the model lightweight, hardware acceleration and hybrid rendering strategy are adopted to optimize the performance of the real-time rendering engine of the generative AI model;

[0085] The process of real-time capture of user operations and dynamic updating of constraint conditions of the generated model is as follows: user operations such as mouse / touch, keyboard, material, VR controller, etc. are captured in real time through a system API, and the sampling frequency is set to 60 Hz to ensure no delay; the incremental change of the constraint parameters is converted from the user operations, and is synchronized to the generated model through an event callback mechanism;

[0086] The trigger delta generation process is as follows: only the region affected by the operation is triggered to generate, if the operation involves the global (such as view rotation), the delta update of the entire intermediate representation is generated; if it is a local operation (such as modifying the local color of an object), the generation range is limited by a spatial mask (mask value 1 represents the region that needs to be updated).

[0087] The performance optimization process of the generative AI model real-time rendering engine is as follows:

[0088] According to the hardware performance (such as GPU computing power, memory), the model precision is automatically switched, FP16 precision is used on high-end GPUs (such as RTX4090), and INT8 precision is automatically switched on mobile devices (such as Snapdragon 8Gen3), and the scale factor is adjusted in real time; enable GPU dedicated acceleration unit, activate TensorCore for Gaussian point cloud matrix operation on NVIDIA GPU, enable Vulkan's dedicated rendering channel (RenderPass) on mobile GPU, and distribute rasterization and ray marching tasks to different calculation units through synchronization barrier for parallel processing; static regions (such as backgrounds and fixed objects) use pre-rendering cache, the results are stored as textures after the first rendering, and subsequent frames are directly sampled, and dynamic regions (such as moving objects) use real-time generation + low-resolution rendering, and the details are restored through a super-resolution module to balance speed and quality.

[0089] Through time filtering and spatial constraint technology, the generated visualization image is consistent in time and space.

[0090] For the current frame and the previous frame , the pixel motion vector is calculated by the optical flow algorithm, which represents the displacement of the pixel from the current frame to the previous frame ; according to the motion vector clustering, the picture is divided into static and dynamic regions; set the smoothing coefficient for static and dynamic regions, and filter the color of each pixel in the current frame according to the motion weight coefficient and the smoothed result of the previous frame; at the same time, the inter-frame brightness difference is calculated for the brightness mutation region (such as light change), if the inter-frame brightness difference is greater than the preset threshold, then trigger Gamma correction.

[0091] The spatial gradient of the generated intermediate representation is checked, the distance between adjacent points of 3D Gaussian point cloud is calculated, and if it exceeds the threshold, a transition point is inserted; the SDF value gradient of adjacent sampling points in space is calculated, and if it is mutated, the SDF function is optimized through a smoothing term to make the object surface smoother; the local texture gradient of the dynamic texture atlas

[0092] is calculated, and if the texture difference between adjacent pixels in a region exceeds the threshold, it is processed by bilateral filtering smoothing.

[0093] The CLIP features of the current frame and the previous frame are extracted and the cosine similarity is calculated, if the cosine similarity is less than a threshold, the latent vector of the generation model is adjusted backtracking, so that the semantics of the generated content are consistent with the historical frames; through a no-reference image quality assessment algorithm (such as BRISQUE), it is detected whether the picture exists blur caused by excessive spatiotemporal constraints, if the quality score is lower than a threshold, then the smoothing coefficient is dynamically reduced or the spatial constraint threshold is increased, balancing consistency and clarity.

[0094] It should be noted that through the spatiotemporal constraint technology, exponential weighted smoothing and Gamma correction are used in the time dimension, and 3D Gaussian point cloud and SDF are processed in the spatial dimension, which overcomes the quality defects of spatiotemporal consistency loss and improves the visual experience.

[0095] Embodiment 2, Figure 2 The application provides a device for constructing a real-time rendering engine of a generative AI model.

[0096] An electronic device comprises at least one processor and a memory connected in communication with the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method for constructing a real-time rendering engine of a generative AI model.

[0097] Through hardware-software collaborative optimization, the rendering efficiency and operation response speed are further improved, the real-time interaction experience is strengthened, and an efficient 3D content generation and rendering solution is provided for multiple fields.

[0098] The above formulas are all dimensionless numerical calculations, the formulas are obtained by software simulation of a large amount of data to obtain a formula of the nearest real situation, and the preset parameters in the formula are set by a person skilled in the art according to the actual situation.

[0099] The above embodiments can be realized wholly or partially by software, hardware, firmware or any combination thereof. When realized by software, the above embodiments can be realized wholly or partially in the form of a computer program product.

[0100] Those skilled in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solutions. The person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0101] In addition, each function module in each embodiment of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module.

[0102] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0103] Finally: the above is only a preferred embodiment of the present application and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A generative AI model real-time rendering engine construction method, characterized in that, The method comprises the following steps: Obtain the multi-modal semantic information input by the user, analyze the multi-modal semantic information through the CLIP model, and convert it into a latent vector and constraint parameters that can be understood by the generative AI model; Construct a lightweight generative model based on the latent vector and constraint parameters, directly generate a renderable intermediate representation using the lightweight generative model, and the intermediate representation includes a 3D Gaussian point cloud, an implicit symbolic distance function, and a dynamic texture atlas; Based on the generated intermediate representation, construct a rendering pipeline, and combine rasterization acceleration, ray marching, and spatial hashing acceleration techniques to convert the intermediate representation into a visual image; Real-time capture user operations and dynamically update the constraint conditions of the generative model, the process is as follows: Real-time capture user operations through system API, sampling frequency is set to 60Hz to ensure no delay, convert user operations into incremental changes of constraint parameters, and synchronize to the generative model through event callback mechanism to dynamically update the constraint conditions of the generative model; Trigger incremental generation, the process is as follows: Only the area affected by the operation is triggered to generate, if the operation involves the whole, the incremental update of all intermediate representations is generated, if it is a local operation, the generation range is limited through spatial mask; At the same time, optimize the system performance by using model lightweight, hardware acceleration and hybrid rendering strategy, the process is as follows: According to the hardware performance, automatically switch the model precision, use FP16 precision on high-end GPU, automatically switch to INT8 precision on mobile GPU, enable GPU dedicated acceleration unit for Gaussian point cloud matrix operation, enable dedicated rendering channel for mobile GPU, distribute rasterization and ray marching tasks to different calculation units through synchronization barrier for parallel processing, use pre-rendering cache for static area, store the result as texture after the first rendering, and directly sample the subsequent frames, use real-time generation + low resolution rendering for dynamic area, restore details through super resolution module, balance speed and quality; Through time filtering and spatial constraint technology, the generated content is consistent in time and space.

2. The generative AI model real-time rendering engine construction method of claim 1, wherein, The specific steps of analyzing the multi-modal semantic information through the CLIP model and converting it into a latent vector and constraint parameters that can be understood by the generative AI model are as follows: Input the preprocessed text sequence into the text encoder of the CLIP model to obtain a text feature vector, wherein the text encoder of the CLIP uses a Transformer structure to extract the semantic features of the text through deep encoding of the text sequence; Input the preprocessed image into the image encoder of the CLIP model to obtain an image feature vector, wherein the image encoder of the CLIP usually uses a convolutional neural network to extract the visual features of the image; Fuse the text feature vector and the image feature vector to obtain a fused multi-modal feature vector, the formula is as follows: ； In the formula, is a multi-modal feature vector, is an aligned text feature vector, is an aligned image feature vector, is a text feature importance weight, and b is an image feature importance weight. Map the fused multi-modal feature vector to the latent space of the generative AI model to obtain the latent vector; Extract the constraint parameters required by the generative AI model from the multi-modal features extracted from the CLIP model and the parameters input by the user.

3. The generative AI model real-time rendering engine construction method of claim 2, wherein, The fused multi-modal feature vector is mapped to the latent space of the generative AI model to obtain a latent vector, and the process is as follows: The latent space dimension of the generative AI model is obtained, and the fused feature vector dimension is adjusted according to the latent space dimension; The adjusted fused feature vector is projected into the latent space of the generative model through a nonlinear mapping network to obtain a latent vector, ensuring that the vector distribution is consistent with the latent space during model training.

4. The generative AI model real-time rendering engine construction method of claim 3, wherein, The process of projecting the adjusted fused feature into the latent space of the generative model through a nonlinear mapping network to obtain a latent vector is as follows: Batch normalization is performed on the fused multi-modal feature vector to stabilize the distribution range; The batch-normalized multi-modal feature vector is input into a multi-layer perceptron (MLP) for nonlinear transformation; the output of the multi-layer perceptron (MLP) is regularized to ensure compliance with the prior distribution of the latent space, and then a latent vector is obtained.

5. The generative AI model real-time rendering engine construction method of claim 4, wherein, The process of directly generating a renderable intermediate representation using the lightweight generative model is as follows: With the latent vector as input, the generator outputs an initial point set and Gaussian parameters, and constraints are imposed on the Gaussian parameters. Similar Gaussian points are merged through clustering to generate a 3D Gaussian point cloud; A micro-MLP is used to generate an implicit signed distance function, with 3D coordinates, latent vectors, and constraint parameters as input, and the signed distance from the coordinates to the object surface as output. Shape constraints are converted into a spatial mask, and coordinates exceeding the boundary are forced to output a positive distance; Based on the latent vector, a lightweight GAN generator is used to generate a basic texture feature map, and material parameters are converted into texture modulation factors. The texture resolution is dynamically adjusted according to the rendering requirements to obtain a dynamic texture atlas.

6. The generative AI model real-time rendering engine construction method of claim 5, wherein, The process of making the generated visual image consistent in time and space through time filtering and spatial constraint technology is as follows: The CLIP features of the current frame and the previous frame are extracted and the cosine similarity is calculated. If the cosine similarity is less than a threshold, the latent vector of the generative model is adjusted to make the semantic content consistent with the historical frame; A no-reference image quality assessment algorithm is used to calculate the picture quality score to detect whether the picture is blurred due to excessive temporal and spatial constraints. If the quality score is below a threshold, the smoothing coefficient is dynamically reduced or the spatial constraint threshold is increased to balance consistency and clarity.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program is executed by the at least one processor, so that the at least one processor can execute the generative AI model real-time rendering engine construction method of any one of claims 1 to 6.

Images