Model processing method and system based on artificial intelligence and related equipment

Abstract

The invention discloses a model processing method and system based on artificial intelligence and related equipment, relates to the technical field of artificial intelligence and computer graphics, does not need manual intervention to optimize a generation result, and only needs to predict depth information of an object from a single or multiple pictures by using a pre-trained depth estimation network through a deep learning technology. The method comprises the steps of generating a target picture, generating a depth map, reconstructing a target three-dimensional grid through a neural rendering technology in combination with the target picture and the depth map, performing texture mapping on the generated target three-dimensional grid, and improving model details and rendering quality through an optimization algorithm so as to quickly generate a texture self-adaptive and optimized high-quality three-dimensional model. Through the lightweight model and memory optimization technology, the texture self-adaptive and optimized three-dimensional model is subjected to hardware adaptation to adapt to different hardware configurations, the efficiency of generating the three-dimensional model is improved, and the cost of generating the three-dimensional model is reduced.

Description

Technical Field

[0001] The present application relates to the technical fields of artificial intelligence and computer graphics, and more specifically, to an artificial intelligence-based model processing method, system, and related equipment. Background Art

[0002] With the rapid development of technologies such as Virtual Reality (VR), Augmented Reality (AR), and Metaverse, the demand for high-quality 3D models is increasing.

[0003] In recent years, artificial intelligence (AI) technology has made significant progress in image processing and generation, especially in the application of generative adversarial networks (GANs) and diffusion models in 2D image generation. However, directly applying these technologies to 3D model generation is limited by factors such as generation efficiency, model detail, and rendering quality.

[0004] Traditional 3D model generation methods rely on complex input data (such as point clouds or depth maps), or require a lot of manual intervention to optimize the generation results, resulting in low efficiency and high cost of generating 3D models.

[0005] Therefore, how to improve the efficiency of generating 3D models and reduce the cost of generating 3D models is an urgent problem to be solved in this application. Summary of the Invention

[0006] In view of this, the present application discloses an artificial intelligence-based model processing method, system and related equipment, aiming to improve the efficiency of generating three-dimensional models and reduce the cost of generating three-dimensional models.

[0007] In order to achieve the above purpose, the disclosed technical solutions are as follows:

[0008] The first aspect of the present application discloses a model processing method based on artificial intelligence, the method comprising:

[0009] Obtain a target image; wherein the target image is an image that meets the input requirements of the adaptation model;

[0010] Predicting the depth information of the object from the target image using a pre-trained depth estimation network, and generating a depth map based on the depth information;

[0011] Constructing a target three-dimensional grid using a preset construction technology, the target image, and the depth map;

[0012] Performing texture mapping on the target three-dimensional grid, and performing texture optimization on the texture-mapped target three-dimensional grid through an optimization algorithm;

[0013] The target 3D mesh after texture optimization is hardware adapted and performance optimized.

[0014] Preferably, obtaining the target image includes:

[0015] Get the image to be processed;

[0016] Preprocessing the image to be processed to obtain a target image;

[0017] The process of preprocessing the image to be processed includes:

[0018] Scaling the image to be processed to a preset resolution;

[0019] Performing color space conversion on the image to be processed; wherein the color space conversion is to convert the format of the image to be processed into a preset color coding format;

[0020] Normalizing the image to be processed so as to map the pixel values ​​of the image to be processed to a preset interval to ensure that the values ​​conform to the input range of the neural network activation function;

[0021] Detect the main area of ​​the image to be processed and segment the background.

[0022] Preferably, predicting the depth information of the object from the target image by using a pre-trained depth estimation network and generating a depth map based on the depth information includes:

[0023] Downsampling the target image through the encoder of the convolutional neural network, and outputting a feature map with a size change;

[0024] Upsampling the target image through transposed convolution and resizing the feature map, outputting a single-channel depth map;

[0025] Constraining the depth value of the single-channel depth map to a preset interval by outputting an activation function;

[0026] The pre-trained model is loaded, and the constrained channel depth map is input into the pre-trained model to generate a depth map.

[0027] Preferably, constructing the target three-dimensional grid by using a preset construction technology, the target image and the depth map includes:

[0028] Optimizing the vertex positions, normal directions, and texture mapping of the initial three-dimensional mesh by differentiable rendering; wherein the initial three-dimensional mesh is a three-dimensional mesh that has not been subjected to rendering optimization;

[0029] The initial 3D mesh after rendering optimization is jointly optimized using graphics processor acceleration and neural rendering functions to generate a target 3D mesh.

[0030] Preferably, the performing texture mapping on the target three-dimensional mesh and optimizing the texture of the texture-mapped target three-dimensional mesh by using an optimization algorithm comprises:

[0031] Through differentiable rendering technology, the vertex color of the target three-dimensional mesh, the UV coordinates of the target three-dimensional mesh, the lighting of the input image of the target three-dimensional mesh and the color information of the target three-dimensional mesh are dynamically aligned to achieve texture optimization of the texture of the target three-dimensional mesh after texture mapping.

[0032] Preferably, performing hardware adaptation and performance optimization on the texture-optimized target three-dimensional mesh includes:

[0033] Through lightweight models, memory optimization technology and dynamic quantization strategy, the target three-dimensional mesh after texture optimization is hardware adapted and performance optimized.

[0034] The second aspect of the present application discloses an artificial intelligence-based model processing system, the system comprising:

[0035] An acquisition unit, configured to acquire a target image; wherein the target image is an image that meets the input requirements of the adaptation model;

[0036] A prediction generation unit, configured to predict depth information of an object from the target image using a pre-trained depth estimation network, and generate a depth map based on the depth information;

[0037] A construction unit, configured to construct a target three-dimensional grid using a preset construction technology, the target image, and the depth map;

[0038] A mapping optimization unit, configured to perform texture mapping on the target three-dimensional mesh and perform texture optimization on the texture-mapped target three-dimensional mesh using an optimization algorithm;

[0039] The adaptation and optimization unit is used to perform hardware adaptation and performance optimization on the target three-dimensional mesh after texture optimization.

[0040] Preferably, the acquisition unit includes:

[0041] Acquisition module, used to obtain the image to be processed;

[0042] A preprocessing module, used to preprocess the image to be processed to obtain a target image;

[0043] The preprocessing module includes:

[0044] A scaling submodule, configured to scale the image to be processed to a preset resolution;

[0045] A conversion submodule, configured to perform color space conversion on the image to be processed; wherein the color space conversion is to convert the format of the image to be processed into a preset color coding format;

[0046] A normalization submodule, configured to perform numerical normalization on the image to be processed so as to map the pixel values ​​of the image to be processed to a preset interval to ensure that the values ​​conform to the input range of the neural network activation function;

[0047] The detection and segmentation submodule is used to detect the main area of ​​the image to be processed and segment the background.

[0048] A third aspect of the present application discloses a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the artificial intelligence-based model processing method as described in any one of the first aspects.

[0049] The fourth aspect of the present application discloses an electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of the first aspects.

[0050] It can be seen from the above technical solution that the present application discloses an artificial intelligence-based model processing method, system and related equipment to obtain a target image, wherein the target image is an image that meets the input requirements of the adaptation model, and predicts the depth information of the object from the target image through a pre-trained depth estimation network, and generates a depth map based on the depth information, constructs a target three-dimensional grid through a preset construction technology, a target image and a depth map, performs texture mapping on the target three-dimensional grid, and performs texture optimization on the target three-dimensional grid after texture mapping through an optimization algorithm, and performs hardware adaptation and performance optimization on the target three-dimensional grid after texture optimization.

[0051] Through the above solution, there is no need for human intervention to optimize the generation results. It is only necessary to use deep learning technology and a pre-trained depth estimation network to predict the depth information of the object from a single or multiple pictures, generate a depth map, combine the target picture and the depth map, reconstruct the target three-dimensional mesh through neural rendering technology, perform texture mapping on the generated target three-dimensional mesh, and improve the model details and rendering quality through optimization algorithms to quickly generate texture self-adaptive and optimized high-quality three-dimensional models. Through lightweight models and memory optimization technology, the texture self-adaptive and optimized three-dimensional models are hardware-adapted to adapt to different hardware configurations, thereby improving the efficiency of generating three-dimensional models and reducing the cost of generating three-dimensional models. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0053] Figure 1 A flowchart of an artificial intelligence-based model processing method disclosed in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the preprocessing code disclosed in the embodiment of this application;

[0055] Figure 3 A schematic diagram of the process of generating a depth map disclosed in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of a depth map generation code disclosed in an embodiment of the present application;

[0057] Figure 5 The target three-dimensional grid is constructed as disclosed in the embodiments of this application;

[0058] Figure 6 A schematic diagram of a target three-dimensional grid code for reconstruction disclosed in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of code for optimizing a target three-dimensional mesh after texture mapping by using an optimization algorithm disclosed in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the code for hardware adaptation of texture self-adaptation and optimized three-dimensional model disclosed in an embodiment of the present application;

[0061] Figure 9 This is a schematic diagram of the structure of an artificial intelligence-based model processing system disclosed in an embodiment of the present application;

[0062] Figure 10 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0065] As can be seen from the background art, traditional 3D model generation methods rely on complex input data (such as point clouds or depth maps) or require a lot of manual intervention to optimize the generation results, resulting in low efficiency and high cost of generating 3D models. Therefore, how to improve the efficiency and reduce the cost of generating 3D models is an urgent problem to be solved in this application.

[0066] In order to solve the above problems, the present application discloses a model processing method, system and related equipment based on artificial intelligence, which does not require human intervention to optimize the generation results. It only needs to use deep learning technology and a pre-trained depth estimation network to predict the depth information of the object from a single or multiple pictures, generate a depth map, combine the target picture and the depth map, reconstruct the target three-dimensional mesh through neural rendering technology, perform texture mapping on the generated target three-dimensional mesh, and improve the model details and rendering quality through optimization algorithms to quickly generate texture self-adaptation and optimized high-quality three-dimensional models. Through lightweight models and memory optimization technology, the texture self-adaptation and optimized three-dimensional models are hardware-adapted to adapt to different hardware configurations, improve the efficiency of generating three-dimensional models, and reduce the cost of generating three-dimensional models. The specific implementation method is specifically described through the following embodiments.

[0067] It should be noted that the artificial intelligence-based model processing method, system and related equipment provided in this application can be used in artificial intelligence and computer graphics, game development, film and television production, virtual reality, e-commerce and other fields. The above is only an example and does not limit the application field of the artificial intelligence-based model processing method, system and related equipment provided in this application.

[0068] refer to Figure 1FIG. 1 is a flow chart of an artificial intelligence-based model processing method disclosed in an embodiment of the present application. The artificial intelligence-based model processing method mainly includes the following steps:

[0069] S101: Obtain a target image; wherein the target image is an image that meets the input requirements of the adaptation model.

[0070] In S101, a picture to be processed is obtained, and preprocessed to obtain a target picture that meets the input requirements of the adaptation model.

[0071] The input image is cropped, scaled, and normalized to fit the input requirements of the depth estimation network. The specific process of preprocessing the image to be processed is as follows:

[0072] A1: Scale the image to be processed to the preset resolution.

[0073] In A1, the image to be processed is scaled to a preset resolution to achieve size standardization.

[0074] The preset resolution is set according to the actual situation and is not specifically limited in this application. The preset resolution is preferably 256x256 resolution.

[0075] The images to be processed are uniformly scaled to a resolution of 256x256 to meet the input size requirements of the subsequent ViT model.

[0076] A2: Performing color space conversion on the image to be processed; wherein the color space conversion is to convert the format of the image to be processed into a preset color coding format.

[0077] The color space conversion converts the OpenCV default BGR format to RGB format.

[0078] A3: Normalize the image to be processed to map its pixel values ​​to a preset range to ensure that it complies with the input range of the neural network activation function.

[0079] The preset interval is not specifically limited in this application and is preferably the interval [0, 1].

[0080] In A3, the pixel values ​​are mapped to the interval [0, 1], which is consistent with the input range of the neural network activation function.

[0081] A4: Detect the subject area of ​​the image to be processed and segment the background.

[0082] In A4, the system automatically detects the subject area and uses models such as SAM to segment the background. It supports GPU-accelerated preprocessing, which includes but is not limited to resizing operations accelerated by cuDNN.

[0083] The specific preprocessing code is as follows Figure 2 shown.

[0084] S102: Predicting the depth information of the object from the target image through the pre-trained depth estimation network, and generating a depth map based on the depth information.

[0085] In S102, a pre-trained depth estimation network is used to predict the depth information of the object from the image and generate a depth map. The depth estimation network can be implemented based on a convolutional neural network (CNN).

[0086] The network architecture of the depth estimation network includes an encoder, a skip connection, a transposed convolution (ConvTranspose2d), a decoder, and an output activation function.

[0087] The encoder consists of two convolutional layers, with the number of channels gradually increasing from 3 (RGB) to 64 and 128, and downsampling using a stride of 2.

[0088] Specifically, the depth information of the object is predicted from the target image through the pre-trained depth estimation network, and the depth map is generated based on the depth information. Figure 3 shown.

[0089] S301: Downsample the target image through the encoder of the convolutional neural network and output the feature map with a changed size.

[0090] Input 256x256 → After the first layer: 128x128 → After the second layer: 64x64.

[0091] The lack of skip connections results in loss of detailed information during decoding.

[0092] S302: Upsample the target image through transposed convolution and feature map size change, and output a single-channel depth map.

[0093] S303: Constraining the depth value of the single-channel depth map to a preset range through an output activation function.

[0094] The preset interval is set according to actual conditions and is not specifically limited in this application. The preset interval of this application is preferably the interval [0, 1].

[0095] In S303 , the depth value is constrained to the interval [0, 1] by outputting the activation function Sigmoid.

[0096] S304: Load the pre-trained model, input the constrained channel depth map into the pre-trained model, and generate a depth map.

[0097] In S304 , a pre-trained model (such as depth_model.pth) is loaded, the constrained channel depth map is input into the pre-trained model, and reasoning is performed through the predict_depth function of the reasoning process to generate a depth map.

[0098] The code to generate the depth map is as follows Figure 4 shown.

[0099] S103: Construct a target three-dimensional mesh using a preset construction technology, the target image, and the depth map.

[0100] In S103, the target three-dimensional mesh is reconstructed by using neural rendering technology and combining the target image and the depth map.

[0101] The process leverages multi-view geometry and deep learning to optimize vertex positions and texture mapping of 3D meshes.

[0102] Fuse multi-view target images and their corresponding depth maps to generate a 3D mesh and save it as an .obj file. This is currently implemented as placeholder logic that randomly generates vertices and faces.

[0103] PyTorch3D: Provides mesh data structures and IO functions, but requires integration with 3D reconstruction algorithms (such as TSDF fusion and neural implicit fields) to implement the complete process.

[0104] Specifically, the process of constructing the target 3D mesh through the preset construction technology, target image and depth map, such as Figure 5 shown.

[0105] S501: Optimizing vertex positions of an initial three-dimensional mesh, a normal direction of the initial three-dimensional mesh, and texture mapping of the initial three-dimensional mesh through differentiable rendering; wherein the initial three-dimensional mesh is a three-dimensional mesh that has not been subjected to rendering optimization.

[0106] S502: Utilizing graphics processor acceleration and neural rendering (neural_rendering) function, jointly optimize the initial 3D mesh after rendering optimization to generate a target 3D mesh.

[0107] Specifically, the process of jointly optimizing the rendered optimized initial 3D mesh using GPU acceleration and the neural_rendering function is as follows:

[0108] This approach jointly optimizes multi-view RGB images and their corresponding depth maps, leveraging the complementary nature of visual appearance (texture, color) and geometric information (depth, spatial structure). While traditional methods typically process either image or depth data separately, this application dynamically fuses the two through deep learning models (such as neural implicit fields or TSDF fusion algorithms), significantly improving the geometric accuracy and surface detail of 3D meshes.

[0109] This application uses a deep learning model (such as a neural implicit field or TSDF fusion algorithm) to achieve dynamic fusion of the two, solving problems such as model holes and texture distortion caused by a single data source (such as relying solely on depth maps or images). It is particularly suitable for complex objects or occluded scenes. The specific process of solving problems such as model holes and texture distortion caused by a single data source (such as relying solely on depth maps or images): This application introduces neural rendering technology (such as the PyTorch3D rendering pipeline) to optimize the vertex position, normal direction, and texture mapping of the mesh through differentiable rendering. Compared with traditional multi-view stereo matching (MVS), this solution can learn the joint representation of geometry and appearance end-to-end, reducing the need for manual post-processing. While ensuring surface smoothness (avoiding jagged edges or holes), it automatically generates high-quality textures consistent with the input image, significantly improving the realism of the rendering.

[0110] This application proposes a deep learning-based multi-view depth map fusion strategy that addresses perspective conflict and noise issues in multi-view data through a self-attention mechanism or graph neural network (GNN). For example, cross-view feature alignment technology is used to map depth maps from different perspectives into a unified 3D space, generating consistent point cloud data. This overcomes the stitching errors caused by perspective differences in traditional methods, and performs particularly well in areas with weak textures or repetitive structures.

[0111] Through GPU parallel computing and memory optimization technology, batch processing of multi-view data is achieved. For example, by leveraging PyTorch's CUDA acceleration capabilities, the computationally intensive tasks of point cloud fusion and mesh reconstruction are deployed to the GPU. Using GPU acceleration, all calculations are performed on the CPU by default, thereby improving generation efficiency.

[0112] Supports batch processing (e.g., processing multiple viewpoints simultaneously). The neural_rendering function implements multi-view data fusion logic to generate a realistic mesh. Randomly generated vertices and patches are applied to the actual scene, and the resulting mesh is smoothed to avoid holes or jagged edges. Texture mapping information (geometry only) is included. Multi-view depth maps are fused into a consistent point cloud, and the target 3D mesh is reconstructed.

[0113] The code for reconstructing the target 3D mesh is as follows Figure 6 shown.

[0114] S104: performing texture mapping on the target three-dimensional mesh, and performing texture optimization on the texture-mapped target three-dimensional mesh using an optimization algorithm.

[0115] In S104, texture mapping is performed on the generated 3D mesh, and the texture of the target 3D mesh after texture mapping is optimized using an optimization algorithm to improve model detail and rendering quality. Specifically, differentiable rendering technology (such as PyTorch3D's differentiable rasterization) is used to dynamically align the vertex colors of the target 3D mesh, the UV coordinates of the target 3D mesh, the lighting of the input image of the target 3D mesh, and the color information of the target 3D mesh to achieve texture optimization of the texture of the target 3D mesh after texture mapping.

[0116] The specific optimization algorithm process is as follows:

[0117] 1. Core objectives:

[0118] Map the texture to a 3D mesh and render the resulting image, saving it as a PNG file.

[0119] Use random vertex colors ( TexturesVertex(verts_features=torch.randn_like(...) ) as textures, which are independent of the input image, to achieve realistic texture mapping. Use the input images parameter to render the result associated with the input image. Match the camera parameters ( PerspectiveCameras ) and lighting ( PointLights ) to the actual scene. PerspectiveCameras and PointLights are both default values.

[0120] 2. Dependent components:

[0121] PyTorch3D's rendering pipeline (Rasterizer + Shader) provides mesh rendering capabilities and combines texture mapping algorithms (such as UV mapping and map generation) to achieve complete functionality.

[0122] This paper uses neural rendering technology and deep learning algorithms to automatically learn the optimal way to map textures, enabling more efficient and accurate mapping of textures onto 3D meshes. For example, using UV mapping technology combined with deep learning optimization algorithms can produce high-quality texture mapping results, avoiding common problems with traditional methods.

[0123] This application proposes an end-to-end texture mapping framework combined with Neural Rendering, which uses the input image to directly drive texture generation. Through differentiable rendering techniques (such as PyTorch3D's differentiable rasterization), the vertex colors and UV coordinates of the 3D mesh are dynamically aligned with the lighting and color information of the input image, achieving automatic adaptation and optimization of the texture. Unlike traditional manual UV unwrapping or fixed texture mapping, this method optimizes texture parameters through backpropagation, significantly reducing manual intervention. It solves the problems of texture stretching and seams in complex geometric structures, generates texture maps that are highly consistent with the input image, and enhances visual realism.

[0124] This application introduces a multi-scale generative adversarial network (Multi-Scale GAN) to enhance details and repair local areas of the initial texture:

[0125] At low resolutions, the overall color distribution is optimized; at high resolutions, surface details (such as scratches and bumps) are added. During detail enhancement and local restoration of the initial texture, an attention mechanism is incorporated to prioritize restoration of key areas (such as object edges and highlights), avoiding wasted global computational resources. While maintaining generation efficiency, this approach improves texture resolution and detail richness, making it particularly suitable for high-fidelity rendering requirements (such as film-quality models).

[0126] Lighting parameters (such as light source position and intensity) and material properties (such as roughness and metallicity) are incorporated into optimization targets and jointly optimized through differentiable rendering. For example, using the physically based rendering (PBR) pipeline, the lighting differences between the rendered image and the input image are minimized, dynamically adjusting the material's reflective properties. The resulting model is not only texture-accurate but also physically consistent under all lighting conditions, making it suitable for dynamic lighting scenarios such as AR / VR.

[0127] Dynamically adjust texture resolution and compression rate based on hardware performance:

[0128] On high-end GPUs, lossless compression formats (such as BC7) are used to preserve 4K texture details. On mobile devices, adaptive resolution (such as downscaling to 1K based on screen size) and lossy compression (such as ASTC) are used. Combined with super-resolution techniques (such as ESRGAN), details can be restored in real time from low-resolution textures.

[0129] Use pre-trained semantic segmentation models (such as Mask R-CNN) to identify local areas of the 3D mesh (such as faces and logos), and apply differentiated optimization strategies for different areas: high-precision texture generation is used for key areas (such as faces); lightweight interpolation algorithms are used for non-critical areas (such as backgrounds).

[0130] A user-interactive optimization framework is built, allowing users to annotate unsatisfactory areas (such as blurred textures or geometric distortion). The system then dynamically adjusts optimization weights through reinforcement learning, iteratively generating improved versions. The optimization process supports real-time preview, shortening debugging cycles.

[0131] The specific code for optimizing the target three-dimensional grid after texture mapping through the optimization algorithm is as follows Figure 7 shown.

[0132] S105: Perform hardware adaptation and performance optimization on the target three-dimensional mesh after texture optimization.

[0133] In S105, the texture-optimized target 3D mesh is hardware-adapted and performance-optimized through lightweight models, memory optimization technology, and dynamic quantization strategies.

[0134] Through lightweight models and memory optimization technology, it adapts to different hardware configurations and improves generation efficiency:

[0135] Dynamic quantization (DynamicQuantization) is performed on the input PyTorch model according to the use_lite parameter to reduce the model size and improve the inference speed.

[0136] When use_lite=True, all nn.Linear layers in the model are dynamically quantized. During dynamic quantization, weights are converted to int8 and dynamically dequantized during inference. The quantized model or the original model is returned.

[0137] In response to different hardware resource limitations, a dynamic quantization strategy is proposed:

[0138] During the inference phase, the system dynamically chooses whether to quantize the model weights (for example, converting floating-point precision FP32 to int8) based on hardware performance (such as GPU memory and CPU computing power) to achieve a flexible balance between model volume compression and inference acceleration.

[0139] The quantization process only targets key computational layers (such as nn.Linear layers), retaining some high-precision calculations to maintain the geometric and texture quality of the model output.

[0140] While reducing the model size by more than 70%, the accuracy loss is controlled within 3%, significantly lowering the deployment threshold for low-end hardware (such as mobile devices and embedded platforms).

[0141] Design a hardware adaptation layer to automatically select the optimal computing backend through runtime environment detection (such as CUDA and OpenCL support):

[0142] Enables full model and CUDA-accelerated preprocessing (such as cuDNN-accelerated resize operations) on high-end GPUs.

[0143] Switch to lightweight models (such as MobileNet-based encoders) and CPU-optimized operators (such as the Intel MKL library) on low-end devices.

[0144] A three-level memory management strategy is proposed to optimize resource allocation for different generation stages:

[0145] Preprocessing stage: Use block loading and streaming processing technology to avoid video memory overflow caused by loading high-resolution images at one time;

[0146] Inference stage: Memory reuse technology is used to share the memory space of intermediate feature maps and reduce duplicate allocation.

[0147] Post-processing stage: Use a compressed storage format (such as the Draco compression algorithm) for the generated 3D mesh data to reduce storage and transmission overhead.

[0148] For example, 4K resolution images can be processed on a GPU with 8GB of video memory, memory usage is reduced by more than 40%, and large-scale batch processing is supported.

[0149] Based on the characteristics of 3D generation tasks, a lightweight network architecture is designed:

[0150] The encoder uses depthwise separable convolution to replace traditional convolution to reduce the number of parameters; the decoder introduces the channel attention mechanism to maintain feature expression capabilities with low computational complexity; key modules (such as the depth estimation network) support dynamic pruning to automatically simplify the network structure according to the input complexity.

[0151] The code for hardware adaptation of texture self-adaptation and optimized 3D model is as follows Figure 8 shown.

[0152] This application aims to use deep learning technology to extract three-dimensional information from a single or multiple images and quickly generate high-quality 3D models. The method includes the following core steps:

[0153] 1. Image preprocessing: Preprocess the input image, including cropping, scaling, and normalization, to adapt to the model input requirements.

[0154] Efficient generation of depth information: This invention predicts depth information from a single or multiple ordinary RGB images through a pre-trained depth estimation network, without the need for additional depth sensors or complex multi-view geometry calculations, greatly reducing the threshold and cost of generating depth maps.

[0155] The depth estimation network architecture (such as the encoder-decoder structure) of this application has been optimized to better handle the depth information of complex scenes and small objects.

[0156] The specific depth estimation network architecture (such as the encoder-decoder structure) is optimized as follows:

[0157] Depth estimation network architecture:

[0158] (1) Encoder input: The input image size is 256×256, and the number of channels is 3 (RGB). Convolutional layer: Convolution kernel size: 3×3, the number of channels: increased from 3 (RGB) to 64, stride: 2 (downsampling), output feature map size: 128×128, activation function: ReLU.

[0159] (2) Decoder input: low-resolution feature map output by the encoder (64×64, 128 channels). First layer convolution kernel size: 4×4. Number of channels: reduced from 128 to 64, stride: 2 (upsampling), activation function: ReLU.

[0160] The encoder can extract rich feature information by gradually increasing the number of channels and performing downsampling; the decoder can generate a high-resolution depth map by upsampling through transposed convolution, and use the Sigmoid activation function to constrain the depth value to the [0, 1] interval, ensuring the stability and consistency of the depth map.

[0161] By encoder optimization:

[0162] Gradually increasing the number of channels and downsampling: The encoder consists of multiple convolutional layers, gradually increasing the number of channels in the input image from 3 (RGB) to higher numbers (such as 64, 128, and so on). This gradual increase in the number of channels allows for the extraction of richer feature information. Downsampling is performed using convolution operations with a stride of 2, gradually reducing the spatial size of the feature map, thereby extracting more abstract features in a smaller spatial dimension. For example, if the input image is 256×256, the feature map size becomes 128×128 after the first convolution layer, and then becomes 64×64 after the second convolution layer.

[0163] The present invention can adapt input images of various formats and qualities to the input requirements of the depth estimation network through image preprocessing modules (such as cropping, scaling, normalization, etc.), thereby improving the universality of the system.

[0164] Through technologies such as GPU acceleration, depth maps can be quickly generated, significantly improving generation efficiency and making them suitable for large-scale applications. This seamless integration not only improves the efficiency of 3D reconstruction, but also enhances the quality of the final 3D model, especially in terms of texture mapping and detail expression.

[0165] 2. Depth information prediction: Use the pre-trained depth estimation network to predict the depth information of the object from the image and generate a depth map.

[0166] 3D mesh reconstruction: Combine the image and depth map to reconstruct the 3D mesh through neural rendering technology.

[0167] 4. Texture mapping and optimization: Texture mapping is performed on the generated 3D mesh, and optimization algorithms are used to improve model details and rendering quality.

[0168] 5. Hardware adaptation and performance optimization: Through lightweight models and memory optimization technology, it adapts to different hardware configurations and improves generation efficiency.

[0169] Compared with the existing technology, this solution has the following advantages and innovations:

[0170] 1. Efficient generation: 3D models are generated directly from images through AI large models, significantly improving generation efficiency and reducing manual intervention.

[0171] 2. High-quality output: Combining deep learning and neural rendering technology, the generated 3D models have high details and rendering quality.

[0172] 3. Hardware adaptability: Through lightweight models and memory optimization technology, it adapts to different hardware configurations and improves the universality of the system.

[0173] 4. Wide application: It can be widely used in game development, film and television production, virtual reality, e-commerce and other fields, and has important commercial value.

[0174] The beneficial effects of the embodiments of the present application are as follows: no manual intervention is required to optimize the generation results. It is only necessary to use deep learning technology and a pre-trained depth estimation network to predict the depth information of the object from a single or multiple pictures, generate a depth map, combine the target picture and the depth map, reconstruct the target three-dimensional mesh through neural rendering technology, perform texture mapping on the generated target three-dimensional mesh, and improve the model details and rendering quality through optimization algorithms to quickly generate a high-quality three-dimensional model with texture self-adaptation and optimization. Through lightweight models and memory optimization technology, the texture self-adaptation and optimized three-dimensional model are hardware-adapted to adapt to different hardware configurations, thereby improving the efficiency of generating three-dimensional models and reducing the cost of generating three-dimensional models.

[0175] Based on the above embodiment Figure 1 The disclosed model processing method based on artificial intelligence, the embodiment of the present application also discloses a model processing system based on artificial intelligence, such as Figure 9 As shown, the artificial intelligence-based model processing system includes:

[0176] The acquisition unit 901 is used to acquire a target image; wherein the target image is an image that meets the input requirements of the adaptation model;

[0177] The prediction generation unit 902 is configured to predict the depth information of the object from the target image using a pre-trained depth estimation network and generate a depth map based on the depth information;

[0178] A construction unit 903 is configured to construct a target 3D mesh using a preset construction technique, a target image, and a depth map;

[0179] A mapping optimization unit 904 is used to perform texture mapping on the target three-dimensional mesh and perform texture optimization on the texture-mapped target three-dimensional mesh using an optimization algorithm;

[0180] The hardware adaptation unit 905 is used to perform hardware adaptation and performance optimization on the target three-dimensional mesh after texture optimization.

[0181] Furthermore, the acquisition unit 901 includes:

[0182] Acquisition module, used to obtain the image to be processed;

[0183] A preprocessing module is used to preprocess the image to be processed to obtain the target image;

[0184] Preprocessing module, including:

[0185] The scaling submodule is used to scale the image to be processed to a preset resolution;

[0186] The conversion submodule is used to perform color space conversion on the image to be processed; wherein the color space conversion is to convert the format of the image to be processed into a preset color coding format;

[0187] The normalization submodule is used to perform numerical normalization on the image to be processed so as to map the pixel values ​​of the image to be processed to a preset interval to ensure that it complies with the input range of the neural network activation function;

[0188] The detection submodule is used to detect the main area of ​​the image to be processed and segment the background.

[0189] Furthermore, the prediction generation unit 902 includes:

[0190] The downsampling module is used to downsample the target image through the encoder of the convolutional neural network and change the size of the output feature map;

[0191] The upsampling module is used to upsample the target image through transposed convolution and feature map size change, and output a single-channel depth map;

[0192] A constraint module, configured to constrain the depth value of a single-channel depth map to a preset interval by outputting an activation function;

[0193] The load generation module is used to load the pre-trained model, input the constrained channel depth map into the pre-trained model, and generate a depth map.

[0194] Furthermore, the construction unit 903 includes:

[0195] A first optimization module is configured to optimize vertex positions, normal directions, and texture mapping of an initial three-dimensional mesh through differentiable rendering; wherein the initial three-dimensional mesh is a three-dimensional mesh that has not been subjected to rendering optimization;

[0196] The second optimization module is used to jointly optimize the initial three-dimensional mesh after rendering optimization by utilizing graphics processor acceleration and neural rendering function to generate a target three-dimensional mesh.

[0197] Furthermore, the mapping optimization unit 904 is specifically used to dynamically align the vertex color of the target three-dimensional mesh, the UV coordinates of the target three-dimensional mesh, the lighting of the input image of the target three-dimensional mesh, and the color information of the target three-dimensional mesh through differentiable rendering technology, so as to achieve texture optimization of the texture of the target three-dimensional mesh after texture mapping.

[0198] Furthermore, the hardware adaptation unit 905 is specifically used to perform hardware adaptation and performance optimization on the texture-optimized target three-dimensional mesh through a lightweight model, memory optimization technology, and dynamic quantization strategy.

[0199] The beneficial effects of the embodiments of the present application are as follows: no manual intervention is required to optimize the generation results. It is only necessary to use deep learning technology and a pre-trained depth estimation network to predict the depth information of the object from a single or multiple pictures, generate a depth map, combine the target picture and the depth map, reconstruct the target three-dimensional mesh through neural rendering technology, perform texture mapping on the generated target three-dimensional mesh, and improve the model details and rendering quality through optimization algorithms to quickly generate a high-quality three-dimensional model with texture self-adaptation and optimization. Through lightweight models and memory optimization technology, the texture self-adaptation and optimized three-dimensional model are hardware-adapted to adapt to different hardware configurations, thereby improving the efficiency of generating three-dimensional models and reducing the cost of generating three-dimensional models.

[0200] The beneficial effects of the embodiments of the present application are as follows: no manual intervention is required to optimize the generation results. It is only necessary to use deep learning technology and a pre-trained depth estimation network to predict the depth information of the object from a single or multiple pictures, generate a depth map, combine the target picture and the depth map, reconstruct the target three-dimensional mesh through neural rendering technology, perform texture mapping on the generated target three-dimensional mesh, and improve the model details and rendering quality through optimization algorithms to quickly generate a high-quality three-dimensional model with texture self-adaptation and optimization. Through lightweight models and memory optimization technology, the texture self-adaptation and optimized three-dimensional model are hardware-adapted to adapt to different hardware configurations, thereby improving the efficiency of generating three-dimensional models and reducing the cost of generating three-dimensional models.

[0201] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned artificial intelligence-based model processing method.

[0202] The present application also provides an electronic device, the structure of which is shown in FIG. Figure 10 As shown, it specifically includes a memory 1001 and one or more instructions 1002, wherein one or more instructions 1002 are stored in the memory 1001 and are configured to be executed by one or more processors 1003 to execute the one or more instructions 1002 to perform the above-mentioned artificial intelligence-based model processing method.

[0203] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0204] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For system-related embodiments, since they are generally similar to method-related embodiments, their description is relatively simple. For relevant details, refer to the description of the method-related embodiments.

[0205] The steps in the methods of the various embodiments of the present application can be adjusted in sequence, combined, and deleted according to actual needs.

[0206] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0207] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0208] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model processing method based on artificial intelligence, characterized in that: The method comprises: Obtain a target image; wherein the target image is an image that meets the input requirements of the adaptation model; Predicting the depth information of the object from the target image using a pre-trained depth estimation network, and generating a depth map based on the depth information; Constructing a target three-dimensional grid using a preset construction technology, the target image, and the depth map; Performing texture mapping on the target three-dimensional grid, and performing texture optimization on the texture-mapped target three-dimensional grid through an optimization algorithm; The target 3D mesh after texture optimization is hardware adapted and performance optimized.

2. The method according to claim 1, characterized in that The acquiring of the target image includes: Get the image to be processed; Preprocessing the image to be processed to obtain a target image; The process of preprocessing the image to be processed includes: Scaling the image to be processed to a preset resolution; Performing color space conversion on the image to be processed; wherein the color space conversion is to convert the format of the image to be processed into a preset color coding format; Normalizing the image to be processed so as to map the pixel values ​​of the image to be processed to a preset interval to ensure that the values ​​conform to the input range of the neural network activation function; Detect the main area of ​​the image to be processed and segment the background.

3. The method according to claim 1, characterized in that The method of predicting the depth information of the object from the target image by using a pre-trained depth estimation network and generating a depth map based on the depth information includes: Downsampling the target image through the encoder of the convolutional neural network, and outputting a feature map with a size change; Upsampling the target image through transposed convolution and resizing the feature map, outputting a single-channel depth map; Constraining the depth value of the single-channel depth map to a preset interval by outputting an activation function; The pre-trained model is loaded, and the constrained channel depth map is input into the pre-trained model to generate a depth map.

4. The method according to claim 1, wherein The constructing of the target three-dimensional grid by using the preset construction technology, the target image, and the depth map includes: Optimizing the vertex positions, normal directions, and texture mapping of the initial three-dimensional mesh by differentiable rendering; wherein the initial three-dimensional mesh is a three-dimensional mesh that has not been subjected to rendering optimization; The initial 3D mesh after rendering optimization is jointly optimized using graphics processor acceleration and neural rendering functions to generate a target 3D mesh.

5. The method according to claim 1, wherein The performing texture mapping on the target three-dimensional mesh and optimizing the texture of the texture-mapped target three-dimensional mesh by using an optimization algorithm comprises: Through differentiable rendering technology, the vertex color of the target three-dimensional mesh, the UV coordinates of the target three-dimensional mesh, the lighting of the input image of the target three-dimensional mesh and the color information of the target three-dimensional mesh are dynamically aligned to achieve texture optimization of the texture of the target three-dimensional mesh after texture mapping.

6. The method according to claim 1, characterized in that The hardware adaptation and performance optimization of the texture-optimized target three-dimensional mesh includes: Through lightweight models, memory optimization technology and dynamic quantization strategy, the target three-dimensional mesh after texture optimization is hardware adapted and performance optimized.

7. A model processing system based on artificial intelligence, characterized in that: The system comprises: An acquisition unit, configured to acquire a target image; wherein the target image is an image that meets the input requirements of the adaptation model; A prediction generation unit, configured to predict depth information of an object from the target image using a pre-trained depth estimation network, and generate a depth map based on the depth information; A construction unit, configured to construct a target three-dimensional grid using a preset construction technology, the target image, and the depth map; A mapping optimization unit, configured to perform texture mapping on the target three-dimensional mesh and perform texture optimization on the texture-mapped target three-dimensional mesh using an optimization algorithm; The adaptation and optimization unit is used to perform hardware adaptation and performance optimization on the target three-dimensional mesh after texture optimization.

8. The system according to claim 7, characterized in that The acquisition unit includes: Acquisition module, used to obtain the image to be processed; A preprocessing module, used to preprocess the image to be processed to obtain a target image; The preprocessing module includes: A scaling submodule, configured to scale the image to be processed to a preset resolution; A conversion submodule, configured to perform color space conversion on the image to be processed; wherein the color space conversion is to convert the format of the image to be processed into a preset color coding format; A normalization submodule, configured to perform numerical normalization on the image to be processed so as to map the pixel values ​​of the image to be processed to a preset interval to ensure that the values ​​conform to the input range of the neural network activation function; The detection and segmentation submodule is used to detect the main area of ​​the image to be processed and segment the background.

9. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the artificial intelligence-based model processing method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: It comprises a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to execute the artificial intelligence-based model processing method as described in any one of claims 1 to 6.

Images