A retrieval-enhanced 3D-GAN small model incremental training and optimization method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- RENMIN UNIVERSITY OF CHINA
- Filing Date
- 2025-11-18
- Publication Date
- 2026-05-12
Smart Images
Figure CN121580796B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D generative adversarial technology, specifically involving a method for incremental training and optimization of a small 3D-GAN model based on retrieval enhancement. Background Technology
[0002] 3D Generative Adversarial Networks (3D-GANs) are a cutting-edge field in computer vision and computer graphics, demonstrating enormous application potential in scenarios such as digital entertainment, industrial design, and autonomous driving simulation. However, current 3D-GAN technology is deeply mired in a dilemma of balancing efficiency and quality in practice, severely restricting its industrial application, especially for small and medium-sized enterprises with limited budgets.
[0003] First, existing technologies face fundamental bottlenecks in model representation methods and rendering efficiency. Mainstream 3D representation methods, such as voxel grids and neural implicit representations, each have significant drawbacks. Voxel grid representations rely on discrete 3D meshes; as resolution increases to pursue finer generation quality, the required memory overhead grows exponentially, making it impossible to train and deploy models on conventional hardware. To circumvent the memory bottleneck, neural implicit representation methods (such as NeRF) can achieve continuous, high-quality representations, but their rendering process requires point-by-point querying and evaluation in space, resulting in enormous computational demands and extremely slow rendering speeds. They typically struggle to reach the real-time interactive standard of 5 frames per second (<5fps), failing to meet the performance requirements of real-time generation and virtual reality applications.
[0004] Secondly, in terms of hardware deployment and ecosystem adaptation, current solutions are costly and pose supply chain risks. The vast majority of advanced 3D-GAN models are developed and optimized for foreign GPUs (Graphics Processing Units), lacking underlying hardware adaptation and optimization for domestic chips such as Ascend. This leads to two major challenges for users during deployment: firstly, additional and costly porting and adaptation work is required; secondly, the long-term reliance on foreign GPU hardware solutions, with their high procurement and maintenance costs, drives up the overall deployment cost, resulting in a high barrier to entry for the technology.
[0005] Furthermore, existing technological approaches fail to effectively resolve the core contradiction between model lightweighting and accuracy. To reduce deployment costs, the industry has attempted to adopt miniaturized models with fewer than 1 billion parameters (i.e., small models). However, due to their limited capacity, these small models inherently have shortcomings in detailed feature extraction and reconstruction capabilities. The accuracy, richness, and fidelity of the generated models are significantly lower than those of large models with hundreds of billions or even trillions of parameters. This sharp contradiction between "low cost" and "high accuracy" makes existing small model solutions unable to meet the core needs of SMEs for technology that is both easy to use and affordable.
[0006] In summary, existing 3D-GAN technology urgently needs an innovative solution to simultaneously overcome memory and computational efficiency bottlenecks, achieve efficient compatibility with domestic computing infrastructure, and fundamentally improve the generation quality of small-parameter models, thereby breaking the "efficiency-quality" paradox that has long plagued industrial development. Summary of the Invention
[0007] The purpose of this invention is to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for incremental training and optimization of small 3D-GAN models based on retrieval enhancement, in order to solve the problem that the existing 3D-GAN technology is seriously restricted from industrial application due to the dilemma of balancing "efficiency" and "quality" in practice.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] Firstly, a method for incremental training and optimization of small 3D-GAN models based on retrieval enhancement is provided, including:
[0010] Construct and maintain a 3D sample library, wherein the 3D sample library contains various categories of 3D sample models generated by a 3D-GAN large model with a total number of parameters of not less than 1 billion or designed manually, as well as the high-dimensional feature representation of the 3D sample models.
[0011] Upon receiving a 3D model generation task, based on the task content and generation objective carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library as a reference sample.
[0012] The high-dimensional feature representation of the reference sample is transferred to a small 3D-GAN model with a total number of parameters of less than 1 billion by knowledge distillation technique, so as to enhance the ability of the small 3D-GAN model to extract and reconstruct detailed features.
[0013] During the knowledge distillation process, an adaptive loss adjustment strategy is introduced to dynamically monitor the incremental training process of the 3D-GAN small model and assess the risk of overfitting. Then, the weights of the loss function are automatically adjusted according to the risk assessment results to balance the fidelity of the generated details and the diversity of the generated results.
[0014] After completing incremental training, the optimized 3D-GAN small model is hardware adapted for the target chip to generate the final 3D-GAN small model for lightweight deployment on the target chip.
[0015] Based on the above-mentioned invention, a novel incremental training and optimization scheme for 3D-GAN small models that can simultaneously balance efficiency and quality is provided. Upon receiving a 3D model generation task, at least one 3D sample model similar to the generation task in geometric structure and / or texture features is retrieved in real-time from a 3D sample library as a reference sample. Then, the high-dimensional feature representation of the reference sample is transferred to a 3D-GAN small model with a total number of parameters of less than 1 billion using knowledge distillation technology. Simultaneously, an adaptive loss adjustment strategy is introduced during the transfer process to dynamically monitor the incremental training process of the small model and assess the risk of overfitting. The weights of the loss function are then automatically adjusted based on the risk assessment results. After completing the incremental training, the optimized small model undergoes hardware adaptation optimization for the target chip, generating a final 3D-GAN small model for lightweight deployment on the target chip. By introducing the "retrieval-enhanced incremental training" framework and "adaptive loss adjustment" core technologies, this scheme effectively solves the "efficiency-quality" paradox and high deployment costs currently faced by 3D-GAN technology, achieving significant technical effects and positive socio-economic benefits, facilitating practical application and promotion.
[0016] In one possible design, based on the task content and generation objective carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library, including:
[0017] Extract the task semantic description and / or low-dimensional feature vectors used as initial inputs to the 3D model generation task as task content and generation target;
[0018] For each 3D sample model in the 3D sample library, the similarity between the task semantic description and / or the low-dimensional feature vector and the corresponding high-dimensional feature representation is calculated.
[0019] Based on a preset similarity threshold and / or a similarity-based Top-K sorting strategy, at most K 3D sample models are selected from the 3D sample library as at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features, where K represents a positive integer.
[0020] In one possible design, based on the task content and generation objective carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library, including:
[0021] For each 3D sample model in the 3D sample library, the corresponding high-dimensional feature representation is dimensionality reduced to obtain the corresponding first vector, and the first vector is added to the vector library.
[0022] Extract the task semantic description and / or low-dimensional feature vectors used as initial inputs to the 3D model generation task as task content and generation target;
[0023] The task semantic description and / or the low-dimensional feature vector are transformed into a second vector, and the ANN retrieval algorithm is used to retrieve the M third vectors most similar to the second vector from the vector library, where M represents a positive integer;
[0024] M 3D sample models corresponding one-to-one with the M third vectors are selected from the 3D sample library as at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features.
[0025] In one possible design, the high-dimensional feature representation of the reference sample is transferred to a small 3D-GAN model with a total number of parameters of less than 1 billion using knowledge distillation techniques, including:
[0026] Construct a distillation loss function that uses the high-dimensional feature representation of the reference sample as the "teacher signal" and the non-input layer features of a small 3D-GAN model with a total number of parameters of less than 1 billion as the "student signal". The distillation loss function includes a content loss for constraining the visual fidelity of the generative model, a style loss for constraining the style features, and a structural loss for constraining the topological structure of the 3D model. The non-input layer features include intermediate layer features and / or output layer features.
[0027] By minimizing the distillation loss function, the 3D-GAN small model is forced to learn and approximate the detailed features of the reference sample.
[0028] In one possible design, the adaptive loss adjustment strategy includes:
[0029] During incremental training, the diversity index of the generated results is calculated periodically, wherein the diversity index includes the difference between generated samples and / or the repetition rate between generated samples and training dataset;
[0030] Based on multiple preset threshold values corresponding to the diversity indicators, if it is detected that the value of any of the diversity indicators is lower than the corresponding preset threshold value, it is determined that there is a risk of overfitting.
[0031] When an overfitting risk is detected, the weight of the distillation loss function is automatically reduced, and the weights of the adversarial loss function and / or reconstruction loss function are increased accordingly to encourage the 3D-GAN small model to generate more diverse outputs.
[0032] In one possible design, the optimized 3D-GAN small model undergoes hardware adaptation optimization for the target chip, including:
[0033] The optimized computational graph structure of the 3D-GAN small model is converted into an intermediate representation format supported by the target chip;
[0034] By utilizing the dedicated operator library provided by the target chip, computationally intensive operators in the optimized 3D-GAN small model are replaced and tuned based on the intermediate representation format to improve the inference efficiency of the final 3D-GAN small model on the target chip.
[0035] Secondly, a retrieval-enhanced 3D-GAN small model incremental training and optimization device is provided, which includes a 3D sample library operation and maintenance unit, a reference sample retrieval unit, a knowledge distillation and transfer unit, an automatic loss adjustment unit, and a hardware adaptation and optimization unit that are connected in sequence.
[0036] The 3D sample library operation and maintenance unit is used to build and maintain a 3D sample library, wherein the 3D sample library contains various types of 3D sample models generated by a 3D-GAN large model with a total number of parameters of not less than 1 billion or designed manually, as well as the high-dimensional feature representation of the 3D sample model.
[0037] The reference sample retrieval unit is used to, upon receiving a 3D model generation task, retrieve in real time from the 3D sample library at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features as a reference sample, based on the task content and generation target carried in the 3D model generation task.
[0038] The knowledge distillation transfer unit is used to transfer the high-dimensional feature representation of the reference sample to a small 3D-GAN model with a total number of parameters of less than 1 billion through knowledge distillation technology, so as to enhance the ability of the small 3D-GAN model to extract and reconstruct detailed features.
[0039] The automatic loss adjustment unit is used to introduce an adaptive loss adjustment strategy during the transfer process through the knowledge distillation technique to dynamically monitor the incremental training process of the 3D-GAN small model and assess the risk of overfitting. Then, it automatically adjusts the weight of the loss function according to the risk assessment result to balance the fidelity of the generated details and the diversity of the generated results.
[0040] The hardware adaptation and optimization unit is used to perform hardware adaptation and optimization of the optimized 3D-GAN small model for the target chip after incremental training is completed, so as to generate the final 3D-GAN small model for lightweight deployment on the target chip.
[0041] Thirdly, the present invention provides a computer device comprising a storage module, a processing module, and a transceiver module connected in sequence for communication, wherein the storage module is used to store a computer program, the transceiver module is used to send and receive messages, and the processing module is used to read the computer program and execute the 3D-GAN small model incremental training and optimization method as described in the first aspect or any possible design in the first aspect.
[0042] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the 3D-GAN small model incremental training and optimization method as described in the first aspect or any possible design in the first aspect.
[0043] Fifthly, the present invention provides a computer program product, including a computer program or instructions, wherein the computer program or instructions, when executed by a computer, implement the 3D-GAN small model incremental training and optimization method as described in the first aspect or any possible design in the first aspect.
[0044] The beneficial effects of the above scheme are:
[0045] (1) This invention creatively provides a novel 3D-GAN small model incremental training and optimization scheme that can simultaneously take into account efficiency and quality. That is, after receiving the 3D model generation task, at least one 3D sample model that is similar to the generation task in terms of geometric structure and / or texture features is retrieved from the 3D sample library in real time as a reference sample. Then, the high-dimensional feature representation of the reference sample is transferred to a 3D-GAN small model with a total number of parameters of less than 1 billion through knowledge distillation technology. At the same time, an adaptive loss adjustment strategy is introduced during the transfer process to dynamically monitor the incremental training process of the small model and assess the risk of overfitting. Then, the weight of the loss function is automatically adjusted according to the risk assessment results. After the incremental training is completed, the optimized small model is hardware adapted for the target chip to generate the final 3D-GAN small model for lightweight deployment on the target chip. Thus, by introducing the "retrieval-enhanced incremental training" framework and "adaptive loss adjustment" and other core technologies, the "efficiency-quality" paradox and the problem of high deployment cost faced by the current 3D-GAN technology are effectively solved, and significant technical effects and positive social and economic benefits are achieved in many aspects.
[0046] (2) It can significantly improve the quality of small model generation and break through the contradiction between "low cost" and "high precision". Through the innovative "retrieval-enhanced incremental training" framework, this solution can provide highly relevant high-quality samples as guidance for each generation task in real time. It also uses knowledge distillation technology to transfer the rich detail features (such as fine texture and complex topology) contained in large models or professional samples as "teacher signals" to small models. This allows small models with a total number of parameters of less than 1 billion to maintain their lightweight nature while significantly improving the accuracy, detail richness and fidelity of their generated models, reaching about 90% of the level of excellent large models. This fundamentally solves the technical pain point that small and medium-sized enterprises cannot obtain high-precision 3D generation capabilities due to limited budgets.
[0047] (3) It can significantly reduce the cost of model training and iteration. That is, this solution does not require training a huge model from scratch, and also avoids the waste of resources of repeated training of small models on low-quality data. Through efficient sample retrieval and targeted knowledge transfer, the training process can be quickly converged. Practice shows that this method can reduce the training cost of the model by about 60%, which not only directly reduces the technical application threshold for small and medium-sized enterprises, but also means a significant reduction in computing power consumption under the same computing task, which helps to reduce carbon emissions and is in line with the development concept of green computing.
[0048] (4) It can effectively ensure the diversity of the generated model and avoid mode collapse and overfitting. That is, by introducing the "adaptive loss adjustment strategy", the training process can be dynamically monitored. When the risk of overfitting such as excessive repetition rate of generated samples is detected, the weight of the loss function is automatically adjusted. This strategy cleverly balances the contradiction between "detail restoration" and "generation diversity". It ensures that the small model can learn the essence from the reference sample and encourages it to generate innovative models, thereby producing high-quality and diverse 3D models, which greatly improves the practicality and robustness of the model.
[0049] (5) It can achieve efficient hardware adaptation and promote the independent control and popularization of technology. That is, this solution clearly includes hardware adaptation optimization steps for the optimized small model at the end of the process. In particular, for the domestic Ascend chip, through model conversion and operator tuning, it realizes efficient inference and lightweight deployment on the domestic computing power platform. This effect has dual strategic significance: First, it reduces the enterprise's dependence on foreign GPUs and reduces high deployment costs and supply chain risks; Second, it strongly supports the ecological landing and application expansion of domestic AI chips in cutting-edge fields such as 3D content generation.
[0050] (6) It can broaden the application scenarios of technology and stimulate industrial innovation. This will enable the high-performance 3D content generation technology to be popularized and directly support and promote the development of multiple cutting-edge fields, such as digital twins and industrial real-time simulation, autonomous driving simulation, cultural and creative industries and metaverse and industrial quality inspection. It is expected to help thousands of small and medium-sized entities enter the field of 3D content production every year, significantly reduce the threshold for technology application, stimulate the innovation vitality of the whole society, and facilitate practical application and promotion. Attached Figure Description
[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is a flowchart illustrating the incremental training and optimization method for a small 3D-GAN model based on retrieval enhancement provided in this application embodiment.
[0053] Figure 2 This is a schematic diagram of the structure of the retrieval-enhanced 3D-GAN small model incremental training and optimization device provided in the embodiments of this application.
[0054] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.
[0056] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.
[0057] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.
[0058] Example
[0059] like Figure 1 As shown, the retrieval-enhanced 3D-GAN small model incremental training and optimization method provided in the first aspect of this embodiment can be executed, but is not limited to, by computer devices with certain computing resources, such as servers, personal computers (PCs, referring to a type of multi-purpose computer suitable for personal use in terms of size, price, and performance; desktops, laptops, mini-laptops, tablets, and ultrabooks are all considered personal computers), smartphones, personal digital assistants (PDAs), or wearable devices. Figure 1 As shown, the incremental training and optimization method for the 3D-GAN small model includes, but is not limited to, the following steps S1 to S5.
[0060] S1. Construct and maintain a 3D sample library, wherein the 3D sample library includes, but is not limited to, various categories of 3D sample models generated by a 3D-GAN large model with a total number of parameters of not less than 1 billion or designed manually, as well as the high-dimensional feature representation of the 3D sample models.
[0061] In step S1, the various categories include, but are not limited to, industrial parts, cultural and creative models, and 3D models of abnormal scenarios used for semi-physical simulation testing of autonomous driving. The high-dimensional feature representation is the "essential archive" or "digital DNA" of the 3D sample model. It is not a specific image or model, but a high-dimensional vector to express the most essential and abstract features of the 3D sample model, which can be conventionally obtained through an encoder in a U-shaped network architecture.
[0062] S2. Upon receiving a 3D model generation task, based on the task content and generation objective carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved from the 3D sample library in real time as a reference sample.
[0063] In step S2, the 3D model generation task can be obtained by the user through regular editing and uploading. The retrieved at least one 3D sample model is used to form the sample set for subsequent incremental training. Specifically, based on the task content and generation target carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library, including but not limited to the following steps S211 to S213.
[0064] S211. Extract the task semantic description and / or low-dimensional feature vectors used as initial inputs to the 3D model generation task as task content and generation target.
[0065] S212. For each 3D sample model in the 3D sample library, calculate the similarity between the task semantic description and / or the low-dimensional feature vector and the corresponding high-dimensional feature representation.
[0066] In step S212, although the task semantic description and / or the low-dimensional feature vector and the high-dimensional feature representation have inconsistent dimensions, they can be projected into a unified and dimensionless common semantic space, and then their similarity can be calculated in this space.
[0067] S213. Based on a preset similarity threshold and / or a similarity-based Top-K sorting strategy, select up to K 3D sample models from the 3D sample library as at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features, where K represents a positive integer.
[0068] In step S2, in order to quickly retrieve the at least one 3D sample model, preferably, based on the task content and generation target carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library, including but not limited to the following steps S221 to S224.
[0069] S221. For each 3D sample model in the 3D sample library, perform dimensionality reduction processing on the corresponding high-dimensional feature representation to obtain the corresponding first vector, and add the first vector to the vector library.
[0070] S222. Extract the task semantic description and / or low-dimensional feature vectors used as initial inputs to the 3D model generation task as task content and generation target.
[0071] S223. Transform the task semantic description and / or the low-dimensional feature vector into a second vector, and use the ANN retrieval algorithm to retrieve the M third vectors most similar to the second vector from the vector library, where M represents a positive integer.
[0072] S224. Select M 3D sample models from the 3D sample library that correspond one-to-one with the M third vectors as at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features.
[0073] S3. The high-dimensional feature representation of the reference sample is transferred to a small 3D-GAN model with a total number of parameters of less than 1 billion through knowledge distillation technology, so as to enhance the ability of the small 3D-GAN model to extract and reconstruct detailed features.
[0074] In step S3, the knowledge distillation technique is a model compression and knowledge transfer technique. Its core idea is to use a large, complex, but high-performance "teacher model" to guide a lightweight and compact "student model" in learning, thereby allowing the student model to achieve performance close to that of the teacher model while maintaining a small size. Specifically, the high-dimensional feature representation of the reference sample is transferred to a small 3D-GAN model with less than 1 billion parameters through knowledge distillation, including but not limited to the following steps S31-S32.
[0075] S31. Construct a distillation loss function that uses the high-dimensional feature representation of the reference sample as the "teacher signal" and the non-input layer features of a small 3D-GAN model with a total number of parameters of less than 1 billion as the "student signal". The distillation loss function includes, but is not limited to, content loss for constraining the visual fidelity of the generative model, style loss for constraining style features, and structural loss for constraining the topological structure of the 3D model. The non-input layer features include, but are not limited to, intermediate layer features and / or output layer features.
[0076] S32. By minimizing the distillation loss function, the 3D-GAN small model is forced to learn and approximate the detailed features of the reference sample.
[0077] S4. During the knowledge distillation process, an adaptive loss adjustment strategy is introduced to dynamically monitor the incremental training process of the 3D-GAN small model and assess the risk of overfitting. Then, the weights of the loss function are automatically adjusted according to the risk assessment results to balance the fidelity of the generated details with the diversity of the generated results.
[0078] In step S4, specifically, the adaptive loss adjustment strategy includes, but is not limited to, the following steps S41 to S43.
[0079] S41. During incremental training, the diversity index of the generated results is calculated periodically, wherein the diversity index includes, but is not limited to, the degree of difference between generated samples and / or the repetition rate between generated samples and training dataset.
[0080] S42. Based on multiple preset index thresholds that correspond one-to-one with the diversity indexes, if it is detected that the index value of any of the diversity indexes is lower than the corresponding preset index threshold, it is determined that there is a risk of overfitting.
[0081] S43. When it is determined that there is a risk of overfitting, the weight of the distillation loss function is automatically reduced, and the weight of the adversarial loss function and / or reconstruction loss function is increased accordingly to encourage the 3D-GAN small model to generate more diverse outputs.
[0082] S5. After completing incremental training, the optimized 3D-GAN small model is hardware adapted for the target chip to generate the final 3D-GAN small model for lightweight deployment on the target chip.
[0083] In step S5, the target chip is specifically, but not limited to, an Ascend chip. Specifically, the optimized 3D-GAN small model undergoes hardware adaptation optimization for the target chip, including but not limited to the following steps S51 to S52.
[0084] S51. Convert the optimized computation graph structure of the 3D-GAN small model into an intermediate representation format supported by the target chip.
[0085] S52. Using the dedicated operator library provided by the target chip, the computationally intensive operators in the optimized 3D-GAN small model are replaced and tuned based on the intermediate representation format to improve the inference efficiency of the final 3D-GAN small model on the target chip.
[0086] Therefore, based on the 3D-GAN small model incremental training and optimization method described in steps S1 to S5 above, a novel 3D-GAN small model incremental training and optimization scheme that can simultaneously balance efficiency and quality is provided. Specifically, upon receiving a 3D model generation task, at least one 3D sample model similar to the generation task in geometric structure and / or texture features is retrieved in real-time from the 3D sample library as a reference sample. Then, the high-dimensional feature representation of the reference sample is transferred to a 3D-GAN small model with a total number of parameters of less than 1 billion using knowledge distillation technology. Simultaneously, an adaptive loss adjustment strategy is introduced during the transfer process to dynamically monitor the incremental training process of the small model and assess the risk of overfitting. The weights of the loss function are then automatically adjusted based on the risk assessment results. After completing the incremental training, the optimized small model undergoes hardware adaptation optimization for the target chip, generating a final 3D-GAN small model for lightweight deployment on the target chip. Thus, by introducing the "retrieval-enhanced incremental training" framework and "adaptive loss adjustment" and other core technologies, the current 3D-GAN model problem is effectively solved. Despite the challenges of the "efficiency-quality" paradox and high deployment costs faced by the technology, significant technological effects and positive socio-economic benefits have been achieved, facilitating practical application and promotion.
[0087] like Figure 2 As shown, the second aspect of this embodiment provides a virtual device for implementing the 3D-GAN small model incremental training and optimization method described in the first aspect, including a 3D sample library operation and maintenance unit, a reference sample retrieval unit, a knowledge distillation and transfer unit, an automatic loss adjustment unit, and a hardware adaptation and optimization unit that are sequentially connected in communication.
[0088] The 3D sample library operation and maintenance unit is used to build and maintain a 3D sample library, wherein the 3D sample library contains various types of 3D sample models generated by a 3D-GAN large model with a total number of parameters of not less than 1 billion or designed manually, as well as the high-dimensional feature representation of the 3D sample model.
[0089] The reference sample retrieval unit is used to, upon receiving a 3D model generation task, retrieve in real time from the 3D sample library at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features as a reference sample, based on the task content and generation target carried in the 3D model generation task.
[0090] The knowledge distillation transfer unit is used to transfer the high-dimensional feature representation of the reference sample to a small 3D-GAN model with a total number of parameters of less than 1 billion through knowledge distillation technology, so as to enhance the ability of the small 3D-GAN model to extract and reconstruct detailed features.
[0091] The automatic loss adjustment unit is used to introduce an adaptive loss adjustment strategy during the transfer process through the knowledge distillation technique to dynamically monitor the incremental training process of the 3D-GAN small model and assess the risk of overfitting. Then, it automatically adjusts the weight of the loss function according to the risk assessment result to balance the fidelity of the generated details and the diversity of the generated results.
[0092] The hardware adaptation and optimization unit is used to perform hardware adaptation and optimization of the optimized 3D-GAN small model for the target chip after incremental training is completed, so as to generate the final 3D-GAN small model for lightweight deployment on the target chip.
[0093] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be found in the 3D-GAN small model incremental training and optimization method described in the first aspect, and will not be repeated here.
[0094] like Figure 3As shown, the third aspect of this embodiment provides a computer device for executing the 3D-GAN small model incremental training and optimization method as described in the first aspect. The device includes a storage module, a processing module, and a transceiver module connected in sequence. The storage module stores a computer program, the transceiver module sends and receives messages, and the processing module reads the computer program and executes the 3D-GAN small model incremental training and optimization method as described in the first aspect. Specifically, the storage module may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processing module may, but is not limited to, use a microprocessor of the STM32F105 series. Furthermore, the computer device may also include, but is not limited to, a power supply module, a display screen, and other necessary components.
[0095] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the 3D-GAN small model incremental training and optimization method described in the first aspect, and will not be repeated here.
[0096] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions comprising the 3D-GAN small model incremental training and optimization method as described in the first aspect. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the 3D-GAN small model incremental training and optimization method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
[0097] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be found in the 3D-GAN small model incremental training and optimization method described in the first aspect, and will not be repeated here.
[0098] This fifth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the 3D-GAN small model incremental training and optimization method as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
[0099] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for incremental training and optimization of a small 3D-GAN model based on retrieval enhancement, characterized in that, include: Construct and maintain a 3D sample library, wherein the 3D sample library contains various categories of 3D sample models generated by a 3D-GAN large model with a total number of parameters of not less than 1 billion or designed manually, as well as the high-dimensional feature representation of the 3D sample models. Upon receiving a 3D model generation task, based on the task content and generation objective carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library as a reference sample. The high-dimensional feature representation of the reference sample is transferred to a small 3D-GAN model with a total number of parameters of less than 1 billion by knowledge distillation technique, so as to enhance the ability of the small 3D-GAN model to extract and reconstruct detailed features. During the knowledge distillation transfer process, an adaptive loss adjustment strategy is introduced to dynamically monitor the incremental training process of the 3D-GAN small model and assess the risk of overfitting. Then, based on the risk assessment results, the weights of the loss function are automatically adjusted to balance the fidelity of generated details with the diversity of generated results. The adaptive loss adjustment strategy includes: periodically calculating a diversity index of generated results during incremental training, where the diversity index includes the difference between generated samples and / or the repetition rate between generated samples and the training dataset; based on multiple preset threshold values corresponding to the diversity index, if any index value is found to be lower than the corresponding preset threshold, an overfitting risk is identified; when an overfitting risk is identified, the weight of the distillation loss function is automatically reduced, and the weights of the adversarial loss function and / or reconstruction loss function are correspondingly increased to encourage the 3D-GAN small model to generate more diverse outputs. After completing incremental training, the optimized 3D-GAN small model is hardware adapted for the target chip to generate the final 3D-GAN small model for lightweight deployment on the target chip.
2. The 3D-GAN small model incremental training and optimization method according to claim 1, characterized in that, Based on the task content and generation objective carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library, including: Extract the task semantic description and / or low-dimensional feature vectors used as initial inputs to the 3D model generation task as task content and generation target; For each 3D sample model in the 3D sample library, the similarity between the task semantic description and / or the low-dimensional feature vector and the corresponding high-dimensional feature representation is calculated. Based on a preset similarity threshold and / or a similarity-based Top-K sorting strategy, at most K 3D sample models are selected from the 3D sample library as at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features, where K represents a positive integer.
3. The incremental training and optimization method for small 3D-GAN models according to claim 1, characterized in that, Based on the task content and generation objective carried in the 3D model generation task, at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features is retrieved in real time from the 3D sample library, including: For each 3D sample model in the 3D sample library, the corresponding high-dimensional feature representation is dimensionality reduced to obtain the corresponding first vector, and the first vector is added to the vector library. Extract the task semantic description and / or low-dimensional feature vectors used as initial inputs to the 3D model generation task as task content and generation target; The task semantic description and / or the low-dimensional feature vector are transformed into a second vector, and the ANN retrieval algorithm is used to retrieve the M third vectors most similar to the second vector from the vector library, where M represents a positive integer; M 3D sample models corresponding one-to-one with the M third vectors are selected from the 3D sample library as at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features.
4. The 3D-GAN small model incremental training and optimization method according to claim 1, characterized in that, The high-dimensional feature representation of the reference samples is transferred to a small 3D-GAN model with less than 1 billion parameters using knowledge distillation techniques, including: Construct a distillation loss function that uses the high-dimensional feature representation of the reference sample as the "teacher signal" and the non-input layer features of a small 3D-GAN model with a total number of parameters of less than 1 billion as the "student signal". The distillation loss function includes a content loss for constraining the visual fidelity of the generative model, a style loss for constraining the style features, and a structural loss for constraining the topological structure of the 3D model. The non-input layer features include intermediate layer features and / or output layer features. By minimizing the distillation loss function, the 3D-GAN small model is forced to learn and approximate the detailed features of the reference sample.
5. The 3D-GAN small model incremental training and optimization method according to claim 1, characterized in that, The optimized 3D-GAN small model undergoes hardware adaptation optimization for the target chip, including: The optimized computational graph structure of the 3D-GAN small model is converted into an intermediate representation format supported by the target chip; By utilizing the dedicated operator library provided by the target chip, computationally intensive operators in the optimized 3D-GAN small model are replaced and tuned based on the intermediate representation format to improve the inference efficiency of the final 3D-GAN small model on the target chip.
6. A retrieval-enhanced 3D-GAN small model incremental training and optimization device, characterized in that, It includes a 3D sample library operation and maintenance unit, a reference sample retrieval unit, a knowledge distillation and transfer unit, an automatic loss adjustment unit, and a hardware adaptation and optimization unit, all connected in sequence. The 3D sample library operation and maintenance unit is used to build and maintain a 3D sample library, wherein the 3D sample library contains various types of 3D sample models generated by a 3D-GAN large model with a total number of parameters of not less than 1 billion or designed manually, as well as the high-dimensional feature representation of the 3D sample model. The reference sample retrieval unit is used to, upon receiving a 3D model generation task, retrieve in real time from the 3D sample library at least one 3D sample model that is similar to the 3D model generation task in terms of geometric structure and / or texture features as a reference sample, based on the task content and generation target carried in the 3D model generation task. The knowledge distillation transfer unit is used to transfer the high-dimensional feature representation of the reference sample to a small 3D-GAN model with a total number of parameters of less than 1 billion through knowledge distillation technology, so as to enhance the ability of the small 3D-GAN model to extract and reconstruct detailed features. The automatic loss adjustment unit is used to introduce an adaptive loss adjustment strategy during the knowledge distillation transfer process to dynamically monitor the incremental training process of the 3D-GAN small model and assess the risk of overfitting. Then, based on the risk assessment results, it automatically adjusts the weights of the loss function to balance the fidelity of generated details with the diversity of generated results. The adaptive loss adjustment strategy includes: periodically calculating a diversity index of generated results during incremental training, where the diversity index includes the difference between generated samples and / or the repetition rate between generated samples and the training dataset; based on multiple preset index thresholds corresponding to the diversity indexes, if the index value of any of the diversity indexes is detected to be lower than the corresponding preset index threshold, an overfitting risk is determined; when an overfitting risk is determined, the weight of the distillation loss function is automatically reduced, and the weights of the adversarial loss function and / or the reconstruction loss function are correspondingly increased to encourage the 3D-GAN small model to generate more diverse outputs. The hardware adaptation and optimization unit is used to perform hardware adaptation and optimization of the optimized 3D-GAN small model for the target chip after incremental training is completed, so as to generate the final 3D-GAN small model for lightweight deployment on the target chip.
7. A computer device, characterized in that, It includes a storage module, a processing module, and a transceiver module that are sequentially connected in communication. The storage module is used to store computer programs, the transceiver module is used to send and receive messages, and the processing module is used to read the computer programs and execute the 3D-GAN small model incremental training and optimization method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores instructions that, when executed on a computer, perform the 3D-GAN small model incremental training and optimization method as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the 3D-GAN small model incremental training and optimization method as described in any one of claims 1 to 5.