Method and device for synthesizing new visual angle of product design image based on cooperation of large model and small model

Abstract

The invention discloses a large and small model collaborative product design image new view angle synthesis method and device, and the method can construct corresponding multi-dimensional sample sets for the main defects of new view angle image synthesis through a large and small model, and carries out the single-dimensional expert model training through each dimensional sample set. The trained expert model is matched with the basic model to solve the corresponding defects, and then weights are allocated to the first, second and third LORA models through a multi-expert gating mechanism based on a common sample set without tendency, so that more weights are allocated to the LORA model with higher difficulty; therefore, the method provided by the invention can be used for synthesizing the new view angle image with high efficiency and high quality.

Description

Technical Field

[0001] The present invention belongs to the field of product concept design, and specifically relates to a method and device for synthesizing new perspectives of product design images in collaboration with large and small models Background Art

[0002] In the field of product concept design, it is a task with important application significance to controllably obtain high-quality multi-view design images from simple inputs, which can help designers comprehensively analyze and consider the feasibility and deficiencies of current design concepts. In the traditional design process, designers use 3D modeling tools to model design concepts into concrete 3D models, thereby observing design solutions from multiple perspectives. Traditional 3D modeling methods have high requirements on the professional level of designers, and greatly prolong the cycle from conceptualization to prototype, which is not conducive to the rapid iteration of product concepts.

[0003] The development of large models enables artificial intelligence to generate multiple views or 3D models from text, which can help designers transform vague design ideas into high-quality images or 3D models in the early stages of design. However, existing 3D synthesis methods have shortcomings such as long synthesis time, poor 3D consistency, and difficulty in fine-tuning and editing the synthesis results, and there are still difficulties in their application in the field of product design.

[0004] Novel View Synthesis (NVS) studies how to synthesize other viewpoints of the same subject through one or a small number of input images. In the product design process, the application of this technology to synthesize multiple views from a single image can take into account both controllability and convenience. At present, there are two main technical routes in the field of NVS: methods based on implicit 3D representations such as Neural Radiance Fields (NeRF) can obtain three-dimensional information from sparse views and synthesize new views, but there are problems such as slow synthesis speed and poor effect under single view input; methods based on diffusion models use the two-dimensional prior of the Wensheng graph model as a guide to directly generate new views, but there are problems such as poor multi-view consistency and poor adaptability to out-of-distribution data. In addition, the current mainstream NVS models generally only provide full-scale training methods that require a large amount of data and training resources, and are trained through a fixed loss function, lacking fine-tuning methods for specific types of samples.

[0005] The mainstream idea in the current field of artificial intelligence is to use large and small models to collaboratively solve complex tasks. This strategy uses the general knowledge of large models and the domain-specific knowledge of small models to improve the generalization and accuracy of artificial intelligence in solving complex problems.

[0006] In terms of large models, multimodal large language models have demonstrated powerful capabilities in image recognition and positional relationship understanding, and can be applied to visual tasks such as combinatorial 3D generation. In terms of small models, the low-rank adaptation (LoRA) method can inject specific objects, styles or concepts into the generation process in a highly controllable manner in visual generation models such as text images and video generation.

[0007] In the process of new perspective synthesis, similar general knowledge and domain knowledge also exist, which is specifically manifested in the large model's cognition of the rationality of overall geometry and texture, and the small model's specialized knowledge of specific categories or specific detail features. However, the existing new perspective synthesis model generation is uncontrollable and lacks fine-tuning strategies.

[0008] Therefore, it is urgent to propose a multi-dimensional evaluation method for new-perspective image synthesis that collaborates with large and small models to improve the quality of new-perspective image synthesis. Summary of the invention

[0009] The present invention provides a new perspective synthesis method for product design images in collaboration with large and small models, which can improve the quality of new perspective image synthesis.

[0010] The present invention provides a new perspective synthesis method for product design images in collaboration with large and small models, comprising: A product design image from one perspective is used as a sample, and real color maps and normal maps from other perspectives are used as labels. A sample set is constructed using multiple samples, wherein the samples include single object samples and multi-object combination samples, and the obtained multi-perspective generation model is used as a basic model; The samples are input into the basic model, and a plurality of single object samples and a plurality of multi-object combination samples corresponding to when the similarity value between the synthetic view generated by the basic model and the label is lower than the similarity threshold are respectively used as the first dimension sample set and the second dimension sample set; The type, thickness and reality of the synthetic views generated by the basic model are comprehensively scored by the large model, and the corresponding multiple samples with scores lower than the set score are used as the third dimension sample set; Randomly select samples from the sample set to construct a common sample set, wherein the common sample set does not include samples from the first dimension sample set, the second dimension sample set, and the third dimension sample set; Construct three expert models, wherein the expert models are constructed by the basic model and the initial LORA, and the initial LORA models of the three expert models are respectively trained based on different loss functions through the first dimension sample set, the second dimension sample set and the third dimension sample set to obtain the first LORA model, the second LORA model and the third LORA model; Based on the common sample set, a new perspective synthesis model of product design images is obtained by assigning weights to the first, second and third LORA models through a multi-expert gating mechanism; When applied, a product design image from one perspective is input into the product design image new perspective synthesis model to obtain the predicted color map and normal map of the product design image from other perspectives.

[0011] Preferably, a new perspective synthesis model of product design images is obtained by assigning weights to the first, second and third LORA models through a multi-expert gating mechanism based on a common sample set, including: A training model is constructed, wherein the training model includes a frozen basic model, a first, second and third LORA models, and a trainable Router model. The Router model of the training model is trained by using a common sample set to obtain a final Router model. A new perspective synthesis model of product design images is obtained based on the basic model, the first, second and third LORA models, and the final Router model.

[0012] Preferably, the final Router model is obtained by training the Router model of the training model with a common sample set, including: The samples are input into the basic model, the first, second and third LORA models, and the Router model respectively. The Router model assigns weights to the output results of the first, second and third LORA models. The three output results with assigned weights are added together and then added to the output result of the basic model to obtain the prediction result. Construct a loss function based on the predicted results and the true color maps and normal maps of multiple views of the product design image; Based on the common sample set, the Router model is trained through the loss function to obtain the final Router model.

[0013] Preferably, the Router model includes linear mapping, activation function and Dropout; Among them, after the common samples are subjected to multiple linear mappings, the mapping results are sequentially passed through the corresponding activation functions and Dropout to obtain the weights assigned to the first, second and third LORA models.

[0014] Preferably, the method for obtaining the similarity value between the synthetic view generated by the base model and the label includes: The loss value between the synthetic view generated by the base model and the label is obtained through the multi-scale structural similarity loss function, and the obtained loss value is used as the similarity value between the synthetic view generated by the base model and the label.

[0015] Preferably, the multi-object combination sample includes a 3D model of a combined object, or is constructed by 3D editing and assembling a single object model.

[0016] Preferably, the type, thickness and reality of the synthetic views generated by the base model are comprehensively scored by the large model, including: Based on the four-dimensional evaluation criteria, a comprehensive scoring prompt is constructed, and the comprehensive scoring prompt, the synthetic view generated by the basic model and the corresponding label are input into the large model to obtain the comprehensive scoring score; The four-dimensional evaluation criteria include object type identifiability, geometric thickness accuracy, realism of the synthetic view, and overall similarity; The object type recognizability is to make a category judgment on the input synthetic view and label, and to make a similarity comparison on the judged categories; The geometric thickness accuracy is to determine the geometric thickness consistency of the synthetic view and the label; The authenticity of the synthetic view is to judge the consistency of structure and appearance of the synthetic view and the label; The overall similarity is the probability that the synthesized view and the label are different perspectives of the same object based on the object type identifiability, geometric thickness accuracy and the realism of the synthesized view.

[0017] Preferably, an expert model is constructed through a basic model and an initial LORA, including: The basic model is a Wonder3D model, which includes a multi-view self-attention layer, a cross-domain self-attention layer and a cross-attention layer. Initial LORA is injected into the multi-view self-attention layer, the cross-domain self-attention layer and the cross-attention layer to obtain an expert model.

[0018] Preferably, the loss function for training the LORA model is , is the loss weight for predicting the normal component of the noise, To predict the loss weight of the color component of the noise, take , when training the second LORA model, take , when training the third LORA model, , , ,in, is the normal component of the prediction noise output by the expert model, is the color component of the predicted noise output by the expert model, is the normal component of random noise that conforms to the standard Gaussian distribution, is the color component of random noise that conforms to the standard Gaussian distribution.

[0019] The present invention also provides a device for synthesizing new perspectives of product design images in collaboration with large and small models, characterized in that it includes: a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for synthesizing new perspectives of product design images in collaboration with large and small models.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The present invention can use large and small models to construct corresponding multi-dimensional sample sets for the main defects in the synthesis of new perspective images, and perform single-dimensional expert model training through each dimensional sample set, so that the trained expert model cooperates with the basic model to solve the corresponding defects, and then allocates weights to the first, second and third LORA models through a multi-expert gating mechanism based on a common sample set without tendency, so as to allocate more weights to the LORA model whose LORA dimension is more matched with the difficulty of sample multi-view generation, so that the method provided by the present invention can efficiently and high-quality synthesize new perspective images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for synthesizing a new perspective of a product design image in collaboration with large and small models provided by a specific embodiment of the present invention; Figure 2 A structural diagram of a training expert model provided for a specific embodiment of the present invention; Figure 3 A structural diagram of training a training model provided in a specific embodiment of the present invention; Figure 4 A structural diagram of a gated router provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to solve the problems in the prior art of uncontrollable generation of new perspective synthesis models, lack of fine-tuning strategies, and poor quality of synthesized new perspective images, a specific embodiment of the present invention provides a new perspective synthesis method for collaborative multi-dimensional evaluation of large and small models for product concept design. Starting from the needs of the product concept design field and the main defects of the current NVS model in this field, this method selects multiple dimensional sample sets and performs targeted optimization by training LoRA. The method provided by the specific embodiment of the present invention, such as Figure 1 As shown, it mainly includes three parts: S1, construction of multidimensional data sets for collaboration of large and small models; S2, training of single-dimensional optimization models for each dimension; S3, training of multi-dimensional optimization models based on multi-expert gating mechanism.

[0023] S1. A specific embodiment of the present invention constructs a multidimensional data set through collaborative large and small models: A specific embodiment of the present invention selects multiple evaluation dimensions required for the synthesis of new perspectives for product concept design, and constructs a multidimensional sample set by collaborative evaluation of large and small models. For each selected dimension, multi-view data is rendered from multiple open source large-scale 3D data sets, and samples are selected by collaborative scoring of large and small models. In the application of other product concept design propositions, different goals can also be adopted under the same technical framework, and appropriate data sets can be selected for targeted optimization.

[0024] A specific embodiment of the present invention uses a product design image from one perspective as a sample, and real color maps and normal maps from other perspectives as labels. Multiple samples construct a sample set, where the samples include single object samples and multi-object combination samples, and the obtained multi-perspective generation model is used as the basic model.

[0025] In a specific embodiment, the training data of the specific embodiment of the present invention is obtained from the Google Scanned Object and OmniObject3D datasets, totaling 1461 3D models. , Elevation The main viewing direction is randomly and uniformly sampled within the range. Then, the blenderproc tool is used to render it, and the color and normal maps of the six standard viewing angles are obtained.

[0026] The specific embodiment of the present invention selects the following three dimensions as the evaluation angles of the NVS model for product concept design: (1) Global image quality: Image quality is one of the main indicators of the generation capability of the NVS model. The image quality is evaluated based on the loss obtained by comparing the synthetic views generated by the base model with the real rendered images, i.e., labels. That is, the overall quality of multi-view generation is characterized by evaluating the similarity between the generated images and the labels, including the geometric and texture quality, and the degree of refinement.

[0027] In the dimension of global image quality, a specific embodiment of the present invention selects samples that the basic model is not good at processing and generates with poor quality. From the aforementioned large-scale data set, a multi-scale structural similarity index measure (MS-SSIM) loss function is used to select samples whose similarity between the results generated by the basic model and the real images is low as the first dimension sample set.

[0028] Specifically, a specific embodiment of the present invention inputs samples into a basic model, and uses multiple single object samples corresponding to a synthetic view generated by the basic model when the similarity value with the label is lower than a similarity threshold as a first dimension sample set. The first dimension sample set is used to train the corresponding expert model to improve the global image quality. This embodiment uses samples with a large gap in similarity with the label as the first dimension sample set, which can avoid local optimality and improve robustness during the training process.

[0029] Specifically, the method for obtaining the similarity value between the synthetic view generated by the basic model and the label includes: The loss value between the synthetic view generated by the base model and the label is obtained through the multi-scale structural similarity loss function, and the obtained loss value is used as the similarity value between the synthetic view generated by the base model and the label.

[0030] (2) Combination and multi-view Figure 1 Multi-view Consistency: The consistency of product multi-views is highly correlated with the retention of design concepts in new views. In the NVS model based on the diffusion model, since the new perspective image needs to be as similar as possible to the same input image, there may be problems such as poor 3D consistency between the synthesized views, such as multi-face. At the same time, after testing, the current NVS model may have some subjects missing or disordered positional relationships in the generated image when processing combined image inputs due to the difficulty in accurately recognizing the positional relationship between multiple subjects. The expert model trained with this dimensional sample set has the ability to maintain geometric consistency such as the number, shape, and positional relationship of subjects in the synthesized new view when facing complex shapes or combined multi-subject inputs.

[0031] Specifically, in a specific embodiment of the present invention, samples are input into a basic model, and multiple multi-object combination samples corresponding to when the synthetic view generated by the basic model and the label similarity value are lower than the similarity threshold are used as a second dimension sample set, and the second dimension sample set is used to train the corresponding expert model to improve the combination and multi-view Figure 1 Consistency.

[0032] In the specific embodiment of the present invention, in the dimensions of combination and multi-view consistency, due to the lack of specialized data sets and evaluation indicators in relevant dimensions, a batch of data is constructed by downloading 3D models containing combined objects, assembling single models in 3D editing software, etc., and samples with subject omissions, geometric inconsistencies, and other phenomena that are difficult for the pre-trained basic model to deal with are selected as the data set of this dimension. Among the above three sub-dimensional data sets, the data format is the real color and normal map of six standard perspectives, which is consistent with the format output by the pre-trained basic model.

[0033] (3) Thickness and common sense matching: In the generation of multi-views of products, it is important to obtain multi-view images that are consistent with the designer's cognition and conform to design common sense. However, due to the limitation of the diffusion model, the NVS model is prone to generate images whose thickness and shape are inconsistent with the facts. This is particularly obvious when the reference view does not provide enough information or when facing the front view input of uncommon and diverse-shaped subjects. The sample set of this dimension is obtained by evaluating the realism of the model-generated images through the general knowledge of the large model on the features of common objects.

[0034] A specific embodiment of the present invention uses a large model to comprehensively score the type, thickness, and reality of the synthetic views generated by the basic model, and uses the corresponding multiple samples with scores lower than the set score as a third dimension sample set.

[0035] Specifically, the specific embodiment of the present invention uses the large model to comprehensively score the type, thickness and reality of the synthetic view generated by the basic model, including: Based on the four-dimensional evaluation criteria, a comprehensive scoring prompt is constructed, and the comprehensive scoring prompt, the synthetic view generated by the basic model and the corresponding label are input into the large model to obtain the comprehensive scoring score; The four-dimensional evaluation criteria include object type identifiability, geometric thickness accuracy, realism of the synthetic view, and overall similarity.

[0036] The object type recognizability provided by the specific embodiment of the present invention is to perform category judgment on the input synthetic view and label respectively, and perform similarity comparison on the judged categories.

[0037] The geometric thickness accuracy provided by the specific embodiment of the present invention is to determine the geometric thickness consistency of the synthetic view and the label.

[0038] The realism of the synthetic view provided by the specific embodiment of the present invention is to judge the consistency of the structure and appearance of the synthetic view and the label.

[0039] The overall similarity provided by the specific embodiment of the present invention is the probability that the synthesized view and the label are different perspectives of the same object based on the identifiability of the object type, the accuracy of the geometric thickness and the realism of the synthesized view.

[0040] Based on the above standards, the large model can find a sample set in which the synthetic view output by the basic model is significantly different from the label in thickness, sensory authenticity and thickness consistency. The expert model is trained with this sample set so that the corresponding expert model can obtain synthetic views that match the thickness, type and structural authenticity under different viewing angles.

[0041] In a specific embodiment, the large language model prompt words provided in this embodiment are as follows: Your goal is to determine whether two images are from different perspectives of the same object. You need to make the judgment based on the following criteria: 1. Object type: The two pictures belong to the same type of object. Please first guess what kind of object they belong to based on the two pictures, and then compare the similarity of the two guesses.

[0042] 2. Geometric thickness: The two images are geometrically consistent, including size and thickness, and consistent with the real object. Please note that the images are taken at different angles, so after rotating in space, pay special attention to the consistency of their thickness.

[0043] 3. Reality: Both images are real and match the structure of real objects. Please analyze the two images separately and give a conclusion. The conclusion can refer to the object type you judged before.

[0044] 4. Overall similarity: Considering the above criteria, these two images belong to different perspectives of the same object.

[0045] Before providing your answer, please review both images carefully and focus on one aspect at a time. Try to judge each criterion independently. To provide your answer, please briefly analyze each of the above evaluation criteria. The analysis should be concise and accurate. For each criterion, you need to make a judgment using the following three options: 1 - The criterion is fully met / 2 - The criterion is partially met / 3 - The criterion is not met at all.

[0046] Important: These images are highly similar. Please evaluate the images carefully and choose option 1 with caution. If you can distinguish the non-compliant parts through subtle details, please choose option 3.

[0047] Summarize your final decision on the last line in the format: "<options for criterion 1><options for criterion 2><options for criterion 3><options for criterion 4>".

[0048] 1. Object type: xxxx on the left; xxxx on the right; they completely / partially / not at all do not meet the standards.

[0049] 2. Geometric thickness: xxxx on the left; xxxx on the right; they completely / partially / completely do not meet the standards.

[0050] 3. Degree of authenticity: xxxx on the left; xxxx on the right; they completely / partially / completely do not meet the standards.

[0051] 4. Overall similarity: xxxx; they completely / partially / not at all meet the criteria.

[0052] 5. Final answer: xxxx (for example: 1 2 1 1 / 3 3 1 3 / 2 1 3 2).

[0053] (4) In addition, a batch of non-dimensional data is randomly selected from the samples that are not in the above-mentioned dimensional data sets as a common sample set, which is used for training and testing the gated router module in the multi-dimensional optimization model. Since the common sample set is not divided into the above-mentioned three dimensions, a larger weight is assigned to the more difficult LORA model during the training process.

[0054] Specifically, a specific embodiment of the present invention randomly selects samples from a sample set to construct a common sample set, and the common sample set does not include samples of a first dimension sample set, a second dimension sample set, and a third dimension sample set.

[0055] In a specific embodiment, finally, the proportion of each type of samples in the dataset constructed by this embodiment is shown in Table 1. The format of each sample is a color image and a normal image rendered at 6 fixed viewing angles (front, left, right, back, left front, right front) of the same 3D model. Table 1 Sample distribution of each dimension dataset

[0056]

[0057] S2. The specific embodiment of the present invention performs single-dimensional expert model training based on each dimension sample set constructed: Figure 2 As shown, the present invention builds an expert model with a plug-and-play LoRA module on the basic model, configures different loss functions to achieve controllable model fine-tuning, and for each of the above-mentioned evaluation dimensions, the specific embodiment of the present invention uses the corresponding dimension sample set as input to train the LoRA module of the corresponding dimension, thereby realizing a single-dimensional optimized NVS model, that is, an expert model. Expert models of different dimensions can solve the corresponding problem of the dimension, thereby obtaining the corresponding synthetic view.

[0058] Specifically, the multi-view generation model of the specific embodiment of the present invention is modified based on the open source Wonder3D model. On the basis of the pre-trained Wonder3D model, the multi-view self-attention layer, cross-domain attention layer and cross-attention layer of the original model are edited, and the QKV matrix of each attention layer is injected with LoRA, and the forward formula is updated as follows: ,In the training stage, since it already has a certain generation ability in the early stage, the two-stage generation strategy of the ,basic model is no longer adopted, but the LoRA of the multi-view self-attention module and ,the cross-domain attention module is trained simultaneously.

[0059] The loss function part provided in the specific embodiment of the present invention assumes is the added random noise that conforms to the standard Gaussian distribution, where represents the color part noise, Represents the normal part of the noise. The diffusion model predicts the noise Split by channel to get a still including color and normal components .in is the target diffusion model, for The combined graph embedding vector at time step, also including color and normal components, is the input reference view, represents the fixed camera parameter embedding under K target views. As normal loss, As color loss.

[0060] In one specific embodiment, the loss function for training the LORA model is ,in, is the loss weight for predicting the normal component of the noise, is the loss weight for predicting the color component of the noise.

[0061] The global image quality dimension provided in the specific embodiment of the present invention mainly measures the global quality of the generated color and normal maps. The two are equally important in the evaluation process, so they are given the same weight when calculating the loss. When training the first LORA model, , that is, the loss formula is , after training, the first LORA model is obtained.

[0062] The thickness and common sense matching dimensions provided in the specific embodiment of the present invention mainly evaluate the matching degree between the synthetic view and the real object, while the color loss refers to the difference between the RGB image finally synthesized by the NVS model and the real image, which is more in line with human judgment of image differences and contains more semantic information. Therefore, color loss is used as the main loss. In one embodiment, the loss formula is , after training, the third LORA model is obtained.

[0063] In the combination and multi-view consistency dimensions, the geometric consistency between multiple views is mainly evaluated. Therefore, the normal loss that represents the geometric quality is used as the main loss. When training the second LoRa model, In one embodiment, the loss formula is , and the second LORA model is obtained after training.

[0064] S3. The specific embodiment of the present invention trains the training model constructed by the trained LORA model based on the common sample set: The specific embodiment of the present invention adopts a hybrid expert mechanism to combine multiple single LoRA models to achieve the evaluation and optimization of the NVS model in multiple parallel dimensions. By training a gating module, the input samples are parsed, so as to assign weights to different single-dimensional LoRAs, and realize a multi-dimensional optimized NVS model.

[0065] In a specific embodiment, based on the common sample set, weights are assigned to the first, second and third LORA models through a multi-expert gating mechanism to obtain a new perspective synthesis model of the product design image, such as Figure 3 As shown, including: A training model is constructed, wherein the training model includes a frozen basic model, a first, second and third LORA models, and a trainable Router model. The Router model of the training model is trained by using a common sample set to obtain a final Router model. A new perspective synthesis model of product design images is obtained based on the basic model, the first, second and third LORA models, and the final Router model.

[0066] Specifically, the Router model of the training model is trained by using the common sample set to obtain the final Router model, including: The samples are input into the basic model, the first, second and third LORA models, and the Router model respectively. The Router model assigns weights to the output results of the first, second and third LORA models. The three output results with assigned weights are added together and then added to the output result of the basic model to obtain the prediction result. Construct a loss function based on the predicted results and the true color maps and normal maps of multiple views of the product design image; Based on the common sample set, the Router model is trained through the loss function to obtain the final Router model.

[0067] The specific embodiment of the present invention implements a weight distribution gating module as a Router structure. Figure 4 As shown in the figure, the Router structure uses four linear mappings and corresponding activation and Dropout operations, and then maps the input image vector to three LoRA weight vectors through Linear and Softmax. After training, the Router can predict the difficulty of the input image in different dimensions, thereby assigning higher weights to more difficult LoRA.

[0068] Assume that the weight matrix of the basic model in the forward process is , thickness and common sense matching, global image quality, combination and multi-view consistency respectively introduce three LoRA low-rank matrices to correct the corresponding dimensions of the basic model, and their weight matrices are . Let the input vector encoding the front reference view, camera parameters, etc. be , Gated Router The weight distribution vector obtained based on the input vector is: , then the forward formula of each layer in the multi-dimensional optimization module is changed to ,in For the matrix connection operation, a randomly selected common data set is used for training to enhance the performance of the model on common samples. The loss function reuses the loss of the single-dimensional LoRA and uses the same weight for the color and normal map. .

[0069] The present invention also provides a device for synthesizing new perspectives of product design images in collaboration with large and small models, characterized in that it includes: a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for synthesizing new perspectives of product design images in collaboration with large and small models.

Claims

1. A new perspective synthesis method for product design images with large and small models, characterized in that: include: A product design image from one perspective is used as a sample, and real color maps and normal maps from other perspectives are used as labels. A sample set is constructed using multiple samples, wherein the samples include single object samples and multi-object combination samples, and the obtained multi-perspective generation model is used as a basic model; The samples are input into the basic model, and a plurality of single object samples and a plurality of multi-object combination samples corresponding to when the similarity value between the synthetic view generated by the basic model and the label is lower than the similarity threshold are respectively used as the first dimension sample set and the second dimension sample set; The type, thickness and reality of the synthetic views generated by the basic model are comprehensively scored by the large model, and the corresponding multiple samples with scores lower than the set score are used as the third dimension sample set; Randomly select samples from the sample set to construct a common sample set, wherein the common sample set does not include samples from the first dimension sample set, the second dimension sample set, and the third dimension sample set; Construct three expert models, wherein the expert models are constructed by the basic model and the initial LORA, and the initial LORA models of the three expert models are respectively trained based on different loss functions through the first dimension sample set, the second dimension sample set and the third dimension sample set to obtain the first LORA model, the second LORA model and the third LORA model; Based on the common sample set, a new perspective synthesis model of product design images is obtained by assigning weights to the first, second and third LORA models through a multi-expert gating mechanism; When applied, a product design image from one perspective is input into the product design image new perspective synthesis model to obtain the predicted color map and normal map of the product design image from other perspectives.

2. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 1 is characterized in that: Based on the common sample set, a multi-expert gating mechanism is used to assign weights to the first, second, and third LORA models to obtain a new perspective synthesis model for product design images, including: A training model is constructed, wherein the training model includes a frozen basic model, a first, second and third LORA models, and a trainable Router model. The Router model of the training model is trained by using a common sample set to obtain a final Router model. A new perspective synthesis model of product design images is obtained based on the basic model, the first, second and third LORA models, and the final Router model.

3. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 2 is characterized in that: The final Router model is obtained by training the Router model of the training model with a common sample set, including: The samples are input into the basic model, the first, second and third LORA models, and the Router model respectively. The Router model assigns weights to the output results of the first, second and third LORA models. The three output results with assigned weights are added together and then added to the output result of the basic model to obtain the prediction result. Construct a loss function based on the predicted results and the true color maps and normal maps of multiple views of the product design image; Based on the common sample set, the Router model is trained through the loss function to obtain the final Router model.

4. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 3 is characterized in that: The Router model includes linear mapping, activation function and Dropout; Among them, after the common samples are subjected to multiple linear mappings, the mapping results are sequentially passed through the corresponding activation functions and Dropout to obtain the weights assigned to the first, second and third LORA models.

5. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 1 is characterized in that: The method for obtaining the similarity value between the synthetic view generated by the basic model and the label includes: The loss value between the synthetic view generated by the base model and the label is obtained through the multi-scale structural similarity loss function, and the obtained loss value is used as the similarity value between the synthetic view generated by the base model and the label.

6. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 1 is characterized in that: The multi-object combination sample includes a 3D model of a combined object, or is constructed by 3D editing and assembling a single object model.

7. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 1 is characterized in that: The synthetic views generated by the large model are scored for type, thickness, and realism, including: Based on the four-dimensional evaluation criteria, a comprehensive scoring prompt is constructed, and the comprehensive scoring prompt, the synthetic view generated by the basic model and the corresponding label are input into the large model to obtain the comprehensive scoring score; The four-dimensional evaluation criteria include object type identifiability, geometric thickness accuracy, realism of the synthetic view, and overall similarity; The object type recognizability is to make a category judgment on the input synthetic view and label, and to make a similarity comparison on the judged categories; The geometric thickness accuracy is to determine the geometric thickness consistency of the synthetic view and the label; The authenticity of the synthetic view is to judge the consistency of structure and appearance of the synthetic view and the label; The overall similarity is the probability that the synthesized view and the label are different perspectives of the same object based on the object type identifiability, geometric thickness accuracy and the realism of the synthesized view.

8. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 1 is characterized in that: Build an expert model through the basic model and initial LORA, including: The basic model is a Wonder3D model, which includes a multi-view self-attention layer, a cross-domain self-attention layer and a cross-attention layer. Initial LORA is injected into the multi-view self-attention layer, the cross-domain self-attention layer and the cross-attention layer to obtain an expert model.

9. The new perspective synthesis method for product design images based on large and small model collaboration according to claim 1 or 8, characterized in that: Loss function for training the LORA model , is the loss weight for predicting the normal component of the noise, To predict the loss weight of the color component of the noise, take , when training the second LORA model, take , when training the third LORA model, , , ,in, is the normal component of the prediction noise output by the expert model, is the color component of the predicted noise output by the expert model, is the normal component of random noise that conforms to the standard Gaussian distribution, is the color component of random noise that conforms to the standard Gaussian distribution.

10. A new perspective synthesis device for product design images with large and small models, characterized in that: include: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the new perspective synthesis method for product design images in collaboration with large and small models as described in any one of claims 1-9.

Images