Facility three-dimensional model generation method and device, equipment and medium

Abstract

The invention provides a facility three-dimensional model generation method and device, equipment and a medium, and the method comprises the steps: carrying out the visual feature extraction of a plurality of images, obtaining a visual style feature vector, carrying out the text feature extraction of information text data, and obtaining a text feature vector; inputting the visual style feature vector and the text feature vector into a cross-modal feature fusion model for cross-modal feature fusion to obtain a cross-modal feature vector; inputting the cross-modal feature vector into a diffusion model to obtain an initial three-dimensional model of the target facility; and finally, inputting the initial three-dimensional model and the cross-modal feature vector into a texture generative adversarial network to obtain a target three-dimensional model of the target facility. Wherein a facility texture map of the target facility is arranged on the surface of the target three-dimensional model. Through the method and the device, the detail richness and the visual reality sense of the generated facility three-dimensional model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional model generation, and in particular to a method, device, equipment and medium for generating a three-dimensional model of a facility. Background Art

[0002] Spatial layout and scene generation are often important parts of architectural space design. Improving the aesthetics of the space by rationally designing the color matching of facilities is an important part of creating atmosphere and enhancing beauty.

[0003] At present, most architectural space layout generation methods focus on layout rationality, but users often do not have detailed and realistic facility models to choose from, and cannot intuitively understand the overall design style of the space through facility models. Therefore, how to generate realistic facility 3D models has become a technical issue that cannot be underestimated. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a method, device, equipment and medium for generating a three-dimensional model of a facility, which integrates the feature information of text description and multi-view images to generate an accurate semantic feature representation of the facility, and then generates and optimizes the input semantic feature vector through a diffusion model to generate a three-dimensional shape model of the facility that meets the description. Finally, a texture generative adversarial network is used to achieve high-fidelity reproduction of the facility texture, thereby improving the detail richness and visual realism of the generated three-dimensional model of the facility.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a three-dimensional model of a facility, the generating method comprising: Acquire images of an original model corresponding to a target facility at multiple viewing angles, and information text data corresponding to the target facility; Extracting visual features from the plurality of images to obtain a visual style feature vector, and extracting text features from the information text data to obtain a text feature vector; Inputting the visual style feature vector and the text feature vector into a pre-trained cross-modal feature fusion model for cross-modal feature fusion to obtain a cross-modal feature vector; Inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility; The initial three-dimensional model and the cross-modal feature vector are input into a pre-trained texture generative adversarial network to obtain a target three-dimensional model of the target facility; wherein the surface of the target three-dimensional model is provided with a facility texture map of the target facility.

[0006] Furthermore, extracting visual features from the plurality of images to obtain a visual style feature vector, and extracting text features from the information text data to obtain a text feature vector, comprises: For each image, convert the image into a plurality of image tag sequences, and input the plurality of image tag sequences into an image encoder to obtain an image embedding vector; Inputting the image embedding vector into the ViT model for encoding to obtain image features, and performing average pooling processing on the image features to obtain an image feature vector of the image; Performing vector concatenation on the image feature vectors of the plurality of images to obtain a target feature vector, and performing normalization processing on the target feature vector to obtain the visual style feature vector; Converting the information text data into a plurality of text tag sequences, and inputting the plurality of text tag sequences into a text encoder for embedding to obtain a text embedding vector; Inputting the text embedding vector into the Transformer model for encoding to obtain text features, and performing average pooling processing on the text features to obtain an initial feature vector of the information text data; The initial feature vector is normalized to obtain a text feature vector of the information text data.

[0007] Furthermore, the step of inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility includes: The cross-modal feature vector and the first random noise vector are used as first input data, and the first input data is input into the diffusion model to obtain a diffusion result outputted by each diffusion stage in the diffusion model; wherein the diffusion stage includes a low-resolution stage, a medium-resolution stage, and a high-resolution stage; The diffusion results outputted in each diffusion stage are fused to obtain the initial three-dimensional model.

[0008] Furthermore, the inputting the initial three-dimensional model and the cross-modal feature vector into a pre-trained texture generative adversarial network to obtain a target three-dimensional model of the target facility includes: Using the cross-modal feature vector and the second random noise vector as second input data, and inputting the second input data into a generator of the texture generative adversarial network to obtain the facility texture map; The target three-dimensional model is generated using the facility texture map and the initial three-dimensional model.

[0009] Furthermore, the texture generation adversarial network is trained by the following steps: Initialize the parameters of the generator and the parameters of the discriminator in the texture generation adversarial network; Inputting first sample input data into the generator to obtain a first generated texture map, and inputting the first generated texture map and the real texture map into the discriminator, calculating the gradient penalty loss value of the discriminator, and updating the parameters of the discriminator using the gradient descent method based on the gradient penalty loss value; Inputting the second sample input data into the generator to obtain a second generated texture map, inputting the second generated texture map into the discriminator, calculating the loss value of the generator based on the discrimination result of the discriminator, and updating the parameters of the generator using the gradient descent method based on the loss value; The generator and the discriminator are repeatedly trained alternately until the generator and the discriminator converge, thereby obtaining the texture generative adversarial network.

[0010] In a second aspect, an embodiment of the present application further provides a device for generating a three-dimensional model of a facility, the device comprising: An acquisition module, used to acquire images of the original model corresponding to the target facility at multiple viewing angles, and information text data corresponding to the target facility; A feature vector extraction module, used for extracting visual features from the plurality of images to obtain visual style feature vectors, and extracting text features from the information text data to obtain text feature vectors; A feature fusion module, used for inputting the visual style feature vector and the text feature vector into a pre-trained cross-modal feature fusion model for cross-modal feature fusion to obtain a cross-modal feature vector; An initial three-dimensional model generation module, used for inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility; A target three-dimensional model generation module is used to input the initial three-dimensional model and the cross-modal feature vector into a pre-trained texture generation adversarial network to obtain a target three-dimensional model of the target facility; wherein the surface of the target three-dimensional model is provided with a facility texture map of the target facility.

[0011] Furthermore, when the feature vector extraction module is used to extract visual features from the plurality of images to obtain visual style feature vectors and to extract text features from the information text data to obtain text feature vectors, the feature vector extraction module is also used to: For each image, convert the image into a plurality of image tag sequences, and input the plurality of image tag sequences into an image encoder to obtain an image embedding vector; Inputting the image embedding vector into the ViT model for encoding to obtain image features, and performing average pooling processing on the image features to obtain an image feature vector of the image; Performing vector concatenation on the image feature vectors of the plurality of images to obtain a target feature vector, and performing normalization processing on the target feature vector to obtain the visual style feature vector; Converting the information text data into a plurality of text tag sequences, and inputting the plurality of text tag sequences into a text encoder for embedding to obtain a text embedding vector; Inputting the text embedding vector into the Transformer model for encoding to obtain text features, and performing average pooling processing on the text features to obtain an initial feature vector of the information text data; The initial feature vector is normalized to obtain a text feature vector of the information text data.

[0012] Furthermore, when the initial three-dimensional model generation module is used to input the cross-modal feature vector into a pre-trained diffusion model to obtain the initial three-dimensional model of the target facility, the initial three-dimensional model generation module is also used to: The cross-modal feature vector and the first random noise vector are used as first input data, and the first input data is input into the diffusion model to obtain a diffusion result outputted by each diffusion stage in the diffusion model; wherein the diffusion stage includes a low-resolution stage, a medium-resolution stage, and a high-resolution stage; The diffusion results outputted in each diffusion stage are fused to obtain the initial three-dimensional model.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for generating a three-dimensional model of a facility as described above are performed.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating a three-dimensional model of a facility as described above are executed.

[0015] The embodiments of the present application provide a method, device, equipment and medium for generating a three-dimensional model of a facility. First, images of an original model corresponding to a target facility at multiple viewing angles and information text data corresponding to the target facility are obtained; then, visual features are extracted from the multiple images to obtain a visual style feature vector, and text features are extracted from the information text data to obtain a text feature vector; the visual style feature vector and the text feature vector are input into a pre-trained cross-modal feature fusion model for cross-modal feature fusion to obtain a cross-modal feature vector; the cross-modal feature vector is input into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility; finally, the initial three-dimensional model and the cross-modal feature vector are input into a pre-trained texture generative adversarial network to obtain a target three-dimensional model of the target facility; wherein a facility texture map of the target facility is provided on the surface of the target three-dimensional model.

[0016] This application uses the natural language description of the facility as input, combined with cross-modal feature extraction and multi-level generation strategies, to provide high-quality facility 3D model generation. First, the feature information of the text description and multi-view images is integrated to generate an accurate semantic feature representation of the facility. Then, the input semantic feature vector is subjected to shape generation and optimization through a diffusion model to generate a 3D shape model of the facility that meets the description. Finally, a texture generative adversarial network is used to achieve high-fidelity reproduction of the facility texture, which improves the detail richness and visual realism of the generated 3D facility model.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a method for generating a three-dimensional model of a facility provided in an embodiment of the present application; Figure 2 One of the structural schematic diagrams of a device for generating a three-dimensional model of a facility provided in an embodiment of the present application; Figure 3 A second structural schematic diagram of a device for generating a three-dimensional model of a facility provided in an embodiment of the present application; Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0021] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied to the field of three-dimensional model generation technology.

[0022] Spatial layout and scene generation are often important parts of architectural space design. Improving the aesthetics of the space by rationally designing the color matching of facilities is an important part of creating atmosphere and enhancing beauty.

[0023] Research has found that, at present, most architectural space layout generation methods mainly focus on layout rationality, and users often do not have detailed and rich facility models to choose from, and cannot intuitively understand the overall design style of the space through facility models. Therefore, how to generate a realistic 3D facility model has become a technical issue that cannot be underestimated.

[0024] Based on this, the embodiments of the present application provide a method, device, equipment and medium for generating a three-dimensional model of a facility, which improves the detail richness and visual realism of the generated three-dimensional model of the facility.

[0025] See also Figure 1 , Figure 1 This is a flow chart of a method for generating a three-dimensional model of a facility provided in an embodiment of the present application. Figure 1 As shown in, the generation method provided in the embodiment of the present application includes: S101, obtaining images of an original model corresponding to a target facility under multiple viewing angles, and information text data corresponding to the target facility.

[0026] It should be noted that the target facility refers to the facility for which the target three-dimensional model is currently being generated. For example, when the spatial layout design of a home building is performed, the target facility may be furniture in the home building, such as a sofa, a chair, etc., which is not specifically limited in this application. The original model refers to a pre-constructed three-dimensional model of the target facility. Multiple perspectives may include a front perspective, a left perspective, a right perspective, a top perspective, a back perspective, and a bottom perspective. The information text data may include the IFC standard product information of the target facility and descriptive information of the facility style that the user currently wants to generate.

[0027] In the above step S101, in the specific implementation, for the target facility for which the 3D model is currently being generated, the original model corresponding to the target facility is obtained. I , and obtain images of the original model under multiple viewing angles. Here, continuing the above embodiment, when the multiple viewing angles include the front viewing angle, the left viewing angle, the right viewing angle, the top viewing angle, the back viewing angle and the bottom viewing angle, the original model I There are 6 images, represented as I 1, I 2,..., I 6. At the same time, obtain the information text data entered by the user for the target facility T .

[0028] S102, extracting visual features from the plurality of images to obtain a visual style feature vector, and extracting text features from the information text data to obtain a text feature vector.

[0029] In the above step S102, in the specific implementation, feature extraction is performed on the acquired data, and visual style feature vectors are extracted from multiple images and information text data of the original model. and text feature vector .

[0030] As an optional embodiment, with respect to the above step S102, extracting visual features from the plurality of images to obtain a visual style feature vector, and extracting text features from the information text data to obtain a text feature vector, comprises: Step 1021 , for each image, convert the image into a plurality of image tag sequences, and input the plurality of image tag sequences into an image encoder to obtain an image embedding vector.

[0031] Regarding the above step 1021, in the specific implementation, for each image, the image I x Converted into multiple image tag sequences { , ,…, }, and then input multiple image tag sequences into the image encoder to obtain an image embedding vector. Here, according to the embodiment provided by the present application, the image encoder adopts a pre-trained CLIP model architecture. The CLIP (Contrastive Language-Image Pre-training) model is a multimodal model developed by OpenAI that can understand and associate images and text. The CLIP model is pre-trained with large-scale image-text data, enabling it to perform well in a variety of visual tasks without fine-tuning for specific tasks.

[0032] Specifically, images I x The image embedding vector It is expressed by the following formula:

[0033] in, m The number of image tag sequences representing this image.

[0034] Step 1022: Input the image embedding vector into the ViT model for encoding to obtain image features, and perform average pooling processing on the image features to obtain an image feature vector of the image.

[0035] Regarding the above step 1022, in the specific implementation, firstly embed the image into the vector Input into the ViT model for encoding to obtain multiple image features , specifically, Here, the ViT (Vision Transformer) model is a deep learning model based on the Transformer architecture. The ViT model converts image data into sequence data and uses a multi-layer self-attention mechanism and a feedforward neural network to capture the global relationship in the image. Then, the image features are Perform average pooling to obtain the image feature vector of the image .

[0036] Specifically, the image feature vector Calculated by the following formula:

[0037] Step 1023, performing vector concatenation on the image feature vectors of the plurality of images to obtain a target feature vector, and performing normalization processing on the target feature vector to obtain the visual style feature vector.

[0038] For the above step 1023, in the specific implementation, the image feature vector of each image After obtaining, the image feature vectors of multiple images Perform vector concatenation to obtain the target feature vector. Then, perform normalization on the target feature vector to obtain the visual style feature vector.

[0039] Specifically, continuing the above example, when the original model I There are 6 images: I 1, I 2,..., I 6, the target feature vector Calculated by the following formula:

[0040] Furthermore, the visual style feature vector Calculated by the following formula:

[0041] Step 1024, converting the information text data into a plurality of text tag sequences, and inputting the plurality of text tag sequences into a text encoder for embedding to obtain a text embedding vector.

[0042] Regarding the above step 1024, in the specific implementation, the information text data T Converted into multiple text token sequences { , ,…, }, and input multiple text token sequences into the text encoder for embedding to obtain the text embedding vector Here, the text encoder also uses the pre-trained CLIP model architecture. Specifically, the text embedding vector It is expressed by the following formula:

[0043] in, n Represents information text data T The number of text token sequences in .

[0044] Step 1025, input the text embedding vector into the Transformer model for encoding to obtain text features, and perform average pooling processing on the text features to obtain the initial feature vector of the information text data.

[0045] Regarding the above step 1025, in the specific implementation, firstly embed the text into the vector Input into the Transformer model for encoding to obtain text features , specifically, Here, the Transformer model also uses a multi-layer self-attention mechanism and a feedforward neural network to encode the text tag sequence. Then, the text features Perform average pooling to obtain the initial feature vector of the information text data T .

[0046] Specifically, the text feature vector Calculated by the following formula:

[0047] Step 1026: normalize the initial feature vector to obtain a text feature vector of the information text data.

[0048] Regarding the above step 1026, in the specific implementation, the initial feature vector Perform normalization to obtain the text feature vector Specifically, the text feature vector Calculated by the following formula:

[0049] S103: Input the visual style feature vector and the text feature vector into a pre-trained cross-modal feature fusion model to perform cross-modal feature fusion to obtain a cross-modal feature vector.

[0050] Regarding the above step S103, in the specific implementation, the visual style feature vector and text feature vector Input into the pre-trained cross-modal feature fusion model for cross-modal feature fusion, and obtain a unified cross-modal feature vector through the cross-modal feature fusion model .

[0051] Here, a cross-modal attention mechanism is used for feature fusion. The attention mechanism has three main elements: query, key, and value. The text feature vector and the visual style feature vector are used as query, key, and value, and the fused feature representation is calculated through cross-attention. The mathematical expression is shown in the following formula:

[0052]

[0053] in, is the query vector, which is used to represent the content of interest; is the key matrix, used to represent the input content; Represents a value matrix, used to represent the content to be obtained; are the dimensions of the query and key vectors; The function is used to obtain the weight of each visual feature in the fusion result and then multiply it by Value to generate the final fusion feature in a weighted summation manner. , , is the learnable projection matrix, is the feature dimension.

[0054] Furthermore, the cross-modal feature vector Calculated by the following formula:

[0055] in, is the fusion weight, is the cross-modal feature vector.

[0056] Here, by designing a cross-modal contrast loss function , ensuring that the multimodal semantic vector of the cross-modal feature fusion model can accurately capture the semantic association between text and visual information. Specifically, the cross-modal contrast loss function It is expressed by the following formula:

[0057] in, is the cosine similarity function, is an adjustable temperature parameter used to adjust the sensitivity of similarity.

[0058] S104, inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility.

[0059] Regarding the above step S104, in the specific implementation, the cross-modal feature vector Input into the pre-trained diffusion model to obtain the initial 3D model of the target facility.

[0060] As an optional embodiment, with respect to the above step S104, the step of inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility includes: Step 1041: taking the cross-modal feature vector and the first random noise vector as first input data, inputting the first input data into the diffusion model, and obtaining a diffusion result outputted at each diffusion stage in the diffusion model.

[0061] Regarding the above step 1041, in the specific implementation, first define a random noise vector , the cross-modal feature vector With the first random noise vector as the first input data . Then the first input data The data is input into the diffusion model to obtain a diffusion result outputted in each diffusion stage of the diffusion model for the first input data.

[0062] Here, the diffusion process in the diffusion model defines the transformation process from data distribution to Gaussian noise. Specifically, the forward diffusion process is defined as a series of steps to add noise. The conditional guidance mechanism is introduced to strengthen the influence of multimodal feature vectors on the generated results. At each time step In the generation process, the conditional features guide the network Control and execute the forward diffusion process. The forward diffusion process gradually adds noise to the data, so that the original data gradually transforms into a noise distribution. Reverse diffusion is the process of returning from the noise distribution to the data distribution. The forward diffusion process is expressed by the following formula:

[0063] in, is the random noise value added, is the diffusion coefficient, which controls the degree of noise added in each step.

[0064] Specifically, according to the embodiments provided by the present application, a multi-level resolution strategy is introduced in different stages of the reverse diffusion process, and the diffusion process in the diffusion model is generally divided into three stages: a low-resolution stage, a medium-resolution stage, and a high-resolution stage. Each stage has its specific goals and processing methods.

[0065] The low-resolution stage mainly processes the basic structure and contour of the 3D model. Specifically, the diffusion result of the low-resolution stage It is expressed by the following formula:

[0066] In the mid-resolution stage, we start to add details and textures to the 3D model. The goal of this stage is to gradually increase the resolution of the 3D model while maintaining the basic structure and outlines generated in the low-resolution stage. The 3D model will introduce more details in this stage, making the 3D model look more realistic and concrete. Specifically, the diffusion result of the mid-resolution stage It is expressed by the following formula:

[0067] In the high-resolution stage, the 3D model is further refined, adding more details and textures to make the 3D model reach the final high resolution. The goal of this stage is to generate a high-quality 3D model with rich details. Specifically, the diffusion result of the high-resolution stage It is expressed by the following formula:

[0068] Furthermore, when training the diffusion model, the loss function of the diffusion model is calculated to constrain the noise estimate generated by the model to be as close to the real noise as possible, thereby guiding the reverse diffusion process to approach the target data distribution in the right direction. Through hierarchical loss evaluation at low, medium, and high resolution stages, the loss function can ensure that the generated model maintains consistency and detail accuracy at different scales to optimize the output quality of each generation stage. Specifically, the loss function of the diffusion model is expressed by the following formula:

[0069] Where E represents the initial input and noise The expected value is obtained by sampling to ensure the stability and robustness of the loss function under different input conditions.

[0070] Step 1042: Fusing the diffusion results outputted from each diffusion stage to obtain the initial three-dimensional model.

[0071] In the specific implementation of step 1042, the diffusion results outputted at each diffusion stage in the diffusion model are fused to obtain an initial three-dimensional model. .

[0072] Here, according to the embodiment provided by the present application, in order to optimize the generated initial three-dimensional model , so that it conforms to the geometric details and structural features described in the text, the shape prior optimization algorithm can be used to adjust the output of the model to improve the accuracy and quality of the generated shape. Given an initial shape, that is, the real shape to be generated, the generated shape is compared with the real shape to find an optimized shape that better conforms to the given geometric shape.

[0073] S105, inputting the initial three-dimensional model and the cross-modal feature vector into a pre-trained texture generative adversarial network to obtain a target three-dimensional model of the target facility.

[0074] Here, the surface of the target 3D model is set with a facility texture map of the target facility. In 3D modeling, a texture map usually refers to a 2D image that is applied to the surface of a 3D model to simulate the object's color, details, material and other visual characteristics. The texture map can contain color information, bump information, reflectivity information, etc. This information is mapped to the surface of the 3D model through mapping technology, making the model look more realistic and detailed when rendered. The facility texture map is set on the surface of the target 3D model of the target facility to achieve high-fidelity reproduction of the facility texture and enhance the detail richness and visual realism of the generated 3D model.

[0075] Specifically, according to the embodiment provided by the present application, the texture generation adversarial network uses a texture generation network based on the Wasserstein generation adversarial network (WGAN) to add high-quality texture to the generated initial three-dimensional model of the target facility to achieve a realistic rendering effect. With respect to the above step S105, in the specific implementation, the initial three-dimensional model and the cross-modal feature vector are input into the pre-trained texture generation adversarial network to obtain a target three-dimensional model with a facility texture map set on the surface.

[0076] As an optional embodiment, with respect to the above step S105, the inputting the initial three-dimensional model and the cross-modal feature vector into a pre-trained texture generative adversarial network to obtain a target three-dimensional model of the target facility includes: Step 1051: Use the cross-modal feature vector and the second random noise vector as second input data, and input the second input data into the generator of the texture generative adversarial network to obtain the facility texture map.

[0077] For the generated initial 3D model ∈R N×3 (R represents the geometry of N vertices in the initial 3D model) The corresponding texture map is generated by the generator of the texture generation adversarial network For the above step 1051, in the specific implementation, first define a random noise vector , the cross-modal feature vector The second random noise vector is input as the second input data to the texture generation adversarial network generator G θ In the above example, we get the facility texture map Specifically, .

[0078] Step 1052: Generate the target three-dimensional model using the facility texture map and the initial three-dimensional model.

[0079] Regarding the above step 1052, in the specific implementation, the facility texture map of the target facility After generation, the facility texture map is transformed into Mapping to the initial 3D model to obtain the target three-dimensional model of the target facility.

[0080] Specifically, according to the generation method provided in the embodiment of the present application, the texture generation adversarial network is trained through the following steps: I: Initialize the parameters of the generator and the discriminator in the texture generative adversarial network.

[0081] II: Input the first sample input data into the generator to obtain a first generated texture map, and input the first generated texture map and the real texture map into the discriminator, calculate the gradient penalty loss value of the discriminator, and update the parameters of the discriminator using the gradient descent method based on the gradient penalty loss value.

[0082] Here, continuing with the embodiment of the above steps, the sample input data should include a cross-modal sample feature vector and a random noise sample vector.

[0083] For the above steps I to II, in the specific implementation, first initialize the parameters of the generator and the discriminator in the texture generative adversarial network. Then, obtain the first sample input data, and then input the first sample input data into the generator to obtain the first generated texture map generated by the generator. The first stage of training is to input the real texture map and the first generated texture map generated by the generator into the discriminator. This process only updates the parameters of the discriminator to train the discriminator's ability to distinguish between true and false signals. Specifically, the first sample input data is output as the first generated texture map after passing through the generator. Then the real texture map and the first generated texture map are compared. Figure 1 And feed it as input to the discriminator, calculate the gradient penalty loss value of the discriminator, fix the weight parameters of the generator, train the discriminator, and use the gradient descent method to continuously update the parameters of the discriminator based on the gradient penalty loss value. Through the above process, a binary classification discriminator is trained. At this time, the discriminator is more likely to distinguish between real texture images and generated texture images because the quality of the generated signal is poor at the beginning.

[0084] Specifically, after the real texture map and the first generated texture map are input into the discriminator, the discriminator Distinguishing between real texture images And the first generated texture map , the output is a scalar , represents the realism estimation of the input texture, expressed as .

[0085] At the same time, WGAN uses Wasserstein distance as the target for generative adversarial training, and adds a gradient penalty term to make the discriminator satisfy the 1-Lipschitz condition to obtain the final optimization target. Specifically, the gradient penalty loss function penalizes the discriminator's gradient norm at the linear interpolation point between the real data and the generated data, making the discriminator's gradient norm close to 1. The gradient penalty loss value of the discriminator is calculated by the following formula:

[0086] in, represents the Wasserstein GAN loss function, represents the gradient penalty term, represents the gradient penalty loss value, is a hyperparameter that controls the weight of the gradient penalty term.

[0087] Thus, during the training of WGAN-GP, the total loss function By combining the Wasserstein GAN loss and the gradient penalty , ensuring that the gradient of the discriminator is close to 1 at the interpolation points between the real data and the generated data, thereby improving the stability and generation quality of the model.

[0088] III: Input the second sample input data into the generator to obtain a second generated texture map, input the second generated texture map into the discriminator, calculate the loss value of the generator based on the discrimination result of the discriminator, and update the parameters of the generator using the gradient descent method based on the loss value.

[0089] IV: Repeatedly and alternately training the generator and the discriminator until the generator and the discriminator converge to obtain the texture generative adversarial network.

[0090] For the above steps III-IV, in the specific implementation, the parameters of the discriminator are fixed in the second stage of training to train the generator. The second sample input data is input into the generator to obtain the second generated texture map. Then the second generated texture map is input into the discriminator, and the discriminator outputs the discrimination result, which is the discriminator's judgment of whether the real texture map and the second generated texture map are true or false. At the same time, the loss value of the generator is calculated based on the discrimination result of the discriminator, and the parameters of the generator are updated using the gradient descent method based on the loss value. The above steps II-III are just a cycle of adversarial training. The above process will be repeated later, and the above step II will be returned to perform to repeatedly train the generator and the discriminator alternately. In the training of the generative adversarial network, the generator and the discriminator compete with each other and continuously optimize their respective parameters until both the generator and the discriminator converge to obtain a texture generative adversarial network. In this state, the signal quality generated by the generator is high, and the discriminator cannot accurately judge whether it is true or false, so that the generated texture map is more realistic. In summary, this embodiment can effectively train the texture generation adversarial network to generate realistic and diverse facility texture maps by adding a gradient penalty term to the discriminator, generating a texture map to input into the discriminator, and iterating adversarial training.

[0091] The method for generating a three-dimensional model of a facility provided in an embodiment of the present application first obtains images of an original model corresponding to a target facility at multiple viewing angles, and information text data corresponding to the target facility; then, visual features are extracted from the multiple images to obtain a visual style feature vector, and text features are extracted from the information text data to obtain a text feature vector; the visual style feature vector and the text feature vector are input into a pre-trained cross-modal feature fusion model for cross-modal feature fusion to obtain a cross-modal feature vector; the cross-modal feature vector is input into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility; finally, the initial three-dimensional model and the cross-modal feature vector are input into a pre-trained texture generative adversarial network to obtain a target three-dimensional model of the target facility; wherein the surface of the target three-dimensional model is provided with a facility texture map of the target facility.

[0092] This application uses the natural language description of the facility as input, combined with cross-modal feature extraction and multi-level generation strategies, to provide high-quality facility 3D model generation. First, the feature information of the text description and multi-view images is integrated to generate an accurate semantic feature representation of the facility. Then, the input semantic feature vector is subjected to shape generation and optimization through a diffusion model to generate a 3D shape model of the facility that meets the description. Finally, a texture generative adversarial network is used to achieve high-fidelity reproduction of the facility texture, which improves the detail richness and visual realism of the generated 3D facility model.

[0093] See also Figure 2 and Figure 3 , Figure 2 This is one of the structural schematic diagrams of a device for generating a three-dimensional model of a facility provided in an embodiment of the present application. Figure 3 This is a second structural diagram of a device for generating a three-dimensional model of a facility provided in an embodiment of the present application. Figure 2 As shown in , the generating device 200 includes: An acquisition module 201 is used to acquire images of an original model corresponding to a target facility at multiple viewing angles, and information text data corresponding to the target facility; A feature vector extraction module 202 is used to extract visual features from the plurality of images to obtain visual style feature vectors, and to extract text features from the information text data to obtain text feature vectors; A feature fusion module 203 is used to input the visual style feature vector and the text feature vector into a pre-trained cross-modal feature fusion model to perform cross-modal feature fusion to obtain a cross-modal feature vector; An initial three-dimensional model generation module 204 is used to input the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility; The target three-dimensional model generation module 205 is used to input the initial three-dimensional model and the cross-modal feature vector into a pre-trained texture generation adversarial network to obtain a target three-dimensional model of the target facility; wherein the surface of the target three-dimensional model is provided with a facility texture map of the target facility.

[0094] Furthermore, when the feature vector extraction module 202 is used to extract visual features from the plurality of images to obtain visual style feature vectors and to extract text features from the information text data to obtain text feature vectors, the feature vector extraction module 202 is also used to: For each image, convert the image into a plurality of image tag sequences, and input the plurality of image tag sequences into an image encoder to obtain an image embedding vector; Inputting the image embedding vector into the ViT model for encoding to obtain image features, and performing average pooling processing on the image features to obtain an image feature vector of the image; Performing vector concatenation on the image feature vectors of the plurality of images to obtain a target feature vector, and performing normalization processing on the target feature vector to obtain the visual style feature vector; Converting the information text data into a plurality of text tag sequences, and inputting the plurality of text tag sequences into a text encoder for embedding to obtain a text embedding vector; Inputting the text embedding vector into the Transformer model for encoding to obtain text features, and performing average pooling processing on the text features to obtain an initial feature vector of the information text data; The initial feature vector is normalized to obtain a text feature vector of the information text data.

[0095] Furthermore, when the initial three-dimensional model generation module 204 is used to input the cross-modal feature vector into a pre-trained diffusion model to obtain the initial three-dimensional model of the target facility, the initial three-dimensional model generation module 204 is also used to: The cross-modal feature vector and the first random noise vector are used as first input data, and the first input data is input into the diffusion model to obtain a diffusion result outputted by each diffusion stage in the diffusion model; wherein the diffusion stage includes a low-resolution stage, a medium-resolution stage, and a high-resolution stage; The diffusion results outputted in each diffusion stage are fused to obtain the initial three-dimensional model.

[0096] Furthermore, when the target 3D model generation module 205 is used to input the initial 3D model and the cross-modal feature vector into a pre-trained texture generation adversarial network to obtain the target 3D model of the target facility, the target 3D model generation module 205 is also used to: Using the cross-modal feature vector and the second random noise vector as second input data, and inputting the second input data into a generator of the texture generative adversarial network to obtain the facility texture map; The target three-dimensional model is generated using the facility texture map and the initial three-dimensional model.

[0097] See also Figure 3 The generating device 200 further includes a model training module 206, and the model training module 206 is used to train the texture generation adversarial network through the following steps: Initialize the parameters of the generator and the parameters of the discriminator in the texture generation adversarial network; Inputting first sample input data into the generator to obtain a first generated texture map, and inputting the first generated texture map and the real texture map into the discriminator, calculating the gradient penalty loss value of the discriminator, and updating the parameters of the discriminator using the gradient descent method based on the gradient penalty loss value; Inputting the second sample input data into the generator to obtain a second generated texture map, inputting the second generated texture map into the discriminator, calculating the loss value of the generator based on the discrimination result of the discriminator, and updating the parameters of the generator using the gradient descent method based on the loss value; The generator and the discriminator are repeatedly trained alternately until the generator and the discriminator converge, thereby obtaining the texture generative adversarial network.

[0098] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0099] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the method for generating the three-dimensional facility model in the method embodiment shown, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0100] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the method for generating the three-dimensional facility model in the method embodiment shown, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0105] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0106] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0107] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a three-dimensional model of a facility, characterized in that: The generation method comprises: Acquire images of an original model corresponding to a target facility at multiple viewing angles, and information text data corresponding to the target facility; Extracting visual features from the plurality of images to obtain a visual style feature vector, and extracting text features from the information text data to obtain a text feature vector; Inputting the visual style feature vector and the text feature vector into a pre-trained cross-modal feature fusion model for cross-modal feature fusion to obtain a cross-modal feature vector; Inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility; The initial three-dimensional model and the cross-modal feature vector are input into a pre-trained texture generative adversarial network to obtain a target three-dimensional model of the target facility; wherein the surface of the target three-dimensional model is provided with a facility texture map of the target facility.

2. The generation method according to claim 1, characterized in that: The step of extracting visual features from the plurality of images to obtain a visual style feature vector and extracting text features from the information text data to obtain a text feature vector comprises: For each image, convert the image into a plurality of image tag sequences, and input the plurality of image tag sequences into an image encoder to obtain an image embedding vector; Inputting the image embedding vector into the ViT model for encoding to obtain image features, and performing average pooling processing on the image features to obtain an image feature vector of the image; Performing vector concatenation on the image feature vectors of the plurality of images to obtain a target feature vector, and performing normalization processing on the target feature vector to obtain the visual style feature vector; Converting the information text data into a plurality of text tag sequences, and inputting the plurality of text tag sequences into a text encoder for embedding to obtain a text embedding vector; Inputting the text embedding vector into the Transformer model for encoding to obtain text features, and performing average pooling processing on the text features to obtain an initial feature vector of the information text data; The initial feature vector is normalized to obtain a text feature vector of the information text data.

3. The generation method according to claim 1, characterized in that: The step of inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility includes: The cross-modal feature vector and the first random noise vector are used as first input data, and the first input data is input into the diffusion model to obtain a diffusion result outputted by each diffusion stage in the diffusion model; wherein the diffusion stage includes a low-resolution stage, a medium-resolution stage, and a high-resolution stage; The diffusion results outputted in each diffusion stage are fused to obtain the initial three-dimensional model.

4. The generation method according to claim 1, characterized in that: The step of inputting the initial three-dimensional model and the cross-modal feature vector into a pre-trained texture generation adversarial network to obtain a target three-dimensional model of the target facility includes: Using the cross-modal feature vector and the second random noise vector as second input data, and inputting the second input data into a generator of the texture generative adversarial network to obtain the facility texture map; The target three-dimensional model is generated using the facility texture map and the initial three-dimensional model.

5. The generation method according to claim 4, characterized in that: The texture generation adversarial network is trained by the following steps: Initialize the parameters of the generator and the parameters of the discriminator in the texture generation adversarial network; Inputting first sample input data into the generator to obtain a first generated texture map, and inputting the first generated texture map and the real texture map into the discriminator, calculating the gradient penalty loss value of the discriminator, and updating the parameters of the discriminator using the gradient descent method based on the gradient penalty loss value; Inputting the second sample input data into the generator to obtain a second generated texture map, inputting the second generated texture map into the discriminator, calculating the loss value of the generator based on the discrimination result of the discriminator, and updating the parameters of the generator using the gradient descent method based on the loss value; The generator and the discriminator are repeatedly trained alternately until the generator and the discriminator converge, thereby obtaining the texture generative adversarial network.

6. A device for generating a three-dimensional model of a facility, characterized in that: The generating device comprises: An acquisition module, used to acquire images of the original model corresponding to the target facility at multiple viewing angles, and information text data corresponding to the target facility; A feature vector extraction module, used for extracting visual features from the plurality of images to obtain visual style feature vectors, and extracting text features from the information text data to obtain text feature vectors; A feature fusion module, used for inputting the visual style feature vector and the text feature vector into a pre-trained cross-modal feature fusion model for cross-modal feature fusion to obtain a cross-modal feature vector; An initial three-dimensional model generation module, used for inputting the cross-modal feature vector into a pre-trained diffusion model to obtain an initial three-dimensional model of the target facility; A target three-dimensional model generation module is used to input the initial three-dimensional model and the cross-modal feature vector into a pre-trained texture generation adversarial network to obtain a target three-dimensional model of the target facility; wherein the surface of the target three-dimensional model is provided with a facility texture map of the target facility.

7. The generating device according to claim 6, characterized in that: When the feature vector extraction module is used to extract visual features from the plurality of images to obtain visual style feature vectors, and to extract text features from the information text data to obtain text feature vectors, the feature vector extraction module is further used to: For each image, convert the image into a plurality of image tag sequences, and input the plurality of image tag sequences into an image encoder to obtain an image embedding vector; Inputting the image embedding vector into the ViT model for encoding to obtain image features, and performing average pooling processing on the image features to obtain an image feature vector of the image; Performing vector concatenation on the image feature vectors of the plurality of images to obtain a target feature vector, and performing normalization processing on the target feature vector to obtain the visual style feature vector; Converting the information text data into a plurality of text tag sequences, and inputting the plurality of text tag sequences into a text encoder for embedding to obtain a text embedding vector; Inputting the text embedding vector into the Transformer model for encoding to obtain text features, and performing average pooling processing on the text features to obtain an initial feature vector of the information text data; The initial feature vector is normalized to obtain a text feature vector of the information text data.

8. The generating device according to claim 6, characterized in that When the initial three-dimensional model generation module is used to input the cross-modal feature vector into a pre-trained diffusion model to obtain the initial three-dimensional model of the target facility, the initial three-dimensional model generation module is further used to: The cross-modal feature vector and the first random noise vector are used as first input data, and the first input data is input into the diffusion model to obtain a diffusion result outputted by each diffusion stage in the diffusion model; wherein the diffusion stage includes a low-resolution stage, a medium-resolution stage, and a high-resolution stage; The diffusion results outputted in each diffusion stage are fused to obtain the initial three-dimensional model.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for generating a three-dimensional model of a facility as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating a three-dimensional model of a facility as described in any one of claims 1 to 5 are executed.

Images