A neural network integrating text and image information and its architectural free-form surface meshing design method

Abstract

The invention discloses a neural network integrating text and image information and a building free-form surface mesh division design method thereof. The method comprises the following steps: step S1, obtaining a building free-form surface to be meshed and a required mesh division type text instruction; step S2, generating a free-form surface curvature cloud map and a free-form surface elevation cloud map corresponding to the building free-form surface, and at the same time, converting the mesh division type text instruction into a word vector through word embedding; step S3, inputting the obtained free-form surface curvature cloud map, free-form surface elevation cloud map and mesh division type instruction word vector into a pre-trained neural network model integrating text and image information, and generating a building free-form surface mesh structure that meets the mesh division type instruction requirements through the neural network model integrating text and image information; step S4, extracting mesh nodes and topological relationships in the building free-form surface mesh structure generated by the neural network model integrating text and image information, then projecting the mesh nodes into a three-dimensional space and connecting them using topological relationships, so as to finally obtain the building free-form surface mesh structure in space.

Description

Technical Field

[0001] The present invention belongs to the field of civil engineering and computer deep learning application technology, and in particular to a neural network integrating text and image information and a building free-form surface meshing design method thereof. Background Art

[0002] In recent years, architectural free-form surface structures have been widely used in large-span space buildings such as stadiums, large airports, and exhibition halls due to their rich and diverse shapes. However, the diversity and irregularity of free-form surface shapes make reasonable meshing design a challenging research topic.

[0003] With the rapid development of artificial intelligence technology, especially the new data-driven paradigm is profoundly changing various fields, image generation algorithms based on deep learning have made significant progress in structural intelligent design methods. Taking the meshing of architectural free-form surfaces as an example, the use of generative adversarial network (GAN) technology can not only reduce the dependence on the experience of engineering designers, but also simplify the complex process of explicit programming that must consider all surface characteristics. However, the existing free-form surface meshing methods based on generative adversarial networks cannot specify the basic unit type of the generated mesh structure. In order to efficiently carry out the meshing design of architectural free-form surfaces, it is urgent to propose a new intelligent design method that can quickly generate architectural free-form surface structures with specified grid basic unit types according to user needs. This will not only improve the flexibility and accuracy of the design, but also greatly improve the efficiency of meshing design and promote the wider application of architectural free-form surface structures. Summary of the invention

[0004] Purpose of the invention: The present invention aims to solve the problem that the existing AI-based architectural free-form surface meshing technology cannot generate a architectural free-form surface mesh structure of a specified mesh basic unit type according to user needs, and proposes a neural network that integrates text and image information and a design method for architectural free-form surface meshing. After a given architectural free-form surface is given, the method can directly perform architectural free-form surface meshing according to the user's meshing type text instruction.

[0005] Technical solution, in order to solve the above technical problems, the present invention proposes a neural network integrating text and image information and a method for designing architectural free-form surface meshes, the method comprising the following steps:

[0006] Step S1, obtaining a free-form surface of a building to be meshed and a required meshing type text instruction;

[0007] Step S2, generating a free-form surface curvature cloud map and a free-form surface elevation cloud map corresponding to the building free-form surface, and converting the grid division type text instruction into a word vector through word embedding;

[0008] Step S3, inputting the obtained free-form surface curvature cloud map, free-form surface elevation cloud map and meshing type instruction word vector into a pre-trained neural network model that fuses text and image information, and generating a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that fuses text and image information;

[0009] Step S4, extracting the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model integrating text and image information, projecting the grid nodes into three-dimensional space and connecting them using topological relationships to obtain the architectural free-form surface grid structure in space.

[0010] Furthermore, in step S2, the grid partitioning type text instruction is converted into a word vector through word embedding, specifically including:

[0011] (2.1) The meshing type text instruction corresponds to the basic unit type of the building free-form surface mesh. According to the generation requirements, the meshing type text instruction is rec or tri, which indicates that the basic unit type of the building free-form surface mesh to be generated is a quadrilateral mesh unit or a triangular mesh unit;

[0012] (2.2) The word embedding model wiki.en.bin in the open source word embedding representation and sentence classification library FastText is used to convert the grid-type text instructions from text to word vectors. The data shape of the word vector is [1, N, 300], where 1 represents the number of text instructions, N represents the maximum word length, and 300 represents the dimension of the word vector generated by FastText. If the length N of the word vector is less than 50, the word vector is filled with the number 0 until the length is equal to 50. The final shape of the word vector is [1, 50, 300].

[0013] Furthermore, in step S3, the construction process of the pre-trained neural network model integrating text and image information specifically includes the following steps:

[0014] (3.1) Model training dataset construction

[0015] (3.1.1) Obtain a series of architectural free-form surfaces and name them S1, S2, …, S n , where n is the total number of free-form surfaces of the building;

[0016] (3.1.2) Using the n building free-form surfaces in step (3.1.1), generate the free-form surface curvature cloud maps corresponding to these n building free-form surfaces and put them in the folder train_A. The n free-form surface curvature cloud maps in the folder train_A are named Similarly, the n building free-form surfaces in step (3.1.1) are used to generate the corresponding free-form surface elevation cloud maps and placed in the folder train_H. The n free-form surface elevation cloud maps in the folder train_H are named

[0017] (3.1.3) Obtain the n building free-form surfaces in step (3.1.1) whose basic grid units are quadrilateral building free-form surface grid structures, and name the building free-form surface grid structures whose basic grid units are quadrilaterals as rec_S1, rec_S2, …, rec_S n Similarly, the grid structure of the building free-form surface whose basic grid unit is a triangle corresponding to the series of building free-form surfaces is obtained, and the grid structure of the building free-form surface whose basic grid unit is a triangle is named tri_S1, tri_S2,…,tri_S n , put all the above-mentioned building free-form surface mesh structures into the folder train_B;

[0018] (3.1.4) Read the file names of the 2n building free-form surface mesh structures in the folder train_B in step (3.1.3) in sequence, and extract the prefix in the file name, and generate the corresponding mesh division type text instruction according to the prefix. For the file names rec_S1, rec_S2, …, rec_S in the folder train_B, n The corresponding grid division type text instruction is to generate a building free-form surface grid structure with a grid basic unit of quadrilateral. The grid division type text instruction is converted from text to word vector using the word embedding model wiki.en.bin in the open source word embedding representation and sentence classification library FastText. The data shape of the word vector is [1, N, 300], where 1 represents the number of text instructions, M represents the maximum word length, and 300 represents the word vector dimension generated by FastText. If the length N of the word vector is less than 50, the number 0 is used to fill the one-dimensional word vector until the length is equal to 50. The final word vector shape is [1, 50, 300]. Finally, the corresponding n word vectors are stored in the folder train_C and the file names are taken in turn. Similarly, for the files in the folder train_B named tri_S1, tri_S2, …, tri_S n n documents, obtain n word vectors and name them Then store it in train_C;

[0019] (3.2) Train a neural network model that integrates text and image information:

[0020] (3.2.1) The free-form surface curvature cloud map, the free-form surface elevation cloud map and the corresponding meshing type text command are used as the input of the neural network model, and the corresponding free-form surface mesh structure is used as the output of the neural network model. The free-form surface curvature cloud map is read from the folder train_A, the free-form surface elevation cloud map is read from the folder train_H, the meshing type text command is read from the folder train_C, and the free-form surface mesh structure is read from train_B;

[0021] (3.2.2) Divide the data in step (3.2.1) into a training data set and a test data set according to the form of input and output data pairs;

[0022] (3.2.3) Based on the variant of the generative adversarial network model GridGAN, its model structure is reconstructed so that the new neural network model can simultaneously integrate text and image information. The new neural network model integrating text and image information is named GridGAN-TXT.

[0023] (3.2.4) training the neural network model GridGAN-TXT that integrates text and image information in step (3.2.3) by using the training set consisting of the building free-form surface curvature cloud map, the free-form surface elevation cloud map, the grid division type text instruction, and the building free-form surface grid structure in step (3.2.2);

[0024] (3.2.5) The neural network model GridGAN-TXT that integrates text and image information trained in step (3.2.3) is tested and evaluated using the test set consisting of the building free-form surface curvature cloud map, free-form surface elevation cloud map, grid division type text instructions, and building free-form surface grid structure in step (3.2.2). The model that meets the requirements of computer vision evaluation indicators and free-form surface grid structure geometry evaluation indicators is used as the final neural network model GridGAN-TXT that integrates text and image information.

[0025] Furthermore, in step (3.2.3), based on the variant of the generative adversarial network model, GridGAN, its model structure is reconstructed so that the new neural network model can simultaneously integrate text and image information, which specifically includes the following steps:

[0026] (4.1) Define a text module for processing the word vector of the grid partitioning type text instruction, that is, the word vector in step (3.1.4). The text module includes a GRU unit, a mask calculation unit, a conditional vector calculation unit, a potential vector calculation unit, a conditional vector normalization and expansion unit in sequence:

[0027] (4.1.1) The GRU unit is used to encode word vectors. The GRU unit processes the content of the word vector from front to back and from back to front in order. Finally, the GRU unit embeds the word vector with an input dimension of 300 into a 512-dimensional hidden state.

[0028] (4.1.2) The mask calculation unit calculates the mask for each time step to process text sequences of different lengths and ensure that the hidden state is updated only at time steps within the valid length range;

[0029] (4.1.3) Conditional vector calculation unit, which combines the forward and backward hidden states to calculate the conditional vector;

[0030] (4.1.4) Latent vector calculation unit. The latent vector calculation consists of a fully connected layer and an activation function. The input feature dimension and output feature dimension of the fully connected layer are 512 and 128 respectively. The activation function uses the LeakyReLU function with a negative slope of 0.2 and performs in-situ calculation. The conditional vector C is generated by reparameterization. current , calculate the conditional vector C current The mean and standard deviation of current and STD current ;

[0031] (4.1.5) Conditional vector normalization and expansion unit, expansion conditional vector C current The dimension of the conditional vector C current The dimension is expanded from 2 to 4, and then the conditional vector C is calculated using formula (1). current Normalize to get C new , where M new and STD new The new mean and standard deviation of the conditional vector are maintained at C. bew The first and second dimensions of remain unchanged, and the third and fourth dimensions are repeated to match the size of the specified image feature F. This process is expressed by formula (2), where C is the final conditional vector, repeat is a function in the Pytorch library for repeated tensor operations, F.size(2) represents the third dimension of F, and F.size(3) represents the fourth dimension of F;

[0032]

[0033] C=C new .repeat(1,1,F.size(2),F.size(3)) (2)

[0034] (4.2) Add the text module in step (4.1) to the generator of GridGAN and perform the following operations:

[0035] (4.2.1) Input the building free-form surface curvature cloud map and the free-form surface elevation cloud map into the convolutional neural network block in the generator, and obtain the image feature e by downsampling the image data. The image feature e is the specific value of the image feature F in the generator in step (4.1.5);

[0036] (4.2.2) Input the word vector in step (3.1.4) into the text module in the generator to obtain the corresponding conditional vector C G ;

[0037] (4.2.3) The image feature e in step (4.2.1) and the conditional vector C in step (4.2.2) are converted into G Concatenate along the channel dimension to achieve image features e and conditional vector C G The fusion of G1 ,In formula (3), torch.cat represents the set-join function in the Pytorch library;

[0038] M G1 = torch.cat((e,C G ),1) (3)

[0039] (4.2.4) The fused feature M G1 The residual block network in the input generator is used to extract features through the residual block. The further fused extracted features are named R. The further fused extracted features R are added to the image features e in step (4.2.1) at the element level, that is, the elements at the same position of R and e are added. This process can be expressed by formula (4), where T is the result after element-level addition;

[0040] T=e+R (4)

[0041] (4.2.5) The added feature T and conditional vector C are G It is spliced ​​again along the channel dimension and further fused, and the fused feature is named M G2 , the fusion feature M G2 Input the deconvolution network block in the generator to finally obtain the building free-form surface mesh structure that meets the requirements of the meshing type text instruction;

[0042] M G2 = torch.cat((T,C G ),1) (5)

[0043] (4.3) Add the text module in step (4.1) to the discriminator of GridGAN and perform the following operations:

[0044] (4.3.1) The building free-form surface curvature cloud map, the free-form surface elevation cloud map and the building free-form surface mesh structure are spliced ​​in the channel dimension, and the spliced ​​image feature is represented as IMG, and the image feature IMG is the specific value of the image feature F in the discriminator in step (4.1.5). Here, the building free-form surface mesh structure is the building free-form surface mesh structure in the training set in step (3.2.2), which is called the true building free-form surface mesh structure, or the building free-form surface mesh structure generated by the generator in step (4.2.5), which is called the fake building free-form surface mesh structure;

[0045] (4.3.2) Input the word vector in step (3.1.4) into the text module in the generator to obtain the corresponding conditional vector C D ;

[0046] (4.3.3) The discriminator contains two sub-discriminators with exactly the same structure. The image feature IMG in step (4.3.1) and the conditional vector C in step (4.3.2) are converted into D Concatenate along the channel dimension to achieve image features IMG and conditional vector C D The fusion of D1 , the fused feature M D1 Pass in the first sub-discriminator and use average pooling nn.AvgPool2d to respectively pool the image feature IMG in step (4.3.1) and the conditional vector C in step (4.3.2) D Perform a downsampling, concatenate the two along the channel dimension, and represent the fused feature as M D2 , the fused feature M D2 The input is passed to the second sub-discriminator. The two sub-discriminators discriminate the current input from different dimensions. The output results are the judgment of the building free-form surface mesh structure and the discrimination of the mesh division type text instruction type.

[0047] M D1 = torch.cat((IMG,C D ),1) (6).

[0048] Furthermore, in step (3.2.3), the loss function of the neural network model GridGAN-TXT that integrates text and image information is calculated as:

[0049] (5.1) The loss function calculation of the neural network model GridGAN-TXT that integrates text and image information is performed in the process of alternating training of the discriminator and the generator. If the input of the discriminator in step (4.3.1) is the true building free-form surface grid structure, then the output of the discriminator in step (4.3.3) is P r and Cr , if the input of the discriminator in step (4.3.1) contains a false building free-form surface mesh structure, then the output of the discriminator in step (4.3.3) is P f and C f , where P r and P f The judgment of the corresponding building free surface mesh results, C r and C f Identify the text instruction type corresponding to the mesh division type;

[0050] (5.2) Generate adversarial loss calculation of the adversarial network, the loss function of the generator L G For formula (7), the loss function L of the discriminator is D It is formula (8), where MSELoss represents the mean square error loss function, True and False represent the true and false judgments respectively;

[0051] L G_GAN =MSELoss(P f ,True) (7)

[0052] L D_GAN =MSELoss(P r , True)+MSELoss(P f , False) (8)

[0053] (5.3) Text loss L C Calculation is used to calculate the consistency of the basic unit of the grid, see formula (9), where NLLLoss represents the negative log-likelihood loss function, C real and C fake They represent the true and false instructions of the meshing type text instructions respectively;

[0054] L C =NLLLoss(C r ,C real )+NLLLoss(C f ,C fake ) (9)

[0055] (5.4) Feature matching loss L G_Feat Calculation, see formula (10), where L1Loss represents the L1 loss function, that is, the mean absolute error, i represents the i-th sub-discriminator, i = 1, 2, j represents the j-th layer feature, j = 1, 2, 3, 4, (P r ) i,j represents the i-th sub-discriminator pair P r The j-th layer feature is calculated, the same, (P f ) i,jrepresents the i-th sub-discriminator pair P f Calculate the j-th layer features;

[0056]

[0057] (5.5) Feature matching loss L based on VGG neural network G_VGG Calculation, see formula (11), where VGG represents the VGG19 neural network;

[0058] L G_VGG =L1Loss{VGG(P r ),VGG(P f )} (11)

[0059] (5.6) Text and image consistency loss L G_CCL Calculation is used to calculate the consistency between the generated building free-form surface mesh and the meshing type text instruction, see formula (12), where CCloss represents the text and image consistency loss function;

[0060] L G_CCL =CCLoss(P r ,P f ,C real ) (12)

[0061] (5.7) Combining the loss functions in steps (5.2) to (5.6) we get the total loss function L of the generator G And the total loss function L of the discriminator D , see formula (13) and formula (14) respectively, where α, β, γ and δ are the corresponding weight coefficients, and the generator and discriminator of the pre-trained model are obtained by L G and L D Continuously optimize model parameters;

[0062] L G =L G_GAN +βL G_Feat +γL G_VGG +δL G_CCL (13)

[0063]

[0064] In addition, the present invention also proposes a neural network integrating text and image information and a building free-form surface meshing design system thereof, the system comprising:

[0065] (1) A data acquisition module, which acquires the free-form surface of the building to be meshed and the text instructions of the required meshing type;

[0066] (2) Data construction module, which generates free-form surface curvature cloud maps and free-form surface elevation cloud maps corresponding to the building free-form surface, and converts the grid division type text instructions into word vectors through word embedding;

[0067] (3) A model building module, which inputs the obtained free-form surface curvature cloud map, free-form surface elevation cloud map, and meshing type instruction word vector into a pre-trained neural network model that integrates text and image information, and generates a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that integrates text and image information;

[0068] (4) The scheme design module extracts the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model that integrates text and image information, then projects the grid nodes into three-dimensional space and connects them using topological relationships to ultimately obtain the architectural free-form surface grid structure in space.

[0069] In addition, the present invention also proposes a computer program product, including a computer program, which, when executed by a processor, implements any one of the neural networks that integrate text and image information and the architectural free-form surface meshing design methods.

[0070] In addition, the present invention also proposes a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement any one of the neural networks that integrate text and image information and the architectural free-form surface meshing design methods described.

[0071] In addition, the present invention proposes an electronic device, including a processor, a storage, and a computer program stored in the storage and runnable on the processor, characterized in that when the processor executes the program, the program implements any one of a neural network that integrates text and image information and a method for designing architectural free-form surface meshing.

[0072] Beneficial effects: Compared with the prior art, the present invention adopts the above technical solution to have the following beneficial technical effects:

[0073] (1) The present invention proposes a neural network that integrates text and image information and a method for designing architectural free-form surface meshing. The method integrates text and image information in a generative adversarial network to solve the problem that the current artificial intelligence-based architectural free-form surface meshing technology cannot generate architectural free-form surface mesh structures of specified grid basic unit types according to user needs;

[0074] (2) This method further improves the efficiency of using AI-based architectural free-form surface meshing technology, avoiding the time-consuming process of training the model separately to generate architectural free-form surface meshes of specific grid basic unit types in existing methods, thereby greatly improving the efficiency of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0076] Figure 1 A schematic diagram of the process of a neural network integrating text and image information and a method for designing architectural free-form surface meshes thereof provided by an embodiment of the present invention;

[0077] Figure 2 A schematic diagram of a process of converting a grid partitioning type text instruction provided in an embodiment of this article into a word vector through word embedding;

[0078] Figure 3 A schematic diagram of a model training data set construction for a neural network integrating text and image information and a building free-form surface meshing design method provided in an embodiment of this article;

[0079] Figure 4 A schematic diagram of a text processing module of a neural network integrating text and image information and a design method for meshing architectural free-form surfaces provided in an embodiment of this article;

[0080] Figure 5 A schematic diagram of the image information and text information fusion process of a neural network that fuses text and image information and a building free-form surface meshing design method provided in an embodiment of this article;

[0081] Figure 6 A schematic diagram of calculating a model generator loss function and a discriminator loss function of a neural network integrating text and image information and a design method for meshing architectural free-form surfaces provided in an embodiment of this article;

[0082] Figure 7 A schematic diagram of the module connection of a neural network integrating text and image information and a building free-form surface mesh division design system provided in an embodiment of this article;

[0083] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this article. DETAILED DESCRIPTION

[0084] In order to make the technical solution, advantages and ultimate purpose of the present invention clearer, the present invention will be further explained and illustrated in conjunction with the accompanying drawings and specific implementation methods.

[0085] like Figure 1 As shown, the present invention proposes a neural network integrating text and image information and a building free-form surface meshing design method thereof, the method comprising the following steps:

[0086] Step S1, obtaining a free-form surface of a building to be meshed and a required meshing type text instruction;

[0087] Step S2, generating a free-form surface curvature cloud map and a free-form surface elevation cloud map corresponding to the building free-form surface, and converting the grid division type text instruction into a word vector through word embedding;

[0088] Step S3, inputting the obtained free-form surface curvature cloud map, free-form surface elevation cloud map and meshing type instruction word vector into a pre-trained neural network model that fuses text and image information, and generating a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that fuses text and image information;

[0089] Step S4, extracting the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model integrating text and image information, projecting the grid nodes into three-dimensional space and connecting them using topological relationships to obtain the architectural free-form surface grid structure in space.

[0090] Furthermore, in step S2, the grid partitioning type text instruction is converted into a word vector through word embedding, specifically including:

[0091] (2.1) The meshing type text instruction corresponds to the basic unit type of the building free-form surface mesh. According to the generation requirements, the meshing type text instruction is rec or tri, which indicates that the basic unit type of the building free-form surface mesh to be generated is a quadrilateral mesh unit or a triangular mesh unit;

[0092] (2.2) The word embedding model wiki.en.bin in the open source word embedding representation and sentence classification library FastText is used to convert the grid-type text instructions from text to word vectors. The data shape of the word vector is [1, N, 300], where 1 represents the number of text instructions, N represents the maximum word length, and 300 represents the dimension of the word vector generated by FastText. If the length N of the word vector is less than 50, the word vector is filled with the number 0 until the length is equal to 50. The final shape of the word vector is [1, 50, 300].

[0093] Furthermore, in step S3, the construction process of the pre-trained neural network model integrating text and image information specifically includes the following steps:

[0094] (3.1) Model training dataset construction

[0095] (3.1.1) Obtain a series of architectural free-form surfaces and name them S1, S2, …, S n , where n is the total number of free-form surfaces of the building;

[0096] (3.1.2) Using the n building free-form surfaces in step (3.1.1), generate the free-form surface curvature cloud maps corresponding to these n building free-form surfaces and put them in the folder train_A. The n free-form surface curvature cloud maps in the folder train_A are named Similarly, the n building free-form surfaces in step (3.1.1) are used to generate the corresponding free-form surface elevation cloud maps and placed in the folder train_H. The n free-form surface elevation cloud maps in the folder train_H are named

[0097] (3.1.3) Obtain the n building free-form surfaces in step (3.1.1) whose basic grid units are quadrilateral building free-form surface grid structures, and name the building free-form surface grid structures whose basic grid units are quadrilaterals as rec_S1, rec_S2, …, rec_S n Similarly, the grid structure of the building free-form surface whose basic grid unit is a triangle corresponding to the series of building free-form surfaces is obtained, and the grid structure of the building free-form surface whose basic grid unit is a triangle is named tri_S1, tri_S2,…,tri_S n , put all the above-mentioned building free-form surface mesh structures into the folder train_B;

[0098] (3.1.4) Read the file names of the 2n building free-form surface mesh structures in the folder train_B in step (3.1.3) in sequence, and extract the prefix in the file name, and generate the corresponding mesh division type text instruction according to the prefix. For the file names rec_S1, rec_S2, …, rec_S in the folder train_B, n The corresponding grid division type text instruction is to generate a building free-form surface grid structure with a grid basic unit of quadrilateral. The grid division type text instruction is converted from text to word vector using the word embedding model wiki.en.bin in the open source word embedding representation and sentence classification library FastText. The data shape of the word vector is [1, N, 300], where 1 represents the number of text instructions, N represents the maximum word length, and 300 represents the word vector dimension generated by FastText. If the length of the word vector N is less than 50, the number 0 is used to fill the one-dimensional word vector until the length is equal to 50. The final word vector shape is [1, 50, 300]. Finally, the corresponding n word vectors are stored in the folder train_C and the file names are taken in turn. Similarly, for the files in the folder train_B named tri_S1, tri_S2, …, tri_S n n documents, obtain n word vectors and name them Then store it in train_C;

[0099] (3.2) Train a neural network model that integrates text and image information:

[0100] (3.2.1) The free-form surface curvature cloud map, the free-form surface elevation cloud map and the corresponding meshing type text command are used as the input of the neural network model, and the corresponding free-form surface mesh structure is used as the output of the neural network model. The free-form surface curvature cloud map is read from the folder train_A, the free-form surface elevation cloud map is read from the folder train_H, the meshing type text command is read from the folder train_C, and the free-form surface mesh structure is read from train_B;

[0101] (3.2.2) Divide the data in step (3.2.1) into a training data set and a test data set according to the form of input and output data pairs;

[0102] (3.2.3) Based on the variant of the generative adversarial network model GridGAN, its model structure is reconstructed so that the new neural network model can simultaneously integrate text and image information. The new neural network model integrating text and image information is named GridGAN-TXT.

[0103] (3.2.4) training the neural network model GridGAN-TXT that integrates text and image information in step (3.2.3) by using the training set consisting of the building free-form surface curvature cloud map, the free-form surface elevation cloud map, the grid division type text instruction, and the building free-form surface grid structure in step (3.2.2);

[0104] (3.2.5) The neural network model GridGAN-TXT that integrates text and image information trained in step (3.2.3) is tested and evaluated using the test set consisting of the building free-form surface curvature cloud map, free-form surface elevation cloud map, grid division type text instructions, and building free-form surface grid structure in step (3.2.2). The model that meets the requirements of computer vision evaluation indicators and free-form surface grid structure geometry evaluation indicators is used as the final neural network model GridGAN-TXT that integrates text and image information.

[0105] Furthermore, in step (3.2.3), based on the variant of the generative adversarial network model, GridGAN, its model structure is reconstructed so that the new neural network model can simultaneously integrate text and image information, which specifically includes the following steps:

[0106] (4.1) Define a text module for processing the word vector of the grid partitioning type text instruction, that is, the word vector in step (3.1.4). The text module includes a GRU unit, a mask calculation unit, a conditional vector calculation unit, a potential vector calculation unit, a conditional vector normalization and expansion unit in sequence:

[0107] (4.1.1) The GRU unit is used to encode word vectors. The GRU unit processes the content of the word vector from front to back and from back to front in order. Finally, the GRU unit embeds the word vector with an input dimension of 300 into a 512-dimensional hidden state.

[0108] (4.1.2) The mask calculation unit calculates the mask for each time step to process text sequences of different lengths and ensure that the hidden state is updated only at time steps within the valid length range;

[0109] (4.1.3) Conditional vector calculation unit, which combines the forward and backward hidden states to calculate the conditional vector;

[0110] (4.1.4) Latent vector calculation unit. The latent vector calculation consists of a fully connected layer and an activation function. The input feature dimension and output feature dimension of the fully connected layer are 512 and 128 respectively. The activation function uses the LeakyReLU function with a negative slope of 0.2 and performs in-situ calculation. The conditional vector C is generated by reparameterization. current , calculate the conditional vector C current The mean and standard deviation of current and STD current ;

[0111] (4.1.5) Conditional vector normalization and expansion unit, expansion conditional vector C current The dimension of the conditional vector C current The dimension is expanded from 2 to 4, and then the conditional vector C is calculated using formula (1). current Normalize to get C new , where M new and STD new The new mean and standard deviation of the conditional vector are maintained at C. new The first and second dimensions of remain unchanged, and the third and fourth dimensions are repeated to match the size of the specified image feature F. This process is expressed by formula (2), where C is the final conditional vector, repeat is a function in the Pytorch library for repeated tensor operations, F.size(2) represents the third dimension of F, and F.size(3) represents the fourth dimension of F;

[0112]

[0113] C=C new .repeat(1,1,F.size(2),F.size(3)) (2)

[0114] (4.2) Add the text module in step (4.1) to the generator of GridGAN and perform the following operations:

[0115] (4.2.1) Input the building free-form surface curvature cloud map and the free-form surface elevation cloud map into the convolutional neural network block in the generator, and obtain the image feature e by downsampling the image data. The image feature e is the specific value of the image feature F in the generator in step (4.1.5);

[0116] (4.2.2) Input the word vector in step (3.1.4) into the text module in the generator to obtain the corresponding conditional vector C G ;

[0117] (4.2.3) The image feature e in step (4.2.1) and the conditional vector C in step (4.2.2) are converted into G Concatenate along the channel dimension to achieve image features e and conditional vector C G The fusion of G1 ,In formula (3), torch.cat represents the set-join function in the Pytorch library;

[0118] M G1 = torch.cat((e,C G ),1) (3)

[0119] (4.2.4) The fused feature M G1 The residual block network in the input generator is used to extract features through the residual block. The further fused extracted features are named R. The further fused extracted features R are added to the image features e in step (4.2.1) at the element level, that is, the elements at the same position of R and e are added. This process can be expressed by formula (4), where T is the result after element-level addition;

[0120] T=e+R (4)

[0121] (4.2.5) The added feature T and conditional vector C are G It is spliced ​​again along the channel dimension and further fused, and the fused feature is named M G2 , the fusion feature M G2 Input the deconvolution network block in the generator to finally obtain the building free-form surface mesh structure that meets the requirements of the meshing type text instruction;

[0122] M G2 = torch.cat((T,C G ),1) (5)

[0123] (4.3) Add the text module in step (4.1) to the discriminator of GridGAN and perform the following operations:

[0124] (4.3.1) The building free-form surface curvature cloud map, the free-form surface elevation cloud map and the building free-form surface mesh structure are spliced ​​in the channel dimension, and the spliced ​​image feature is represented as IMG, and the image feature IMG is the specific value of the image feature F in the discriminator in step (4.1.5). Here, the building free-form surface mesh structure is the building free-form surface mesh structure in the training set in step (3.2.2), which is called the true building free-form surface mesh structure, or the building free-form surface mesh structure generated by the generator in step (4.2.5), which is called the fake building free-form surface mesh structure;

[0125] (4.3.2) Input the word vector in step (3.1.4) into the text module in the generator to obtain the corresponding conditional vector C D ;

[0126] (4.3.3) The discriminator contains two sub-discriminators with exactly the same structure. The image feature IMG in step (4.3.1) and the conditional vector C in step (4.3.2) are converted into D Concatenate along the channel dimension to achieve image features IMG and conditional vector C D The fusion of D1 , the fused feature M D1 Pass in the first sub-discriminator and use average pooling nn.AvgPool2d to respectively pool the image feature IMG in step (4.3.1) and the conditional vector C in step (4.3.2) D Perform a downsampling, concatenate the two along the channel dimension, and represent the fused feature as M D2 , the fused feature M D2 The input is passed to the second sub-discriminator. The two sub-discriminators discriminate the current input from different dimensions. The output results are the judgment of the building free-form surface mesh structure and the discrimination of the mesh division type text instruction type.

[0127] M D1 = torch.cat((IMG,C D ),1) (6).

[0128] Furthermore, in step (3.2.3), the loss function of the neural network model GridGAN-TXT that integrates text and image information is calculated as:

[0129] (5.1) The loss function calculation of the neural network model GridGAN-TXT that integrates text and image information is performed in the process of alternating training of the discriminator and the generator. If the input of the discriminator in step (4.3.1) is the true building free-form surface grid structure, then the output of the discriminator in step (4.3.3) is P r and C r , if the input of the discriminator in step (4.3.1) contains a false building free-form surface mesh structure, then the output of the discriminator in step (4.3.3) is P f and C f , where P r and P f The judgment of the corresponding building free surface mesh results, C r and C f Identify the text instruction type corresponding to the mesh division type;

[0130] (5.2) Generate adversarial loss calculation of the adversarial network, the loss function of the generator L G For formula (7), the loss function L of the discriminator is D It is formula (8), where MSELoss represents the mean square error loss function, True and False represent the true and false judgments respectively;

[0131] L G_GAN =MSELoss(P f ,True) (7)

[0132] L D_GAN =MSELoss(P r , True)+MSELoss(P f , False) (8)

[0133] (5.3) Text loss L C Calculation is used to calculate the consistency of the basic unit of the grid, see formula (9), where NLLLoss represents the negative log-likelihood loss function, C real and C fake They represent the true and false instructions of the meshing type text instructions respectively;

[0134] L C =NLLLoss(C r ,C real )+NLLLoss(C f ,C fake ) (9)

[0135] (5.4) Feature matching loss L G_FeatCalculation, see formula (10), where L1Loss represents the L1 loss function, that is, the mean absolute error, i represents the i-th sub-discriminator, i = 1, 2, j represents the j-th layer feature, j = 1, 2, 3, 4, (P r ) i,j represents the i-th sub-discriminator pair P r The j-th layer feature is calculated, the same, (P f ) i,j represents the i-th sub-discriminator pair P f Calculate the j-th layer features;

[0136]

[0137] (5.5) Feature matching loss L based on VGG neural network G_VGG Calculation, see formula (11), where VGG represents the VGG19 neural network;

[0138] L G_VGG =L1Loss{VGG(P r ),VGG(P f )} (11)

[0139] (5.6) Text and image consistency loss L G_CCL Calculation is used to calculate the consistency between the generated building free-form surface mesh and the meshing type text instruction, see formula (12), where CCloss represents the text and image consistency loss function;

[0140] L G_CCL =CCLoss(P r ,P f ,C real ) (12)

[0141] (5.7) Combining the loss functions in steps (5.2) to (5.6) we get the total loss function L of the generator G And the total loss function L of the discriminator D , see formula (13) and formula (14) respectively, where α, β, γ and δ are the corresponding weight coefficients, and the generator and discriminator of the pre-trained model are obtained by L G and L D Continuously optimize model parameters;

[0142] L G =L G_GAN +βL G_Feat +γL G_VGG +δL G_CCL (13)

[0143]

[0144] In addition, the present invention also proposes a neural network integrating text and image information and a building free-form surface meshing design system thereof, the system comprising:

[0145] (1) A data acquisition module, which acquires the free-form surface of the building to be meshed and the text instructions of the required meshing type;

[0146] (2) Data construction module, which generates free-form surface curvature cloud maps and free-form surface elevation cloud maps corresponding to the building free-form surface, and converts the grid division type text instructions into word vectors through word embedding;

[0147] (3) A model building module, which inputs the obtained free-form surface curvature cloud map, free-form surface elevation cloud map, and meshing type instruction word vector into a pre-trained neural network model that integrates text and image information, and generates a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that integrates text and image information;

[0148] (4) The scheme design module extracts the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model that integrates text and image information, then projects the grid nodes into three-dimensional space and connects them using topological relationships to ultimately obtain the architectural free-form surface grid structure in space.

[0149] The present invention also proposes that the physical structure of an electronic device may include the following components: a processor 810, a communications interface 820, a memory 930, and a communication bus 840. These components communicate with each other through the communication bus 840. The processor 810 can access the logic instructions in the memory 830 to execute a neural network integrating text and image information and a method for meshing a free-form surface of an architecture, the method comprising:

[0150] Step S1, obtaining a free-form surface of a building to be meshed and a required meshing type text instruction;

[0151] Step S2, generating a free-form surface curvature cloud map and a free-form surface elevation cloud map corresponding to the building free-form surface, and converting the grid division type text instruction into a word vector through word embedding;

[0152] Step S3, inputting the obtained free-form surface curvature cloud map, free-form surface elevation cloud map and meshing type instruction word vector into a pre-trained neural network model that fuses text and image information, and generating a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that fuses text and image information;

[0153] Step S4, extracting the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model integrating text and image information, projecting the grid nodes into three-dimensional space and connecting them using topological relationships to obtain the architectural free-form surface grid structure in space.

[0154] In addition, the logic instructions in the above-mentioned memory 830 can be implemented by a software function module and can be sold or used as an independent product. The logic instructions can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium (which can be a U disk, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a disk or an optical disk, etc., which can store program codes) and contains several instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0155] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a neural network that integrates text and image information and a method for meshing a free-form surface of a building provided in the above method. The method includes:

[0156] Step S1, obtaining a free-form surface of a building to be meshed and a required meshing type text instruction;

[0157] Step S2, generating a free-form surface curvature cloud map and a free-form surface elevation cloud map corresponding to the building free-form surface, and converting the grid division type text instruction into a word vector through word embedding;

[0158] Step S3, inputting the obtained free-form surface curvature cloud map, free-form surface elevation cloud map and meshing type instruction word vector into a pre-trained neural network model that fuses text and image information, and generating a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that fuses text and image information;

[0159] Step S4, extracting the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model integrating text and image information, projecting the grid nodes into three-dimensional space and connecting them using topological relationships to obtain the architectural free-form surface grid structure in space.

[0160] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a neural network integrating text and image information and a method for meshing a building free-form surface are implemented. The method includes:

[0161] Step S1, obtaining a free-form surface of a building to be meshed and a required meshing type text instruction;

[0162] Step S2, generating a free-form surface curvature cloud map and a free-form surface elevation cloud map corresponding to the building free-form surface, and converting the grid division type text instruction into a word vector through word embedding;

[0163] Step S3, inputting the obtained free-form surface curvature cloud map, free-form surface elevation cloud map and meshing type instruction word vector into a pre-trained neural network model that fuses text and image information, and generating a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that fuses text and image information;

[0164] Step S4, extracting the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model integrating text and image information, projecting the grid nodes into three-dimensional space and connecting them using topological relationships to obtain the architectural free-form surface grid structure in space.

[0165] The above embodiments are only used to illustrate the technical solutions of the present invention and do not constitute limitations thereon. Although the present invention is described in detail through the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or perform equivalent replacements on some of the technical features therein. These modifications or replacements will not change the essence of the corresponding technical solutions, nor will they deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network integrating text and image information and a method for designing architectural free-form surface meshes, characterized in that: The method comprises the following steps: Step S1, obtaining a free-form surface of a building to be meshed and a required meshing type text instruction; Step S2, generating a free-form surface curvature cloud map and a free-form surface elevation cloud map corresponding to the building free-form surface, and converting the grid division type text instruction into a word vector through word embedding; Step S3, inputting the obtained free-form surface curvature cloud map, free-form surface elevation cloud map and meshing type instruction word vector into a pre-trained neural network model that fuses text and image information, and generating a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that fuses text and image information; Step S4, extracting grid nodes and topological relationships in the building free-form surface grid structure generated by the neural network model integrating text and image information, projecting the grid nodes into a three-dimensional space and connecting them using topological relationships to obtain the building free-form surface grid structure in space; In step S2, the grid division type text instruction is converted into a word vector through word embedding, which specifically includes: (2.1) The meshing type text instruction corresponds to the basic unit type of the building free-form surface mesh. According to the generation requirements, the meshing type text instruction is rec or tri, which indicates that the basic unit type of the building free-form surface mesh to be generated is a quadrilateral mesh unit or a triangular mesh unit; (2.2) The word embedding model wiki.en.bin in the open source word embedding representation and sentence classification library FastText is used to convert the grid-type text instructions from text to word vectors. The data shape of the word vector is [1, N, 300], where 1 represents the number of text instructions, N represents the maximum word length, and 300 represents the dimension of the word vector generated by FastText. If the length N of the word vector is less than 50, the word vector is filled with the number 0 until the length is equal to 50. The final shape of the word vector is [1, 50, 300].

2. A neural network integrating text and image information and a method for designing architectural free-form surface meshes according to claim 1, characterized in that: In step S3, the construction process of the pre-trained neural network model integrating text and image information specifically includes the following steps: (3.1) Model training dataset construction (3.1.1) Obtain a series of architectural free-form surfaces and name them S1, S2, …, S n , where n is the total number of free-form surfaces of the building; (3.1.2) Using the n building free-form surfaces in step (3.1.1), generate the free-form surface curvature cloud maps corresponding to these n building free-form surfaces and put them in the folder train_A. The n free-form surface curvature cloud maps in the folder train_A are named Similarly, the n building free-form surfaces in step (3.1.1) are used to generate the corresponding free-form surface elevation cloud maps and placed in the folder train_H. The n free-form surface elevation cloud maps in the folder train_H are named (3.1.3) Obtain the n building free-form surfaces in step (3.1.1) whose basic grid units are quadrilateral building free-form surface grid structures, and name the building free-form surface grid structures whose basic grid units are quadrilaterals as rec_S1, rec_S2, …, rec_S n Similarly, the grid structure of the building free-form surface whose basic grid unit is a triangle corresponding to the series of building free-form surfaces is obtained, and the grid structure of the building free-form surface whose basic grid unit is a triangle is named tri_S1, tri_S2,…,tri_S n , put all the above-mentioned building free-form surface mesh structures into the folder train_B; (3.1.4) Read the file names of the 2n building free-form surface mesh structures in the folder train_B in step (3.1.3) in sequence, and extract the prefix in the file name, and generate the corresponding mesh division type text instruction according to the prefix. For the file names rec_S1, rec_S2, …, rec_S in the folder train_B, n The corresponding grid division type text instruction is to generate a building free-form surface grid structure with a grid basic unit of quadrilateral. The grid division type text instruction is converted from text to word vector using the word embedding model wiki.en.bin in the open source word embedding representation and sentence classification library FastText. The data shape of the word vector is [1, N, 300], where 1 represents the number of text instructions, N represents the maximum word length, and 300 represents the word vector dimension generated by FastText. If the length of the word vector N is less than 50, the number 0 is used to fill the one-dimensional word vector until the length is equal to 50. The final word vector shape is [1, 50, 300]. The corresponding n word vectors are stored in the folder train_C and the file names are taken in turn. Similarly, for the files in the folder train_B named tri_S1, tri_S2, …, tri_S n n documents, obtain n word vectors and name them Stored in train_C; (3.2) Train a neural network model that integrates text and image information: (3.2.1) The free-form surface curvature cloud map, the free-form surface elevation cloud map and the corresponding meshing type text command are used as the input of the neural network model, and the corresponding free-form surface mesh structure is used as the output of the neural network model. The free-form surface curvature cloud map is read from the folder train_A, the free-form surface elevation cloud map is read from the folder train_H, the meshing type text command is read from the folder train_C, and the free-form surface mesh structure is read from train_B; (3.2.2) Divide the data in step (3.2.1) into a training data set and a test data set according to the form of input and output data pairs; (3.2.3) Based on the variant of the generative adversarial network model GridGAN, its model structure is reconstructed so that the new neural network model can simultaneously integrate text and image information. The new neural network model integrating text and image information is named GridGAN-TXT. (3.2.4) training the neural network model GridGAN-TXT that integrates text and image information in step (3.2.3) by using the training set consisting of the building free-form surface curvature cloud map, the free-form surface elevation cloud map, the grid division type text instruction, and the building free-form surface grid structure in step (3.2.2); (3.2.5) The neural network model GridGAN-TXT that integrates text and image information trained in step (3.2.3) is tested and evaluated using the test set consisting of the building free-form surface curvature cloud map, free-form surface elevation cloud map, grid division type text instructions, and building free-form surface grid structure in step (3.2.2). The model that meets the requirements of computer vision evaluation indicators and free-form surface grid structure geometry evaluation indicators is used as the final neural network model GridGAN-TXT that integrates text and image information.

3. The neural network integrating text and image information and the architectural free-form surface meshing design method according to claim 2 is characterized in that: In step (3.2.3), based on the variant of the generative adversarial network model, GridGAN, its model structure is reconstructed so that the new neural network model can simultaneously integrate text and image information, which specifically includes the following steps: (4.1) Define a text module for processing the word vector of the grid partitioning type text instruction, that is, the word vector in step (3.1.4). The text module includes a GRU unit, a mask calculation unit, a conditional vector calculation unit, a potential vector calculation unit, a conditional vector normalization and expansion unit in sequence: (4.1.1) The GRU unit is used to encode word vectors. The GRU unit processes the content of the word vector from front to back and from back to front in order. Finally, the GRU unit embeds the word vector with an input dimension of 300 into a 512-dimensional hidden state. (4.1.2) The mask calculation unit calculates the mask for each time step to process text sequences of different lengths and ensure that the hidden state is updated only at time steps within the valid length range; (4.1.3) Conditional vector calculation unit, which combines the forward and backward hidden states to calculate the conditional vector; (4.1.4) Latent vector calculation unit. The latent vector calculation consists of a fully connected layer and an activation function. The input feature dimension and output feature dimension of the fully connected layer are 512 and 128 respectively. The activation function uses the LeakyReLU function with a negative slope of 0.2 and performs in-situ calculation. The conditional vector C is generated by reparameterization. current , calculate the conditional vector C current The mean and standard deviation of current and STD current ; (4.1.5) Conditional vector normalization and expansion unit, expansion conditional vector C current The dimension of the conditional vector C current The dimension is expanded from 2 to 4, and then the conditional vector C is calculated using formula (1). current Normalize to get C new , where M new and STD new The new mean and standard deviation of the conditional vector are maintained at C. new The first and second dimensions of remain unchanged, and the third and fourth dimensions are repeated to match the size of the specified image feature F. This process is expressed by formula (2), where C is the final conditional vector, repeat is a function in the Pytorch library for repeated tensor operations, F.size(2) represents the third dimension of F, and F.size(3) represents the fourth dimension of F; C=C new .repeat(1,1,F.size(2),F.size(3)) (2) (4.2) Add the text module in step (4.1) to the generator of GridGAN and perform the following operations: (4.2.1) Input the building free-form surface curvature cloud map and the free-form surface elevation cloud map into the convolutional neural network block in the generator, and obtain the image feature e by downsampling the image data. The image feature e is the specific value of the image feature F in the generator in step (4.1.5); (4.2.2) Input the word vector in step (3.1.4) into the text module in the generator to obtain the corresponding conditional vector C G ; (4.2.3) The image feature e in step (4.2.1) and the conditional vector C in step (4.2.2) are converted into G Concatenate along the channel dimension to achieve image features e and conditional vector C G The fusion of G1 ,In formula (3), torch.cat represents the set-join function in the Pytorch library; M G1 =torch.cat((e,C G ),1) (3) (4.2.4) The fused feature M G1 The residual block network in the input generator is used to extract features through the residual block. The further fused extracted features are named R. The further fused extracted features R are added to the image features e in step (4.2.1) at the element level, that is, the elements at the same position of R and e are added. This process can be expressed by formula (4), where T is the result after element-level addition; T=e+R (4) (4.2.5) The added feature T and conditional vector C are G It is spliced ​​again along the channel dimension and further fused, and the fused feature is named M G2 , the fusion feature M G2 Input the deconvolution network block in the generator to finally obtain the building free-form surface mesh structure that meets the requirements of the meshing type text instruction; M G2 =torch.cat((T,C G ),1) (5) (4.3) Add the text module in step (4.1) to the discriminator of GridGAN and perform the following operations: (4.3.1) The building free-form surface curvature cloud map, the free-form surface elevation cloud map and the building free-form surface mesh structure are spliced ​​in the channel dimension, and the spliced ​​image feature is represented as IMG, and the image feature IMG is the specific value of the image feature F in the discriminator in step (4.1.5). Here, the building free-form surface mesh structure is the building free-form surface mesh structure in the training set in step (3.2.2), which is called the true building free-form surface mesh structure, or the building free-form surface mesh structure generated by the generator in step (4.2.5), which is called the fake building free-form surface mesh structure; (4.3.2) Input the word vector in step (3.1.4) into the text module in the generator to obtain the corresponding conditional vector C D ; (4.3.3) The discriminator contains two sub-discriminators with exactly the same structure. The image feature IMG in step (4.3.1) and the conditional vector C in step (4.3.2) are converted into D Concatenate along the channel dimension to achieve image features IMG and conditional vector C D The fusion of D1 , the fused feature M D1 Pass in the first sub-discriminator and use average pooling nn.AvgPool2d to respectively pool the image feature IMG in step (4.3.1) and the conditional vector C in step (4.3.2) D Perform a downsampling, concatenate the two along the channel dimension, and represent the fused feature as M D2 , the fused feature M D2 The input is passed to the second sub-discriminator. The two sub-discriminators discriminate the current input from different dimensions. The output results are the judgment of the building free-form surface mesh structure and the discrimination of the mesh division type text instruction type. M D1 =torch.cat((IMG,C D ),1) (6)。 4. A neural network integrating text and image information and a method for designing architectural free-form surface meshes according to claim 3, characterized in that: In step (3.2.3), the loss function of the neural network model GridGAN-TXT that integrates text and image information is calculated as: (5.1) The loss function calculation of the neural network model GridGAN-TXT that integrates text and image information is performed in the process of alternating training of the discriminator and the generator. If the input of the discriminator in step (4.3.1) is the true building free-form surface grid structure, then the output of the discriminator in step (4.3.3) is P r and C r , if the input of the discriminator in step (4.3.1) contains a false building free-form surface mesh structure, then the output of the discriminator in step (4.3.3) is P f and C f , where P r and P f The judgment of the corresponding building free surface mesh results, C r and C f Identify the text instruction type corresponding to the mesh division type; (5.2) Generate adversarial loss calculation of the adversarial network, the loss function of the generator L G For formula (7), the loss function L of the discriminator is D It is formula (8), where MSELoss represents the mean square error loss function, True and False represent the true and false judgments respectively; L G_GAN =MSELoss(P f ,True) (7) L D_GAN =MSELoss(P r ,True)+MSELoss(P f ,False) (8) (5.3) Text loss L C Calculation is used to calculate the consistency of the basic unit of the grid, see formula (9), where NLLLoss represents the negative log-likelihood loss function, C real and C fake They represent the true and false instructions of the meshing type text instructions respectively; L C =NLLLoss(C r ,C real )+NLLLoss(C f ,C fake ) (9) (5.4) Feature matching loss L G_Feat Calculation, see formula (10), where L1Loss represents the L1 loss function, that is, the mean absolute error, i represents the i-th sub-discriminator, i = 1, 2, j represents the j-th layer feature, j = 1, 2, 3, 4, (P r ) i,j represents the i-th sub-discriminator pair P r The j-th layer feature is calculated, the same, (P f ) i,j represents the i-th sub-discriminator pair P f Calculate the j-th layer features; (5.5) Feature matching loss L based on VGG neural network G_VGG Calculation, see formula (11), where VGG represents the VGG19 neural network; L G_VGG =L1Loss{VGG(P r ),VGG(P f )} (11) (5.6) Text and image consistency loss L G_CCL Calculation is used to calculate the consistency between the generated building free-form surface mesh and the meshing type text instruction, see formula (12), where CCloss represents the text and image consistency loss function; L G_CCL =CCLoss(P r ,P f ,C real ) (12) (5.7) Combining the loss functions in steps (5.2) to (5.6) we get the total loss function L of the generator G And the total loss function L of the discriminator D , see formula (13) and formula (14) respectively, where α, β, γ and δ are the corresponding weight coefficients, and the generator and discriminator of the pre-trained model are obtained by L G and L D Continuously optimize model parameters; L G =L G_GAN +βL G_Feat +γL G_VGG +δL G_CCL (13) 5. A neural network integrating text and image information and its architectural free-form surface meshing design system, characterized in that: The system comprises: (1) A data acquisition module, which acquires the free-form surface of the building to be meshed and the text instructions of the required meshing type; (2) Data construction module, which generates free-form surface curvature cloud maps and free-form surface elevation cloud maps corresponding to the building free-form surface, and converts the grid division type text instructions into word vectors through word embedding; (3) A model building module, which inputs the obtained free-form surface curvature cloud map, free-form surface elevation cloud map, and meshing type instruction word vector into a pre-trained neural network model that integrates text and image information, and generates a building free-form surface mesh structure that meets the requirements of the meshing type instruction through the neural network model that integrates text and image information; (4) Scheme design module, which extracts the grid nodes and topological relationships in the architectural free-form surface grid structure generated by the neural network model that integrates text and image information, then projects the grid nodes into three-dimensional space and connects them using topological relationships to ultimately obtain the architectural free-form surface grid structure in space. In the data construction module, the grid division type text instructions are converted into word vectors through word embedding, including: (2.1) The meshing type text instruction corresponds to the basic unit type of the building free-form surface mesh. According to the generation requirements, the meshing type text instruction is rec or tri, which indicates that the basic unit type of the building free-form surface mesh to be generated is a quadrilateral mesh unit or a triangular mesh unit; (2.2) The word embedding model wiki.en.bin in the open source word embedding representation and sentence classification library FastText is used to convert the grid-type text instructions from text to word vectors. The data shape of the word vector is [1, N, 300], where 1 represents the number of text instructions, N represents the maximum word length, and 300 represents the dimension of the word vector generated by FastText. If the length N of the word vector is less than 50, the word vector is filled with the number 0 until the length is equal to 50. The final shape of the word vector is [1, 50, 300].

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the neural network integrating text and image information and the architectural free-form surface meshing design method thereof are implemented as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by the processor to implement the neural network integrating text and image information and the architectural free-form surface meshing design method as described in any one of claims 1 to 4.

8. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the program implements a neural network integrating text and image information and a method for designing architectural free-form surface meshes thereof as claimed in any one of claims 1 to 4.

Images