Dam twinborn scene modeling method based on man-machine coupling

Abstract

The invention discloses a dam twinborn scene modeling method based on man-machine coupling, and particularly relates to the technical field of generating images for texts in artificial intelligence, and the method comprises the steps: firstly, converting human voice into texts through a voice-to-text construction method, and then transmitting the texts into a model; a generator in the model can retain more critical text information through text image feature fusion and map semantics into an image area, a high-quality and vivid image can be generated, the generated image is matched with the whole sentence semantics, the good text image semantic consistency effect is achieved, and the user experience is improved. The discriminator can better discriminate the image area more related to the text information through the attention mechanism, the text image semantic consistency of the generated image is promoted, the stronger discriminator also promotes the generator to generate the higher-quality image in turn, and the authenticity of the semantic consistency of the generated image is promoted by introducing contrast learning.

Description

Technical Field

[0001] The present invention relates to the technical field of generating images for texts in artificial intelligence, and in particular to a dam twin scene modeling method based on human-machine coupling. Background Art

[0002] With the increasing global climate change and demand for water resources management, dams, as important water conservancy facilities, play a vital role. The design, construction, operation and maintenance of dams are directly related to human life safety, ecological environment and economic development. Therefore, how to effectively manage and optimize the operation of dams to ensure their long-term safety and sustainability has become one of the key issues in the current field of water conservancy engineering. Dam modeling requires not only accurate modeling of the physical environment, but also the combination of the professional knowledge and experience of operators. This requires the human-computer interaction mechanism to play a bridging role in the modeling process. Through human-computer coupling, the system can not only provide real-time feedback and adjustments based on the input of experts, but also gradually improve and optimize the dam model in multiple rounds of interaction, thereby realizing "people-oriented" dynamic modeling. By combining technologies such as speech recognition, natural language processing and intelligent image generation, the dam twin scenario modeling method based on human-computer coupling enables engineers to directly input, modify and update the dam model through voice or text, greatly improving the modeling efficiency and accuracy. This intelligent modeling method allows experts to quickly generate modeling scenarios that meet actual needs when faced with complex engineering environments, and adjust model parameters in real time according to changes to ensure the accuracy and flexibility of the model.

[0003] With the development of GAN, it has made great contributions in the field of text-generated images. Under the GAN architecture, text can be input to GAN and then an image corresponding to the text can be generated. And when GAN implements the task of generating images from text, the speed of generating images is very fast and has a smooth latent space, thus achieving more controllable synthesis. So GAN is also used to complete the task of generating images from text. The task of generating images from text is to generate images that are consistent with the text description based on the text information. In this study, both the semantic consistency of the text and the authenticity of the image must be guaranteed.

[0004] In recent years, there have been many advances in the field of text-generated image research, but there are still some difficulties to be solved: (1) First of all, it is impossible to align the key text information with the corresponding image area. (2) In the existing model, the discriminator can only extract the entire image features and then connect them with the text features to determine the semantic consistency of the text image, but it cannot determine which part of the local image features is more relevant to the text information. How to better align text features with image features is an unresolved problem. In the generation of the twin scene of the dam, special attention should be paid to the requirements of the text description, because the type of dam and the description of the surrounding natural environment will affect the generation of its twin scene. Summary of the invention

[0005] To this end, the present invention provides a dam twin scenario modeling method based on human-machine coupling to solve the problems raised in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a dam twin scene modeling method based on human-machine coupling, comprising the following steps:

[0007] S1: Collect dam modeling related information, including the dam modeling structure and graphic style, for establishing a text-image pair dataset;

[0008] S2: Combine the collected data with the scene description to form text, perform preprocessing and data enhancement on the text and images, build a text-image pair dataset, and divide the text-image pair dataset into a training set and a validation set in a ratio of 7:3;

[0009] S3: Construct speech-to-text method;

[0010] S4: Build a generator. The speech-converted text will be input into the generator to generate images.

[0011] S5: Construct a discriminator. The image generated by the generator is input into the discriminator for discrimination.

[0012] S6: Use the dataset to train the text-to-image model, calculate the adversarial loss, and optimize the parameters;

[0013] S7: Use the trained text-to-image model to convert the user's spoken instructions into text, and then input the text into the text-to-image model to generate modeling.

[0014] Preferably, the information related to dam modeling collected in S1 is as follows:

[0015] S11: The information collected for dam modeling includes the structural characteristics, functional characteristics and environmental factors of the dam; the structural characteristics include the type of dam, including concrete gravity dam, arch dam and earth-rock dam; the functional characteristics include the operating water level, minimum operating water level and flood control high water level of the dam; the purpose of the reservoir dam, including power generation, irrigation, flood control and drinking water supply; the environmental factors include the geographical location, the specific geographical location of the dam, including rivers, lakes and valleys; hydrological characteristics; vegetation, river banks and natural landscapes in the surrounding environment; weather and climate, local climate conditions, rainfall, temperature and other potential impacts;

[0016] S12: Use point cloud technology to capture the three-dimensional information of the dam obtained in S11. After generating a three-dimensional model, extract the two-dimensional information of the dam's key cross-sections, top views, and plane profiles to provide input for text-generated plane model.

[0017] Preferably, the specific contents of constructing text and preprocessing text and image in S2 are as follows:

[0018] S21: Construct text. The text should cover the geometric features, structural details, design parameters (height, width, material type) and environmental conditions (topography and climate around the dam) of the dam as much as possible, so that the image generation model can understand the complete scene. The same dam can be described in different ways, including technical parameter descriptions, scene descriptions and use cases, which increases the diversity of the training set and helps the model learn different styles of text expressions.

[0019] S22: Text preprocessing, removing useless symbols, emoticons, and special characters, generating different text expressions through synonym replacement, and enhancing the diversity of the dataset;

[0020] S23: Image preprocessing: resize all images to a uniform size of 256x256 pixels, and perform data augmentation on the images, including rotation, translation, and flipping.

[0021] Preferably, the speech-to-text conversion in S3 is configured as follows:

[0022] S31: Use Baidu Speech Recognition to implement online speech recognition and convert real-time speech into text;

[0023] S32: The context is updated through the recurrent neural network RNN ​​so that the system can understand and remember the task status and the user's historical input; at the same time, when the user proposes to modify the task, the system uses the recurrent neural network RNN ​​based on the historical context to understand the modified content and combine it with the current state of the task to generate new task content;

[0024] S33: The updated text content will be used by the system to generate new dam modeling images or models to reflect the user's latest requirements.

[0025] Preferably, in step S32, the context is updated by the recurrent neural network RNN ​​and the user proposes to modify the task as follows:

[0026] S321: Recurrent neural network RNN ​​through hidden state h t Gradually accumulate and store the user's historical input information. At time step t, the recurrent neural network RNN ​​receives the current input x t and the hidden state h at the previous time step t-1 , generate the current hidden state h t , the expression is:

[0027] h t =σ(W x x t +W h h t-1 + b);

[0028] h t Represents the hidden state at time step t, which contains the accumulated information of current input and historical input, W x is the weight matrix of the current input, W h is the weight matrix of the previous hidden state, b is the bias term, σ is the activation function tanh, which is used to introduce nonlinearity; the hidden state h t Contains all historical information, that is, each input is updated based on the integration of the previous hidden state and the new input. As the time step t increases, the recurrent neural network RNN ​​gradually accumulates the user's historical input to remember the previous input content;

[0029] S322: When the user proposes to modify the task content, the system generates updated task content based on the current context state and the new input; the expression is:

[0030] h t+1 =σ(W x x t+1 +W h h t + b);

[0031] In this formula, the hidden state h t+1 Combined with modified input x t+1 and the previous context h t Forming an updated context;

[0032] S323: In order to generate a task description that meets the user's latest needs, the system updates the hidden state h t+1 Generate new output st+1 :

[0033] s t+1 =g(W s h t+1 + b);

[0034] s t+1 Represents the task description text generated at the current time step, W s is the weight matrix from hidden state to output, b is the bias term, and g is the activation function sigmoid of the output layer; the output s t+1 It is the updated task content, generated based on the latest context and the user's latest input, reflecting the user's modification intention.

[0035] Preferably, the components and settings of the generator in step S4 are as follows:

[0036] S41: The text encoder adopts a bidirectional LSTM network structure;

[0037] S42: The text-image feature fusion block consists of LSTM, semantic mapping module and MLP;

[0038] S43: The generator loss consists of adversarial loss and DAMSM loss, expressed as:

[0039]

[0040] Where s is a text description, is the generated image, D() is the decision given by the discriminator whether the input image matches the input sentence, and λ DA is the weight of DAMSM loss; DAMSM loss is used to measure the semantic consistency between text and image.

[0041] Preferably, the components and settings of the text image feature fusion block in step S42 are as follows:

[0042] S421: In LSTM, first, LSTM obtains the initial hidden state of the recurrent neural network RNN ​​through the initialization of noise z, and then obtains a new hidden state through the subsequent input gate, forget gate and output gate to obtain more important text information, and can establish long-term dependencies between fusion blocks and reduce the difficulty of jump training;

[0043] S422: The semantic mapping module contains a structure in the order of upsample, conv, BN, Relu, conv, upsample, which is used to generate the semantic map p i ;

[0044] S423: MLP contains the structure in the order of Linear, Relu, and Linear;

[0045] S424: The final generated semantic graph map p i Added to the affine transformation.

[0046] Preferably, the LSTM in step S421 is set as follows:

[0047] S4211: Initialize LSTM using noise z; the expression is:

[0048] h0=MLP1(z),c0=MLP2(z);

[0049] h0 is the initial hidden state of the recurrent neural network RNN, and c0 is the initial cell state;

[0050] S4212:h t and γ i and β i The update rule of ; the expression is:

[0051]

[0052] c t =f t ☉c t-1 +i t ☉u t ;

[0053] h t =o t ☉tan c t ;

[0054] i t 、f t and O t They are input gate, forget gate and output gate respectively, s is the sentence vector, c t is the cell state, u t is a linear combination of the input to the hidden state at the current time step t, where h t is the hidden state of the recurrent neural network RNN, and σ is the Sigmoid activation function.

[0055] Preferably, the MLP in step S423 is set as follows:

[0056] S4231: Two MLPs predict the channel scaling parameter γ and shift parameter β under language conditions respectively, and the expressions are:

[0057] γ=MLP1(h t ),β=MLP2(h t );

[0058] S4232: First, use parameter γ to perform channel-direction scaling operation on x, and then use shift parameter β to perform channel-direction shift operation; the expression is:

[0059] Affine(x i |h t )=γ i *x i +β i ;

[0060] where x i is the i-th channel information of the visual feature map, γ i and β i are the scaling and shifting parameters of the i-th channel of the visual feature map.

[0061] Preferably, the semantic graph map p finally generated in step S424 i Added to the affine transformation, the expression is as follows:

[0062] S4241: The expression is:

[0063]

[0064] p i,(h,w) It is the semantic graph map generated in the semantic mapping module. Pi can be used as a weight to add semantic information to a more important position and determine how much text information to strengthen. t is the scaling parameter generated by the MLP, β t is the offset parameter generated by MLP, h t is the hidden state of the recurrent neural network RNN, x thw is the image feature, It is the image feature after affine transformation. t, h, and w represent the number of channels, height, and width of the image respectively.

[0065] Preferably, the discriminator in step S5 is composed and configured as follows:

[0066] S51: downsampling block; downsampling uses a convolutional layer with a step size of 2; the residual block consists of a 4x4 convolutional layer, a ReLU layer, a 3x3 convolutional layer, and a ReLU layer;

[0067] S52: Attention mechanism; First, the input image will be downsampled by the downsampling block of S51, and the image feature G will be extracted and connected with the sentence vector s copied from the space to be input into the attention mechanism; the attention mechanism is incorporated into the discriminator, and a stronger discriminator in turn promotes the generator to generate images with stronger semantic consistency of the text, so as to promote the semantic consistency of the generated image and the text;

[0068] S53: Cross-modal alignment: In the task of generating images from text, cross-modal alignment is particularly important. By performing cross-modal training on the sentence vector s and the image, as well as the generated fake image and the real image, and introducing contrast loss to align the semantics and image of the common space, the generated image has higher semantic consistency and realism.

[0069] S54: The discriminator loss uses the matching-aware gradient penalty MA-GP loss related adversarial loss:

[0070]

[0071] Among them, s is the sentence vector, is the sentence vector of the unmatched text, x is the real image corresponding to s, is the generated image, D() is the decision given by the discriminator whether the input image matches the input sentence, and λ MA and p are the hyperparameters of MA-GP; E x ~pdata is the mathematical expectation.

[0072] Preferably, the attention mechanism in step S52 is composed and configured as follows:

[0073] S521: The attention mechanism generates an attention map of the sentence vector that suppresses the irrelevant parts of the image and text; the expression is:

[0074] v w,h =MLP(G w,h ,s);

[0075]

[0076] S w,h =s×θ w,h ;

[0077] S w,h is the feature channel of the sentence vector s at position {w,h}; first extract the image feature G w,h Together with s, it is fed into the MLP to generate the energy value v w,h , and then use the energy value v w,h Calculate the attention weight θ w,h , and finally θ w,h Combined with the sentence vector s, we get S w,h , the sentence vector s is connected with G and then extracted through a downsampling block to extract image features, which are then combined with the copied sentence vector s to calculate the adversarial loss to evaluate the authenticity of the generated image and the consistency of the text image; the discriminator strengthened by the attention mechanism can better determine which part of the local image features is more relevant to the text information, and a stronger discriminator can in turn promote a stronger generator.

[0078] Preferably, the cross-modal alignment in step S53 is composed and configured as follows:

[0079] S531: Establish cross-modal alignment at the image-image and image-text levels; the expression is:

[0080]

[0081] cos(·,·) cosine similarity is used as a distance metric, u: represents the embedding vector of the image; v represents the embedding vector of the text or the embedding vector of another image, and T is the transpose in the matrix operation;

[0082] The contrast loss function is:

[0083]

[0084] m is the mini-batch size of the input samples, u i is the i-th sample in the mini-batch u, v i refers to the i-th sample in the mini-batch v, v j refers to the jth sample of sample v, η is the temperature hyperparameter, exp is the exponential function, D img (·) is the extraction process of image embedding in the discriminator and projects the input image into the image embedding, e s is the sentence vector; x f is the generated fake image, and x is the real image; is the loss function between the sentence and the real image, is the loss function between the sentence and the generated fake graph; It is the loss function between the real image and the fake image;

[0085] S532: The cross-modal alignment objective function expression is:

[0086]

[0087] is the contrastive loss of the generator, is the contrastive loss of the discriminator.

[0088] Preferably, the training model in step S6 specifically includes the following steps:

[0089] S61: Text description input text encoder generates sentence vectors and word features;

[0090] S62: Input the noise vector sampled from the normal distribution into the fully connected layer and reshape it into the required size, and then input it together with the sentence vector into 7 text-image feature fusion blocks to generate an image;

[0091] S63: In the discriminator, the generated image is further discriminated by the downsampling block and the attention mechanism to calculate the adversarial loss, and the extracted image features are aligned across modalities;

[0092] S64: The final overall objective function is obtained by weighted combination of adversarial loss and contrast loss. λ1 and λ2 are coefficient weights. The overall objective function is:

[0093]

[0094] The present invention has the following advantages:

[0095] 1. The present invention can generate twin scenes of various dams, and can generate twin scenes of dams that meet the requirements through human oral description, which has a good reference value for dam modeling.

[0096] 2. The model designed by the present invention is a single-stage end-to-end GAN model. After realizing human-computer interaction through speech-to-text, the model can understand complex scenes and key semantics in texts, and can also generate natural and realistic twin scenes for cluttered backgrounds; the generator can retain more critical text information and map semantics to image areas through text-image feature fusion, and can generate high-quality realistic images. The generated images match the overall sentence semantics, achieving a good text-image semantic consistency effect. The discriminator can better identify image areas that are more relevant to text information through the attention mechanism, which promotes the text-image semantic consistency of the generated images. A stronger discriminator in turn promotes the generator to generate higher quality images, and promotes the authenticity of the semantic consistency of the generated images by introducing contrastive learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is a schematic diagram of the process of the present invention.

[0098] Figure 2 A schematic diagram of the text generation image model structure of the present invention.

[0099] Figure 3 Schematic diagram of the text image feature fusion block structure.

[0100] Figure 4 Schematic diagram of the training process for the text-to-image model. DETAILED DESCRIPTION

[0101] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0102] like Figure 1-Figure 4 As shown, the present invention provides a dam twin scene modeling method based on human-machine coupling, which includes the following steps:

[0103] S1: Collect dam modeling related information, including the dam modeling structure and graphic style, for establishing a text-image pair dataset;

[0104] The relevant information collected for dam modeling is:

[0105] S11: The information collected for dam modeling includes the structural characteristics, functional characteristics and environmental factors of the dam; the structural characteristics include the types of dams, including concrete gravity dams, arch dams and earth-rock dams; the functional characteristics include: the operating water level, minimum operating water level and high flood control water level of the dam; the purpose of the reservoir dam, including power generation, irrigation, flood control and drinking water supply; the environmental factors include: geographical location, the specific geographical location of the dam, including rivers, lakes and valleys; hydrological characteristics; vegetation, river banks and natural landscapes in the surrounding environment; weather and climate, local climate conditions, rainfall, temperature and other potential impacts;

[0106] S12: The three-dimensional information of the dam is captured using point cloud technology. After generating a three-dimensional model, key two-dimensional information such as the dam’s key cross-sections, top views, and plane contours are extracted to provide rich input for text-generated plane model.

[0107] S2: Combine the collected data with the scene description to form text, perform preprocessing and data enhancement on the text and images, build a text-image pair dataset, and divide the dataset into a training set and a validation set in a ratio of 7:3;

[0108] The specific contents of constructing text and preprocessing text and images are as follows:

[0109] S21: Construct text. The text should cover the geometric features, structural details, design parameters (height, width, material type) and environmental conditions (topography and climate around the dam) of the dam as much as possible, so that the image generation model can understand the complete scene. The same dam can be described in different ways, including technical parameter descriptions, scene descriptions and use cases, which increases the diversity of the training set and helps the model learn different styles of text expressions.

[0110] S22: Text preprocessing, removing useless symbols, emoticons, and special characters, generating different text expressions through synonym replacement, and enhancing the diversity of the dataset;

[0111] S23: Image preprocessing: resize all images to a uniform size of 256x256 pixels, and perform data augmentation (rotation, translation, flipping) on ​​the images.

[0112] S3: Construct speech-to-text method;

[0113] The speech-to-text method consists of the following components and settings:

[0114] S31: Use Baidu Speech Recognition to implement online speech recognition and convert real-time speech into text;

[0115] S32: The context is updated through a recurrent neural network (RNN) so that the system can understand and remember the task status and the user's historical input; when the user proposes to modify the task, the system will use the RNN network structure to understand the modified content based on the historical context, and combine it with the current state of the task to generate new task content;

[0116] S321: RNN through hidden state h t Gradually accumulate and store the user's historical input information. At time step t, the RNN receives the current input x t and the hidden state h at the previous time step t-1 , generate the current hidden state h t , the expression is:

[0117] h t =σ(W x x t +W h h t-1 + b);

[0118] h t Represents the hidden state at time step t, which contains the accumulated information of current input and historical input, W x is the weight matrix of the current input, W h is the weight matrix of the previous hidden state, b is the bias term, σ is the activation function tanh, which is used to introduce nonlinearity; the hidden state h t It contains all historical information, that is, each input is updated based on the integration of the previous hidden state and the new input. In this way, as the time step increases, the RNN will gradually accumulate the user's historical input and "remember" the previous input content;

[0119] S322: When the user proposes to modify the task content, the system generates updated task content based on the current context state and the new input; the expression is:

[0120] h t+1 =σ(W x x t+1 +W h h t + b);

[0121] In this formula, the hidden state h t+1 Combined with the modified input x t+1 and the previous context h t An updated context is formed;

[0122] S323: In order to generate a task description that meets the user's latest needs, the system will update the hidden state h t+1 Generate new output s t+1 :

[0123] s t+1 =g(W s h t+1 + b);

[0124] s t+1 Represents the task description text generated at the current time step, W s is the weight matrix from hidden state to output, b is the bias term, and g is the activation function sigmoid of the output layer; the output s t+1 It is the updated task content, generated based on the latest context and the user's latest input, reflecting the user's modification intention.

[0125] S33: The updated text content will be used by the system to generate new dam modeling images or models, reflecting the user's latest requirements.

[0126] S4: Build generator;

[0127] The generator composition and settings are as follows:

[0128] S41: The text encoder adopts a bidirectional LSTM network structure;

[0129] S42: The text-image feature fusion block consists of LSTM, semantic mapping module and MLP;

[0130] The components and settings of the text image feature fusion block are as follows:

[0131] S421: In LSTM, first, LSTM is initialized by noise z, and this embodiment obtains the initial hidden state of RNN, and then obtains a new hidden state through subsequent input gates, forget gates, and output gates. This not only obtains more important text information, but also establishes long-term dependencies between fusion blocks and reduces the difficulty of jump training;

[0132] The LSTM settings are as follows:

[0133] S4211: Initialize LSTM using noise z; the expression is:

[0134] h0=MLP1(z),c0=MLP2(z);

[0135] h0 is the initial hidden state of the recurrent neural network RNN, and c0 is the initial cell state;

[0136] S4212:h t and γ i and β i The update rule of ; the expression is:

[0137]

[0138] c t =f t ☉c t-1 +i t ☉u t ;

[0139] h t =o t ☉tan c t ;

[0140] i t 、f t and t They are input gate, forget gate and output gate respectively, s is the sentence vector, c t is the cell state, u t is a linear combination of the input to the hidden state at the current time step t, where h t is the RNN hidden state, and σ is the Sigmoid activation function.

[0141] S422: The semantic mapping module contains a structure in the order of upsample, conv, BN, Relu, conv, upsample, which is used to generate the semantic map p i ;

[0142] S423: MLP contains the structure in the order of Linear, Relu, and Linear;

[0143] The MLP setup is as follows:

[0144] S4231: Two MLPs predict the channel scaling parameter γ and shift parameter β under language conditions respectively, and the expressions are:

[0145] γ=MLP1(h t ),β=MLP2(h t );

[0146] S4232: First, use parameter γ to perform channel-direction scaling operation on x, and then use shift parameter β to perform channel-direction shift operation; the expression is:

[0147] Affine(x i |h t )=γ i *x i +β i ;

[0148] where x i is the i-th channel information of the visual feature map, γ i and β i are the scaling and shifting parameters of the i-th channel of the visual feature map.

[0149] S424: The final generated semantic graph map p i Added to the affine transformation.

[0150] S4241: The expression is:

[0151]

[0152] p i,(h,w) is the semantic graph map generated in the semantic mapping module. Pi, as a weight, can add semantic information to a more important position and determine how much text information to strengthen; γ t is the scaling parameter generated by the MLP, β t is the offset parameter generated by MLP, h t is the hidden state of the recurrent neural network RNN, x thw is the image feature, It is the image feature after affine transformation; t, h, and w represent the number of channels, height, and width of the image respectively.

[0153] S43: The generator loss consists of adversarial loss and DAMSM loss, expressed as:

[0154]

[0155] Where s is a text description, is the generated image, D() is the decision given by the discriminator whether the input image matches the input sentence, and λDA is the weight of DAMSM loss; DAMSM loss is used to measure the semantic consistency between text and image.

[0156] S5: Build the discriminator;

[0157] S51: downsampling block; downsampling uses a convolutional layer with a step size of 2; the residual block consists of a 4x4 convolutional layer, a ReLU layer, a 3x3 convolutional layer, and a ReLU layer;

[0158] S52: Attention mechanism; first, the input image is downsampled by a series of downsampling blocks, and the image feature G is extracted and connected with the spatially copied sentence vector s to be input into the attention mechanism; in order to promote the semantic consistency between the generated image and the text, this embodiment incorporates the attention mechanism into the discriminator, so that a stronger discriminator will in turn promote the generator to generate images with stronger semantic consistency of the text;

[0159] S521: The attention mechanism generates an attention map of the sentence vector that suppresses the irrelevant parts of the image and text; the expression is:

[0160] v w,h =MLP(G w,h ,s);

[0161]

[0162] S w,h =s×θ w,h ;

[0163] S w,h is the feature channel of the sentence vector s at position {w,h}; first extract the image feature G w,h Together with s, it is fed into the MLP to generate the energy value v w,h , and then use the energy value v w,h Calculate the attention weight θ w,h , and finally θ w,h Combined with the sentence vector s to get S w,h , the sentence vector s is connected with G and then extracted through a downsampling block to extract image features, which are then combined with the copied sentence vector s to calculate the adversarial loss to evaluate the authenticity of the generated image and the consistency of the text image; the discriminator strengthened by the attention mechanism can better determine which part of the local image features is more relevant to the text information, and a stronger discriminator can in turn promote a stronger generator.

[0164] S53: cross-modal alignment; in the task of generating images from text, cross-modal alignment is particularly important. In order to make the generated images have higher semantic consistency and realism, this embodiment performs cross-modal training in the sentence vector s and the image, as well as the generated fake image and the real image; this embodiment introduces contrast loss to align the semantics and image of the common space;

[0165] S531: This embodiment establishes cross-modal alignment at the image-image and image-text levels; the expression is:

[0166]

[0167] cos(·,·) cosine similarity is used as a distance metric, u: represents the embedding vector of the image; v represents the embedding vector of the text or the embedding vector of another image, and T is the transpose in the matrix operation;

[0168] The contrast loss function is:

[0169]

[0170] m is the mini-batch size of the input samples, u i is the i-th sample in the mini-batch u, v i refers to the i-th sample of the small batch sample v, v j refers to the jth sample of sample v, η is the temperature hyperparameter, exp is the exponential function, D img (·) is the extraction process of image embedding in the discriminator and projects the input image into the image embedding, e s is the sentence vector; x f is the generated fake image, and x is the real image; is the loss function between the sentence and the real image, is the loss function between the sentence and the generated fake graph; It is the loss function between the real image and the fake image;

[0171] S532: The cross-modal alignment objective function expression is:

[0172]

[0173] is the contrastive loss of the generator, is the contrastive loss of the discriminator;

[0174] S54: The discriminator loss uses an adversarial loss related to the matching-aware gradient penalty (MA-GP) loss:

[0175]

[0176] Where s is the sentence vector, is the sentence vector of the unmatched text, x is the real image corresponding to s, is the generated image, D() is the decision given by the discriminator whether the input image matches the input sentence, and λ MA and p are the hyperparameters of MA-GP; E x ~pdata is the mathematical expectation.

[0177] S6: Use the dataset to train the text-to-image model, calculate the adversarial loss, and optimize the parameters;

[0178] S61: Text description input text encoder generates sentence vectors and word features;

[0179] S62: Input the noise vector sampled from the normal distribution into the fully connected layer and reshape it into the required size, and then input it together with the sentence vector into 7 text-image feature fusion blocks to generate an image;

[0180] S63: In the discriminator, the generated image is further discriminated by the downsampling block and the attention mechanism to calculate the adversarial loss, and the extracted image features are aligned across modalities;

[0181] S64: The final overall objective function is obtained by weighted combination of adversarial loss and contrast loss. λ1 and λ2 are coefficient weights. The overall objective function is:

[0182]

[0183] S7: Use the trained text-to-image model to convert the user's spoken instructions into text, and then input the text into the text-to-image model to generate modeling.

[0184] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A dam twin scenario modeling method based on human-machine coupling, comprising the following steps: S1: Collect dam modeling related information, including the dam modeling structure and graphic style, for establishing a text-image pair dataset; S2: The collected data is combined with the scene description to form text, and the text and image are preprocessed and data enhanced to construct a text-image pair dataset. The text-image pair dataset is divided into a training set and a validation set in a ratio of 7:

3. It is characterized by: S3: Construct speech-to-text method; S4: Build a generator. The speech-converted text will be input into the generator to generate images. S5: Construct a discriminator. The image generated by the generator is input into the discriminator for discrimination. The discriminator composition and settings are as follows: S51: downsampling block; downsampling uses a convolutional layer with a step size of 2; the residual block consists of a 4x4 convolutional layer, a ReLU layer, a 3x3 convolutional layer, and a ReLU layer; S52: Attention mechanism; First, the input image is downsampled by the downsampling block of S51, and the image feature G is extracted and connected with the spatially copied sentence vector s to be input into the attention mechanism; the attention mechanism is incorporated into the discriminator, and a stronger discriminator in turn promotes the generator to generate images with stronger semantic consistency of the text, so as to promote the semantic consistency between the generated image and the text; The composition and settings of the attention mechanism are as follows: S521: The attention mechanism generates an attention map of the sentence vector that suppresses the irrelevant parts of the image and text; the expression is: v w,h =MLP(G w,h ,s); S w,h =s×θ w,h ; S w,h It is the feature channel of the sentence vector s at position {w,h}; First, extract the image feature G w,h Together with the sentence vector s, it is fed into the MLP to generate the energy value v w,h , and then use the energy value v w,h Calculate the attention weight θ w,h , and finally θ w,h Combined with the sentence vector s, we get S w,h ; The sentence vector s is concatenated with G and then passed through a downsampling block to extract image features, which are then combined with the copied sentence vector s to calculate the adversarial loss to evaluate the authenticity of the generated image and the consistency of the text image. The discriminator strengthened by the attention mechanism can better determine which local image features are more relevant to the text information, and a stronger discriminator can in turn promote a stronger generator. S53: Cross-modal alignment: By performing cross-modal training on the sentence vector s and the image as well as the generated fake image and the real image, and introducing contrastive loss to align the semantics and image of the common space, the generated image has higher semantic consistency and realism; The cross-modal alignment composition and settings are as follows: S531: Establish cross-modal alignment at the image-image and image-text levels; the expression is: cos(·,·) cosine similarity is used as a distance metric, u: represents the embedding vector of the image; v represents the embedding vector of the text or the embedding vector of another image, and T is the transpose in the matrix operation; The contrast loss function is: m is the size of the mini-batch u of input samples; u i is the i-th sample in the mini-batch u; v i refers to the i-th sample in the mini-batch v; v j refers to the jth sample of sample v; η is the temperature hyperparameter; exp is the exponential function; D img (·) is the extraction process of image embedding in the discriminator and projects the input image into the image embedding; e s is the sentence vector; x f is a generated fake image; x is a real image; is the loss function between the sentence and the real image; is the loss function between the sentence and the generated fake graph; It is the loss function between the real image and the fake image; S532: The cross-modal alignment objective function expression is: is the contrastive loss of the generator, is the contrastive loss of the discriminator; S54: The discriminator loss uses the matching-aware gradient penalty MA-GP loss related adversarial loss: Among them, s is the sentence vector, is the sentence vector of the unmatched text, x is the real image corresponding to s, is the generated image, D() is the decision given by the discriminator whether the input image matches the input sentence, and λ MA and p are the hyperparameters of MA-GP; E x ~pdata is the mathematical expectation; S6: Use the dataset to train the text-to-image model, calculate the adversarial loss, and optimize the parameters; S7: Use the trained text-to-image model to convert the user's spoken instructions into text, and then input the text into the text-to-image model to generate modeling.

2. According to claim 1, a dam twin scenario modeling method based on human-machine coupling is characterized by: The speech-to-text settings in S3 are as follows: S31: Use Baidu Speech Recognition to implement online speech recognition and convert real-time speech into text; S32: The context is updated through the recurrent neural network RNN ​​so that the system can understand and remember the task status and the user's historical input; at the same time, when the user proposes to modify the task, the system uses the recurrent neural network RNN ​​based on the historical context to understand the modified content and combine it with the current state of the task to generate new task content; S33: The updated text content will be used by the system in S6 to generate a new dam modeling image or model to reflect the user's latest requirements.

3. The dam twin scenario modeling method based on human-machine coupling according to claim 2 is characterized in that: In step S32, the context is updated by the recurrent neural network RNN ​​and the user proposes to modify the task as follows: S321: Recurrent neural network RNN ​​through hidden state h t Gradually accumulate and store the user's historical input information. At time step t, the recurrent neural network RNN ​​receives the current input x t and the hidden state h at the previous time step t-1 , generate the current hidden state h t , the expression is: h t =σ(W x x t +W h h t-1 +b); h t Represents the hidden state at time step t, which contains the accumulated information of current input and historical input, W x is the weight matrix of the current input, W h is the weight matrix of the previous hidden state, b is the bias term, σ is the activation function tanh, which is used to introduce nonlinearity; the hidden state h t Contains all historical information, that is, each input is updated based on the integration of the previous hidden state and the new input. As the time step t increases, the recurrent neural network RNN ​​gradually accumulates the user's historical input to remember the previous input content; S322: When the user proposes to modify the task content, the system generates updated task content based on the current context state and combined with the new input; The expression is: h t+1 =σ(W x x t+1 +W h h t +b); In this formula, the hidden state h t+1 Combined with modified input x t+1 and the previous context h t Forming an updated context; S323: In order to generate a task description that meets the user's latest needs, the system updates the hidden state h t+1 Generate new output s t+1 : s t+1 =g(W s h t+1 +b); s t+1 Represents the task description text generated at the current time step, W s is the weight matrix from hidden state to output, b is the bias term, and g is the activation function sigmoid of the output layer; the output s t+1 It is the updated task content, generated based on the latest context and the user's latest input, reflecting the user's modification intention.

4. The dam twin scenario modeling method based on human-machine coupling according to claim 1 is characterized by: The components and settings of the generator in step S4 are as follows: S41: The text encoder adopts a bidirectional LSTM network structure; S42: The text-image feature fusion block consists of LSTM, semantic mapping module and MLP; S43: The generator loss consists of adversarial loss and DAMSM loss, expressed as: Where s is a text description, is the generated image, D() is the decision given by the discriminator whether the input image matches the input sentence, and λ DA is the weight of DAMSM loss; DAMSM loss is used to measure the semantic consistency between text and image.

5. The dam twin scenario modeling method based on human-machine coupling according to claim 4 is characterized in that: The components and settings of the text image feature fusion block in step S42 are as follows: S421: In LSTM, first, LSTM obtains the initial hidden state of the recurrent neural network RNN ​​through the initialization of noise z, and then obtains a new hidden state through the subsequent input gate, forget gate and output gate to obtain more important text information, and can establish long-term dependencies between fusion blocks and reduce the difficulty of jump training; S422: The semantic mapping module contains a structure in the order of upsample, conv, BN, Relu, conv, upsample, which is used to generate the semantic map p i ; S423: MLP contains the structure in the order of Linear, Relu, and Linear; S424: The final generated semantic graph map p i Added to the affine transformation.

6. The dam twin scenario modeling method based on human-machine coupling according to claim 5 is characterized by: In step S421, the LSTM is set as follows: S4211: Initialize LSTM using noise z; the expression is: h0=MLP1(z),c0=MLP2(z); h0 is the initial hidden state of the recurrent neural network RNN, and c0 is the initial cell state; S4212:h t and γ i and β i The update rule of ; the expression is: c t =f t ☉c t-1 +i t ☉u t ; h t =o t ☉tan c t ; i t 、f t and t They are input gate, forget gate and output gate respectively, s is the sentence vector, c t is the cell state, u t is a linear combination of the input to the hidden state at the current time step t, where h t is the hidden state of the recurrent neural network RNN, and σ is the Sigmoid activation function.

7. The dam twin scenario modeling method based on human-machine coupling according to claim 5 is characterized by: In step S423, the MLP is set as follows: S4231: Two MLPs predict the channel scaling parameter γ and shift parameter β under language conditions respectively, and the expressions are: γ=MLP1(h t ),β=MLP2(h t ); S4232: First, use parameter γ to perform channel-direction scaling operation on x, and then use shift parameter β to perform channel-direction shift operation; the expression is: Affine(x i |h t )=γ i *x i +b i ; where x i is the i-th channel information of the visual feature map, γ i and β i are the scaling and shifting parameters of the i-th channel of the visual feature map.

8. The dam twin scenario modeling method based on human-machine coupling according to claim 5 is characterized by: In step S424, the semantic graph map p is finally generated. i The expression added to the affine transformation is as follows: S4241: The expression is: p i,(h,w) is the semantic graph map generated in the semantic mapping module. Pi, as a weight, can add semantic information to a more important position and determine how much text information to strengthen; γ t is the scaling parameter generated by the MLP, β t is the offset parameter generated by MLP, h t is the hidden state of the recurrent neural network RNN, x thw is the image feature, It is the image feature after affine transformation; t, h, and w represent the number of channels, height, and width of the image, respectively.

9. The dam twin scenario modeling method based on human-machine coupling according to claim 1 is characterized in that: The training model in step S6 specifically includes the following steps: S61: Text description input text encoder generates sentence vectors and word features; S62: Input the noise vector sampled from the normal distribution into the fully connected layer and reshape it into the required size, and then input it together with the sentence vector into 7 text-image feature fusion blocks to generate an image; S63: In the discriminator, the generated image is further discriminated by the downsampling block and the attention mechanism to calculate the adversarial loss, and the extracted image features are aligned across modalities; S64: Finally, the overall objective function is obtained by weighted combination of adversarial loss and contrast loss, λ1 and λ2 are coefficient weights, and the overall objective function is:

Images