Ship appearance and interior concept graph design method based on text and graph relation

Abstract

The invention discloses a ship appearance and interior concept graph design method based on a text and graph relationship. The method comprises the following steps: S1, performing entity recognition and relationship extraction on ship appearance and interior images; s2, based on a text graph modeling neural network architecture, splicing and fusing the image feature representation and the text feature representation corresponding to the mark to generate a fused feature representation; and S3, fusing the self-learning potential space representation of the variational auto-encoder and the image generation capability of the generative adversarial network, optimizing the parameters of the two through joint training, and generating a ship appearance and interior concept design drawing based on regular transformation of a recursive model or differentiable generation.

Description

Technical Field

[0001] The present invention relates to the field of ship transportation technology, and in particular to a method for designing a conceptual drawing of a ship's appearance and interior based on a text-drawing relationship. Background Art

[0002] The design method of the ship's exterior concept map emphasizes function-orientedness, and determines its appearance by analyzing the purpose of the ship. For example, cargo ships focus on practicality and cargo capacity, while cruise ships focus on comfort and beauty. The modular method or similar rectangle method is used in the design to establish a harmonious proportional relationship between the various parts to ensure the beauty of the overall shape. The ship's interior concept map design starts with demand analysis and functional layout, fully communicates with the ship owner to understand the interior needs, and reasonably plans the location and area of ​​the cabin. Style positioning and color matching are the key to shaping the interior atmosphere. Material selection and texture expression are related to the quality and comfort of the interior. It is necessary to select appropriate materials for texture matching according to different functional areas.

[0003] The existing ship exterior and interior concept design methods still have the following problems: (1) There is a large deviation between demand communication and understanding. Although the design requirements are fully communicated and understood with the shipowner, the shipowner's requirements may be vague or unprofessional, resulting in deviations in the designer's understanding and transformation of the design requirements. There is a gap between the design plan and the shipowner's expectations. In addition, the communication cost is high and the process is cumbersome, which affects the design progress.

[0004] (2) Low design efficiency and visualization. During the design process, especially in the initial design stage, designers need to spend a lot of time building and modifying models, resulting in low design efficiency. In addition, for complex ship exterior and interior designs, existing modeling software may have certain limitations in terms of visualization and detail expression, and cannot fully display the details of the ship's exterior and interior effects, affecting the accuracy of design decisions, and requiring a large number of adjustments and modifications, increasing costs.

[0005] (3) Style positioning and material selection are highly subjective. Style positioning, color matching, material selection and texture expression rely on the designer's personal experience and subjective judgment, and lack more objective and scientific evaluation standards and decision-making basis. As a result, the interior styles designed by different designers vary greatly, and it is difficult to ensure that the selected styles and materials are universally applicable to different ship usage scenarios and user groups. Summary of the invention

[0006] The purpose of the present invention is to provide a method for designing a ship exterior and interior concept drawing based on the relationship between text and drawing to solve the above problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions: A method for designing a conceptual drawing of a ship's exterior and interior based on a text-drawing relationship comprises the following steps: S1. Perform entity recognition and relationship extraction on the images of the ship's exterior and interior, specifically: S11, extracting features of the ship appearance and interior images based on the CNN convolutional neural network, obtaining image feature representations of the ship appearance and interior images after multiple convolutional layers and pooling layers, and mapping the text to the appropriate ship appearance and interior vector space through the embedding layer; S12, processing text sequences and relation extraction based on RNN recurrent neural network, obtaining text feature representation by performing aggregation operation on the last hidden state or multiple hidden states, and inputting the text feature representation corresponding to the entity into the relation classifier to obtain the relation category probability distribution; S13, based on the BERT deep learning model, input the token sequence of the text sequence after tokenization, add special tags, and output the text feature representation of each token through the BERT deep learning model; S2, based on the text graph modeling neural network architecture, concatenate and fuse the image feature representation with the text feature representation corresponding to the tag to generate a fused feature representation; S3. Integrate the latent space representation of the variational autoencoder and the image generation capability of the generative adversarial network, optimize the parameters of the two through joint training, and generate conceptual design drawings of the ship's exterior and interior based on the transformation or differentiability of the rules of the recursive model.

[0008] Preferably, the method for extracting the features of the ship appearance and interior image in step S11 is as follows: assuming that the input ship appearance and interior image is I, and its dimension is , where H represents the image height, W represents the image width, and C represents the image channel; the convolution kernel is K, and its size is ,in, and are the sizes of the convolution kernel in height and width respectively; The calculation formula for the convolution operation is: ,in, is the value of the convolution result at position (i, j), and b is the bias term; After multiple convolutional layers and pooling layers of the CNN convolutional neural network, the feature representation of the ship's appearance and interior images is obtained. .

[0009] Preferably, the text mapping method in step S11 is specifically as follows: the text is represented as a word vector sequence {x1, x2, ..., xT}, where T is the length of the text sequence, each is a word vector with dimension d, which maps the text into the appropriate ship appearance and interior vector space through the embedding layer.

[0010] Preferably, step S12 specifically includes: processing text sequence and relation extraction based on RNN recurrent neural network, and the RNN recurrent neural network hidden state update formula is: , where h t is the hidden state at time t, is the weight matrix input to the hidden layer, is the weight matrix from hidden layer to hidden layer, is the bias term of the hidden layer; After processing the entire text sequence, the final feature representation of the text is obtained by performing aggregation operations on the last hidden state or multiple hidden states. ; The image features and text features After concatenation and fusion, fusion features are generated, and the text feature representation corresponding to the entity is input into the relationship classifier to calculate the probability distribution of the relationship category; Assume that the classification function used for entity recognition is , where y represents the entity category label, Fcombined is the fusion feature, and the softmax function is used to construct the probability distribution: ,in, is the score of the corresponding category k, calculated by the fully connected layer, C is the total number of entity categories, s j is the score corresponding to category j.

[0011] Preferably, step S13 specifically includes: based on the BERT deep learning model, inputting the token sequence of the text sequence after tokenization, and adding special tags, specifically: The input is represented as , where N is the number of input tokens, and each e i Contains word embedding and position embedding information; The BERT deep learning model calculates the self-attention of a single head as follows: ,in , W is the corresponding weight matrix, which is used to map the input E to the query Q, key K and value V respectively, and dk is the dimension of the key vector;

[0012] Through the multi-head self-attention of the BERT deep learning model, multiple single heads are spliced ​​together: , where h is the number of heads, It is the output linear transformation weight matrix, which outputs the feature representation of each token after passing through multiple multi-head self-attention layers and the intermediate feedforward neural network layer of the BERT deep learning model.

[0013] Preferably, the process of splicing and fusing the image feature representation and the text feature representation corresponding to the mark in step S2 is specifically as follows: assuming that the fused feature representation is , the image features The dimension is and text features , the dimension is To perform the fusion: ,in, is the weight matrix of image features, with dimension , is the weight matrix of text features, with dimension , The fused bias term, is the expected dimension of the fused features.

[0014] Preferably, the variational autoencoder in step S3 includes an encoder and a decoder. For the input ship appearance and interior image I, the encoder maps it to a latent variable space, assuming that the latent variable z obeys a Gaussian distribution, and the encoder outputs a mean μ and a variance σ 2 , and its calculation formula is: , the latent variable z is sampled from the distribution through the reparameterization technique: , ε is a random variable that follows the standard normal distribution N(0,1); the decoder maps the latent variable z back to the image space, trying to reconstruct the input image and reconstruct the image The probability distribution of is: , the goal of the variational autoencoder is to maximize the variational lower bound of the likelihood function, and its mathematical expression is: ,in, is the posterior distribution of the latent variables learned by the encoder, is the generated distribution defined by the decoder, represents the KL divergence, which is used to measure the difference between two distributions. and are the parameters of the encoder and decoder respectively.

[0015] Preferably, the generative adversarial network in step S3 includes a generator and a discriminator. The goal of the generator is to generate an image as realistic as possible, map the random noise vector z to the image space, and generate an image = G(z), the task of the discriminator is to distinguish between the real image I and the image generated by the generator , output a probability value indicating the probability that the input image is a real image, namely D(I) and D( ), the objective function of the generated adversarial network is expressed as: , where p data(I) is the distribution of real images, and Pz(z) is the distribution of noise vectors.

[0016] Preferably, step S3 is specifically: S31. Fusion of the latent space representation learned by the variational autoencoder and the image generation capability of the generative adversarial network, and optimization of the parameters of the two through joint training to generate conceptual design drawings of the ship's exterior and interior; S32. Use the recursive model to gradually construct the structure of the graph or process the sequence representation of the graph, further learn the distribution law of the graph, and combine it with rule-based transformation to generate a new and meaningful graph structure while maintaining the characteristics of the original data. Specifically, let the rule-based transformation function be T, the update function of the recursive model be R, and the generated graph structure be for: ,in, is the state variable of the recursive model at time t, is the final state variable after the number of iterations.

[0017] After adopting the above technical solution, the present invention has the following advantages compared with the background technology: The present invention provides a ship appearance and interior concept map design method based on the relationship between text and graph, which adopts convolutional neural network, recurrent neural network (RNN) and deep learning model to improve the accuracy of entity recognition and relationship extraction, and realizes the rapid generation of ship appearance and interior design concept map by establishing a text graph modeling neural network architecture model, integrating image features and text description features. Improve the accuracy of demand communication and understanding, reduce deviation, quickly generate design solutions based on the needs of ship owners, realize real-time communication with ship owners, reduce communication costs, and ensure the correctness of design direction; improve design efficiency, reduce the time consumption of designers in model building and modification, and improve design efficiency. Real-time rendering display is realized, so that designers and ship owners can feel the design effect more intuitively; reduce the subjectivity of style positioning and material selection, and formulate standards for style positioning and material selection based on the needs and preferences of different scenarios and user groups, realize the rationality of style positioning, color matching, material selection and texture performance, and improve the versatility and market competitiveness of design. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] Please refer to Figure 1 As shown, the present invention discloses a method for designing a conceptual drawing of a ship's exterior and interior based on a text-drawing relationship, comprising the following steps: S1. Perform entity recognition and relationship extraction on the images of the ship's exterior and interior, specifically: S11, extracting features of the ship appearance and interior images based on the CNN convolutional neural network, obtaining image feature representations of the ship appearance and interior images after multiple convolutional layers and pooling layers, and mapping the text to the appropriate ship appearance and interior vector space through the embedding layer; S12, processing text sequences and relation extraction based on RNN recurrent neural network, obtaining text feature representation by performing aggregation operation on the last hidden state or multiple hidden states, and inputting the text feature representation corresponding to the entity into the relation classifier to obtain the relation category probability distribution; S13, based on the BERT deep learning model, input the token sequence of the text sequence after tokenization, add special tags, and output the text feature representation of each token through the BERT deep learning model; S2, based on the text graph modeling neural network architecture, concatenate and fuse the image feature representation with the text feature representation corresponding to the tag to generate a fused feature representation; S3. The latent space representation of the self-learning of the variational autoencoder and the image generation capability of the generative adversarial network are integrated, the parameters of the two are optimized through joint training, and the conceptual design drawings of the ship's exterior and interior are generated based on the transformation or differentiability of the rules of the recursive model; The method for extracting the features of the ship appearance and interior images in step S11 is as follows: Assume that the input ship appearance and interior images are I, and their dimensions are , where H represents the image height, W represents the image width, and C represents the image channel (for example, for a color image, C = 3); the convolution kernel is K, and its size is ,in, and are the sizes of the convolution kernel in height and width respectively; The calculation formula for the convolution operation is: ,in, is the value of the convolution result at position (i, j), and b is the bias term; After multiple convolutional layers and pooling layers of the CNN convolutional neural network, the feature representation of the ship's appearance and interior images is obtained. .

[0021] The text mapping method in step S11 is specifically as follows: the text is represented as a word vector sequence {x1, x2, ..., x T}, where T is the length of the text sequence, each is a word vector with dimension d, which maps the text into the appropriate ship appearance and interior vector space through the embedding layer.

[0022] Step S12 specifically includes: processing text sequences and extracting relations based on the RNN recurrent neural network, wherein the RNN recurrent neural network hidden state update formula is: , where h t is the hidden state at time t, is the weight matrix input to the hidden layer, is the weight matrix from hidden layer to hidden layer, is the bias term of the hidden layer; After processing the entire text sequence, the final feature representation of the text is obtained by performing aggregation operations on the last hidden state or multiple hidden states. ; The image features and text features After concatenation and fusion, fusion features are generated, and the text feature representation corresponding to the entity is input into the relationship classifier to calculate the probability distribution of the relationship category; Assume that the classification function used for entity recognition is , where y represents the entity category label, Fcombined is the fusion feature, and the softmax function is used to construct the probability distribution: ,in, is the score of the corresponding category k, calculated by the fully connected layer, C is the total number of entity categories, s j is the score corresponding to category j.

[0023] Step S13 is specifically as follows: based on the BERT deep learning model, the token sequence of the input text sequence after tokenization is added, and special tags are added, specifically: The input is represented as , where N is the number of input tokens, and each e i Contains word embedding and position embedding information; The BERT deep learning model calculates the self-attention of a single head as follows: ,in , W is the corresponding weight matrix, which is used to map the input E to the query Q, key K and value V respectively, and dk is the dimension of the key vector; Through the multi-head self-attention of the BERT deep learning model, multiple single heads are spliced ​​together: , where h is the number of heads, It is the output linear transformation weight matrix, which outputs the feature representation of each token after passing through multiple multi-head self-attention layers and the intermediate feedforward neural network layer of the BERT deep learning model.

[0024] The process of splicing and fusing the image feature representation and the text feature representation corresponding to the mark in step S2 is as follows: Suppose the fused feature representation is , the image features The dimension is and text features , the dimension is To perform the fusion: ,in, is the weight matrix of image features, with dimension , is the weight matrix of text features, with dimension , The fused bias term, is the expected dimension of the fused features.

[0025] The variational autoencoder in step S3 includes an encoder and a decoder. For the input ship appearance and interior image I, the encoder maps it to the latent variable space. Assume that the latent variable z obeys Gaussian distribution, and the encoder outputs mean μ and variance σ 2 , and its calculation formula is: , the latent variable z is sampled from the distribution through the reparameterization technique: , ε is a random variable that follows the standard normal distribution N(0,1); the decoder maps the latent variable z back to the image space, trying to reconstruct the input image and reconstruct the image The probability distribution of is: , the goal of the variational autoencoder is to maximize the variational lower bound of the likelihood function, and its mathematical expression is: ,in, is the posterior distribution of the latent variables learned by the encoder, is the generated distribution defined by the decoder, represents the KL divergence, which is used to measure the difference between two distributions. and are the parameters of the encoder and decoder respectively.

[0026] The generative adversarial network described in step S3 includes a generator and a discriminator. The goal of the generator is to generate an image as realistic as possible, map the random noise vector z to the image space, and generate an image = G(z), the task of the discriminator is to distinguish between the real image I and the image generated by the generator , output a probability value indicating the probability that the input image is a real image, namely D(I) and D( ), the objective function of the generated adversarial network is expressed as: , where p data(I) is the distribution of real images, and Pz(z) is the distribution of noise vectors.

[0027] Step S3 is specifically as follows: S31. The latent space representation of the self-learning variational autoencoder and the image generation capability of the generative adversarial network are integrated, and the parameters of the two are optimized through joint training, so that the generated ship appearance and interior conceptual design drawings can not only learn the distribution characteristics of the drawings (similar to the variational autoencoder), but also generate high-quality and realistic images (similar to the generative adversarial network); S32. Use the recursive model to gradually construct the structure of the graph or process the sequence representation of the graph, further learn the distribution law of the graph, and combine it with rule-based transformation (such as local or global transformation operations on the graph according to known image layout rules, geometric relationship rules, etc.) to generate a new and meaningful graph structure while maintaining the characteristics of the original data. Specifically, let the rule-based transformation function be T, the update function of the recursive model be R, and the generated graph structure be for: ,in, is the state variable of the recursive model at time t, is the final state variable after the number of iterations.

[0028] During the model training process, various objective functions, such as the ELBO of the variational autoencoder, the minimax game objective function of the generative adversarial network, and the joint loss with the text-graph relationship model, need to continuously adjust the model parameters so that the generated graph structure meets the requirements of matching the text description and maintaining the characteristics of the original data, improving the understanding and modeling capabilities of the relationship between graphs and texts, and achieving matching between the conceptual graphs of the ship's exterior and interior and the text element descriptions.

[0029] The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for designing a conceptual image of a ship's exterior and interior based on the relationship between text and images, characterized in that: The following steps are involved: S1. Perform entity recognition and relationship extraction on the images of the ship's exterior and interior, specifically: S11, extracting features of the ship appearance and interior images based on the CNN convolutional neural network, obtaining image feature representations of the ship appearance and interior images after multiple convolutional layers and pooling layers, and mapping the text to the appropriate ship appearance and interior vector space through the embedding layer; S12, processing text sequences and relation extraction based on RNN recurrent neural network, obtaining text feature representation by performing aggregation operation on the last hidden state or multiple hidden states, and inputting the text feature representation corresponding to the entity into the relation classifier to obtain the relation category probability distribution; S13, based on the BERT deep learning model, input the token sequence of the text sequence after tokenization, add special tags, and output the text feature representation of each token through the BERT deep learning model; S2, based on the text graph modeling neural network architecture, concatenate and fuse the image feature representation with the text feature representation corresponding to the tag to generate a fused feature representation; S3. Integrate the latent space representation of the variational autoencoder and the image generation capability of the generative adversarial network, optimize the parameters of the two through joint training, and generate conceptual design drawings of the ship's exterior and interior based on the transformation or differentiability of the rules of the recursive model.

2. A method for designing a ship exterior and interior concept map based on the relationship between text and map as claimed in claim 1, characterized in that: The method for extracting the features of the ship appearance and interior images in step S11 is as follows: Assume that the input ship appearance and interior images are I, whose dimensions are , where H represents the image height, W represents the image width, and C represents the image channel; the convolution kernel is K, and its size is ,in, and are the sizes of the convolution kernel in height and width respectively; The calculation formula for the convolution operation is: ,in, is the value of the convolution result at position (i, j), and b is the bias term; After multiple convolutional layers and pooling layers of the CNN convolutional neural network, the feature representation of the ship's appearance and interior images is obtained. .

3. A method for designing a ship exterior and interior concept map based on the relationship between text and map as claimed in claim 2, characterized in that: The text mapping method in step S11 is specifically as follows: the text is represented as a word vector sequence {x1, x2, ..., x T }, where T is the length of the text sequence, each is a word vector with dimension d, which maps the text into the appropriate ship appearance and interior vector space through the embedding layer.

4. A method for designing a ship exterior and interior concept map based on the relationship between text and map as claimed in claim 1, characterized in that: Step S12 specifically includes: processing text sequences and extracting relations based on the RNN recurrent neural network, wherein the RNN recurrent neural network hidden state update formula is: , where h t is the hidden state at time t, is the weight matrix input to the hidden layer, is the weight matrix from hidden layer to hidden layer, is the bias term of the hidden layer; After processing the entire text sequence, the final feature representation of the text is obtained by performing aggregation operations on the last hidden state or multiple hidden states. ; The image features and text features After concatenation and fusion, fusion features are generated, and the text feature representation corresponding to the entity is input into the relationship classifier to calculate the probability distribution of the relationship category; Assume that the classification function used for entity recognition is , where y represents the entity category label, Fcombined is the fusion feature, and the softmax function is used to construct the probability distribution: ,in, is the score of the corresponding category k, calculated by the fully connected layer, C is the total number of entity categories, s j is the score corresponding to category j.

5. The method for designing a ship exterior and interior concept map based on the relationship between text and map as claimed in claim 1, characterized in that: Step S13 is specifically as follows: based on the BERT deep learning model, the token sequence of the input text sequence after tokenization is added, and special tags are added, specifically: The input is represented as , where N is the number of input tokens, and each e i Contains word embedding and position embedding information; The BERT deep learning model calculates the self-attention of a single head as follows: ,in , W is the corresponding weight matrix, which is used to map the input E to the query Q, key K and value V respectively, and dk is the dimension of the key vector; Through the multi-head self-attention of the BERT deep learning model, multiple single heads are spliced ​​together: , where h is the number of heads, It is the output linear transformation weight matrix, which outputs the feature representation of each token after passing through multiple multi-head self-attention layers and the intermediate feedforward neural network layer of the BERT deep learning model.

6. A method for designing a ship exterior and interior concept map based on the relationship between text and map as claimed in claim 1, characterized in that: The specific process of splicing and fusing the image feature representation and the text feature representation corresponding to the mark in step S2 is: assuming that the fused feature representation is , the image features The dimension is and text features , the dimension is To perform the fusion: ,in, is the weight matrix of image features, with dimension , is the weight matrix of text features, with dimension , The fused bias term, is the expected dimension of the fused features.

7. The method for designing a conceptual drawing of a ship's exterior and interior based on the relationship between text and drawing as claimed in claim 1, characterized in that: The variational autoencoder in step S3 includes an encoder and a decoder. For the input ship appearance and interior image I, the encoder maps it to the latent variable space. Assume that the latent variable z obeys Gaussian distribution, and the encoder outputs mean μ and variance σ 2 , and its calculation formula is: , the latent variable z is sampled from the distribution through the reparameterization technique: , ε is a random variable that follows the standard normal distribution N(0,1); the decoder maps the latent variable z back to the image space, trying to reconstruct the input image and reconstruct the image The probability distribution of is: , the goal of the variational autoencoder is to maximize the variational lower bound of the likelihood function, and its mathematical expression is: ,in, is the posterior distribution of the latent variables learned by the encoder, is the generated distribution defined by the decoder, represents the KL divergence, which is used to measure the difference between two distributions. and are the parameters of the encoder and decoder respectively.

8. A method for designing a ship exterior and interior concept drawing based on the relationship between text and drawing as claimed in claim 7, characterized in that: The generative adversarial network described in step S3 includes a generator and a discriminator. The goal of the generator is to generate an image as realistic as possible, map the random noise vector z to the image space, and generate an image = G(z), the task of the discriminator is to distinguish between the real image I and the image generated by the generator , output a probability value indicating the probability that the input image is a real image, namely D(I) and D( ), the objective function of the generated adversarial network is expressed as: , where p data(I) is the distribution of real images, and Pz(z) is the distribution of noise vectors.

9. A method for designing a conceptual drawing of a ship's exterior and interior based on the relationship between text and drawing as claimed in claim 8, characterized in that: Step S3 is specifically as follows: S31. Fusion of the latent space representation learned by the variational autoencoder and the image generation capability of the generative adversarial network, and optimization of the parameters of the two through joint training to generate conceptual design drawings of the ship's exterior and interior; S32. Use the recursive model to gradually construct the structure of the graph or process the sequence representation of the graph, further learn the distribution law of the graph, and combine it with rule-based transformation to generate a new and meaningful graph structure while maintaining the characteristics of the original data. Specifically, let the rule-based transformation function be T, the update function of the recursive model be R, and the generated graph structure be for: ,in, is the state variable of the recursive model at time t, is the final state variable after the number of iterations.

Images