A Design Method for Conceptual Diagrams of Ship Exteriors and Interiors Based on the Relationship between Text and Images

Abstract

The present invention discloses a method for designing a conceptual diagram of the appearance and interior decoration of a ship based on the relationship between text and graph, comprising the following steps: S1, performing entity recognition and relationship extraction on the images of the appearance and interior decoration of the ship; S2, based on the neural network architecture of text-graph modeling, splicing and fusing the image feature representation and the text feature representation corresponding to the labels to generate a fused feature representation; S3, fusing the latent space representation self-learned by the variational autoencoder and the image generation ability of the generative adversarial network, optimizing the parameters of the two through joint training, and generating a conceptual design diagram of the appearance and interior decoration of the ship based on the transformation or differentiability of the rules of the recursive model.

Description

Technical Field

[0001] The present invention relates to the technical field of ship transportation, and particularly relates to a design method for ship appearance and interior concept drawings based on the relationship between text and graphics. Background Art

[0002] The design method of ship appearance concept drawings emphasizes function dominance. By analyzing the use of the ship, its appearance form is determined. For example, cargo ships focus on practicality and cargo capacity, while cruise ships focus on comfort and aesthetics. In the design, the modular method or the similar rectangle method is used to establish the harmonious proportional relationship between various parts to ensure the beauty of the overall shape. The design of ship interior concept drawings starts from requirement analysis and functional layout, communicates fully with the shipowner to understand the interior requirements, reasonably plans the position and area of the cabins, and the style positioning and color matching are the keys to shaping the interior atmosphere. The material selection and texture performance are related to the quality and comfort of the interior, and appropriate materials need to be selected according to different functional areas for texture matching.

[0003] The existing design methods for ship appearance and interior concept drawings still have the following problems:

[0004] (1) There are relatively large deviations in requirement communication and understanding. Although the designer communicates fully with the shipowner to understand the design requirements, the shipowner's requirements may be relatively vague or unprofessional, resulting in deviations when the designer understands and transforms the design requirements. There is a gap between the design scheme and the shipowner's expectations, and the communication cost is high and the process is cumbersome, affecting the design progress.

[0005] (2) The design efficiency and visualization degree are low. During the design process, especially in the initial stage of scheme conception, the designer needs to spend a lot of time on model building and modification, resulting in low design efficiency. And for complex ship appearance and interior designs, the existing modeling software may have certain limitations in visualization degree and detail performance, unable to fully display all details of the ship appearance and interior effects, affecting the accuracy of design decisions, and a large number of adjustments and modifications increase the cost.

[0006] (3) The subjectivity of style positioning and material selection is relatively strong. In aspects such as style positioning and color matching, and material selection and texture performance, it depends on the designer's personal experience and subjective judgment, lacking more objective and scientific evaluation criteria and decision-making basis, resulting in relatively large differences in the interior styles designed by different designers, and it is difficult to ensure that the selected styles and materials can be generally applicable to different ship usage scenarios and user groups. Summary of the Invention

[0007] The purpose of the present invention is to provide a design method for ship appearance and interior concept drawings based on the relationship between text and graphics to solve the above problems.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for designing a conceptual diagram of the appearance and interior of a ship based on the relationship between text and graph, comprising the following steps:

[0010] S1. Perform entity recognition and relationship extraction on the ship appearance and interior images, specifically:

[0011] S11. Extract features of the ship appearance and interior images based on the CNN convolutional neural network. After operations of multiple convolutional layers and pooling layers, obtain the image feature representations of the ship appearance and interior images, and map the text into the corresponding ship appearance and interior vector spaces through the embedding layer;

[0012] S12. Process the text sequence and perform relationship extraction based on the RNN recurrent neural network. By aggregating operations on the last hidden state or multiple hidden states, obtain the text feature representation, and input the text feature representation corresponding to the entity into the relationship classifier to obtain the relationship category probability distribution;

[0013] S13. Based on the BERT deep learning model, input the token sequence after tokenization of the text sequence, and add special markers. Output the text feature representation of each token through the BERT deep learning model;

[0014] S2. Based on the text graph modeling neural network architecture, splice and fuse the image feature representation and the text feature representation corresponding to the markers to generate a fused feature representation;

[0015] S3. Integrate the latent space representation self-learned by the variational autoencoder and the image generation ability of the generative adversarial network. Optimize the parameters of both through joint training, and generate the conceptual design diagrams of the ship appearance and interior based on the transformation or differentiability of the rules of the recursive model.

[0016] Preferably, the method for extracting the ship appearance and interior image features in step S11 is specifically: Let the input ship appearance and interior images be I, and its dimension is , where H represents the image height, W represents the image width, and C represents the image channels; the convolutional kernel is K, and its size is , where, and are the sizes of the convolutional kernel in the height and width directions respectively;

[0017] The calculation formula for the convolution operation is: , where, is the value of the convolution result at the position (i, j), and b is the bias term;

[0018] After passing through the operations of multiple convolutional layers and pooling layers of the CNN convolutional neural network, a feature representation of the ship's appearance and interior images is obtained. .

[0019] Preferably, the method of text mapping in step S11 is specifically: representing the text as a sequence of word vectors {x1, x2,..., x T} where T is the length of the text sequence, and each is a word vector with dimension d, and the text is mapped into the corresponding ship appearance and interior vector space through the embedding layer.

[0020] Preferably, step S12 is specifically: processing the text sequence and relation extraction based on the RNN recurrent neural network. The hidden state update formula of the RNN recurrent neural network is: , where h t is the hidden state at time t, is the weight matrix input to the hidden layer, is the weight matrix from the hidden layer to the hidden layer, is the bias term of the hidden layer;

[0021] After processing the entire text sequence, the final feature representation of the text is obtained by aggregating operations on the last hidden state or multiple hidden states ;

[0022] The image features and text features are concatenated and fused to generate fused features, and the text feature representation corresponding to the entity is input into the relation classifier to calculate the relation category probability distribution;

[0023] Assume that the classification function for entity recognition is , where y represents the entity category label, Fcombined is the fused feature, and the probability distribution is constructed using the softmax function: , where, is the score for the corresponding category k, calculated through the fully connected layer, C is the total number of entity categories, and s j is the score for the corresponding category j.

[0024] Preferably, step S13 is specifically: based on the BERT deep learning model, input the token sequence after tokenization of the text sequence and add special markers, specifically:

[0025] The input representation is , where N is the number of input tokens, and each e i contains word embedding and position embedding information;

[0026] The self-attention calculation of a single head by the BERT deep learning model is as follows: , where , W is the corresponding weight matrix, which is used to map the input E to the query Q, key K, and value V respectively, and dk is the dimension of the key vector;

[0027] The multiple single heads are concatenated through the multi-head self-attention of the BERT deep learning model: , where h is the number of heads, is the weight matrix of the linear transformation of the output. After passing through multiple multi-head self-attention layers and the intermediate feed-forward neural network layer of the BERT deep learning model, the feature representation of each token is output.

[0028] Preferably, the process of concatenating and fusing the image feature representation and the text feature representation corresponding to the label in step S2 is specifically as follows: Let the fused feature representation be , and the image feature with dimension and the text feature , with dimension are fused: , where is the weight matrix of the image feature, with dimension , is the weight matrix of the text feature, with dimension , is the bias term of the fusion, is the expected dimension of the fused feature.

[0029] Preferably, the variational autoencoder described in step S3 includes an encoder and a decoder. For the input ship appearance and interior image I, the encoder maps it to the latent variable space. Let the latent variable z follow a Gaussian distribution, and the encoder outputs the mean μ and variance σ 2 , and its calculation formula is: , and the latent variable z is sampled from the distribution through the reparameterization trick: , ε is a random variable that follows the standard normal distribution N(0,1); the decoder remaps the latent variable z back to the image space and attempts to reconstruct the input image. The probability distribution of the reconstructed image is: , the goal of the variational autoencoder is to maximize the variational lower bound of the likelihood function, and its mathematical expression is: , where is the posterior distribution of the latent variables learned by the encoder, is the generative distribution defined by the decoder, represents the KL divergence, which is used to measure the difference between two distributions, and are the parameters of the encoder and decoder respectively.

[0030] Preferably, the generative adversarial network described in step S3 includes a generator and a discriminator. The goal of the generator is to generate as realistic images as possible, mapping the random noise vector z to the image space to generate the image = G(z). The task of the discriminator is to distinguish between the real image I and the image generated by the generator, outputting a probability value representing the probability that the input image is a real image, that is, D(I) and D(

[0031] , where, p data(I) is the distribution of real images, and Pz(z) is the distribution of noise vectors.

[0032] Preferably, step S3 is specifically:

[0033] S31. Integrate the latent space representation self-learned by the variational autoencoder and the image generation ability of the generative adversarial network, and optimize the parameters of both through joint training to generate the conceptual design drawings of the ship's appearance and interior fittings;

[0034] S32. Use the recursive model to gradually construct the structure of the graph or process the sequential representation of the graph, further learn the distribution law of the graph, and combine with rule-based transformations to generate new and meaningful graph structures while maintaining the characteristics of the original data. Specifically: Let the rule-based transformation function be T, and the update function of the recursive model be R. The generated graph structure is: , where is the state variable of the recursive model at time t, is the final state variable after the number of iterations.

[0035] After adopting the above technical solution, compared with the background art, the present invention has the following advantages:

[0036] The present invention provides a method for designing a concept map of the appearance and interior decoration of a ship based on the relationship between text and graph. The convolutional neural network, recurrent neural network (RNN) and deep learning model are used to improve the accuracy of entity recognition and relationship extraction. By establishing a text-graph modeling neural network architecture model and integrating the features of images and text descriptions, the rapid generation of the concept design map of the appearance and interior decoration of the ship is realized. It improves the accuracy of requirement communication and understanding, reduces deviations, quickly generates design schemes based on the requirements of the shipowner, realizes real-time communication with the shipowner, reduces communication costs, and ensures the correctness of the design direction; improves design efficiency, reduces the time consumption of designers in model construction and modification, and improves design efficiency. Realize real-time rendering and display, enabling designers and shipowners to more intuitively feel the design effect; reduce the subjectivity of style positioning and material selection, and formulate standards for style positioning and material selection based on the needs and preferences of different scenarios and user groups, so as to realize the rationality of style positioning, color matching, material selection and texture performance, and improve the generality and market competitiveness of the design. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0039] Please refer to Figure 1 As shown, the present invention discloses a method for designing a concept map of the appearance and interior decoration of a ship based on the relationship between text and graph, including the following steps:

[0040] S1. Perform entity recognition and relationship extraction on the images of the appearance and interior decoration of the ship, specifically:

[0041] S11. Extract the image features of the appearance and interior decoration of the ship based on the CNN convolutional neural network. After operations of multiple convolutional layers and pooling layers, obtain the image feature representations of the appearance and interior decoration of the ship, and map the text to the corresponding ship appearance and interior vector spaces through the embedding layer;

[0042] S12. Process the text sequence and relationship extraction based on the RNN recurrent neural network. By performing an aggregation operation on the last hidden state or multiple hidden states, obtain the text feature representation, and input the text feature representation corresponding to the entity into the relationship classifier to obtain the relationship category probability distribution;

[0043] S13. Based on the BERT deep learning model, input the token sequence after tokenization of the text sequence, add special markers, and output the text feature representation of each token through the BERT deep learning model;

[0044] S2. Based on the neural network architecture for text graph modeling, splice and fuse the image feature representation and the text feature representation corresponding to the markers to generate a fused feature representation;

[0045] S3. Integrate the latent space representation self - learned by the variational auto - encoder and the image generation ability of the generative adversarial network, optimize the parameters of both through joint training, and generate the conceptual design drawings of the ship's appearance and interior based on the rule - based transformation or differentiable generation of the recurrent model;

[0046] The method for extracting the ship's appearance and interior image features in step S11 is specifically as follows: Let the input ship's appearance and interior image be I, and its dimension is , where H represents the image height, W represents the image width, and C represents the image channel (for example, for a color image C = 3); the convolution kernel is K, and its size is , where, and are the sizes of the convolution kernel in the height and width directions respectively;

[0047] The calculation formula for the convolution operation is: , where, is the value of the convolution result at the position (i, j), and b is the bias term;

[0048] After the operations of multiple convolution layers and pooling layers of the CNN convolutional neural network, the feature representation of the ship's appearance and interior image is obtained .

[0049] The method for text mapping in step S11 is specifically as follows: Represent the text as a sequence of word vectors {x1, x2,..., x T}, where T is the length of the text sequence, and each is a word vector with dimension d, and map the text into the corresponding ship's appearance and interior vector space through the embedding layer.

[0050] Step S12 is specifically as follows: Based on the RNN recurrent neural network to process the text sequence and relationship extraction, the hidden state update formula of the RNN recurrent neural network is: , where, ht is the hidden state at time t, is the weight matrix input to the hidden layer, is the weight matrix from the hidden layer to the hidden layer, is the bias term of the hidden layer;

[0051] After processing the entire text sequence, the final feature representation of the text is obtained by taking the last hidden state or aggregating multiple hidden states. ;

[0052] The image features and text features are concatenated and fused to generate fused features, and the text feature representation corresponding to the entity is input into the relation classifier to calculate the relation category probability distribution;

[0053] Assume the classification function for entity recognition is , where y represents the entity category label, Fcombined is the fused feature, and the probability distribution is constructed using the softmax function: , where, is the score corresponding to category k, calculated through the fully connected layer, C is the total number of entity categories, s j is the score corresponding to category j.

[0054] Step S13 is specifically as follows: Based on the BERT deep learning model, the token sequence after tokenization of the input text sequence is input, and special markers are added, specifically:

[0055] The input representation is , where N is the number of input tokens, and each e i contains word embedding and position embedding information;

[0056] The self-attention calculation of a single head by the BERT deep learning model is as follows: , where , W is the corresponding weight matrix, used to map the input E to the query Q, key K, and value V respectively, and dk is the dimension of the key vector;

[0057] The multiple single heads are concatenated through the multi-head self-attention of the BERT deep learning model: , where h is the number of heads, is the weight matrix of the output linear transformation. After passing through multiple multi-head self-attention layers and the intermediate feed-forward neural network layer of the BERT deep learning model, the feature representation of each token is output.

[0058] The process of splicing and fusing the image feature representation and the text feature representation corresponding to the tag described in step S2 is specifically as follows: Let the fused feature representation be , and the image feature with a dimension of and the text feature , with a dimension of are fused: , where is the weight matrix of the image feature, with a dimension of , is the weight matrix of the text feature, with a dimension of , is the bias term of the fusion, is the expected dimension of the fused feature.

[0059] The variational autoencoder described in step S3 includes an encoder and a decoder. For the input ship appearance and interior image I, the encoder maps it to the latent variable space. Let the latent variable z follow a Gaussian distribution, and the encoder outputs the mean μ and variance σ 2 , and its calculation formula is: . The latent variable z is sampled from the distribution through the reparameterization trick: , where ε is a random variable following the standard normal distribution N(0,1); the decoder remaps the latent variable z back to the image space and attempts to reconstruct the input image. The probability distribution of the reconstructed image is: . The goal of the variational autoencoder is to maximize the variational lower bound of the likelihood function, and its mathematical expression is: , where is the posterior distribution of the latent variable learned by the encoder, is the generative distribution defined by the decoder, represents the KL divergence, which is used to measure the difference between two distributions, and are the parameters of the encoder and decoder respectively.

[0060] The generative adversarial network described in step S3 includes a generator and a discriminator. The goal of the generator is to generate as realistic images as possible, mapping the random noise vector z to the image space to generate the image = G(z), and the task of the discriminator is to distinguish the real image I from the image generated by the generator , and output a probability value representing the probability that the input image is a real image, that is, D(I) and D( ). The objective function of the generative adversarial network is expressed as:

[0061] , where p data(I) is the distribution of real images, and Pz(z) is the distribution of noise vectors.

[0062] Step S3 is specifically as follows:

[0063] S31. Integrate the latent space representation self-learned by the variational autoencoder and the image generation ability of the generative adversarial network, and optimize the parameters of both through joint training, so that the generated concept design drawings of the ship's appearance and interior can not only learn the distribution characteristics of the drawings (similar to the variational autoencoder), but also generate high-quality and realistic images (similar to the generative adversarial network);

[0064] S32. Use a recursive model to gradually construct the structure of the graph or process the sequence representation of the graph, further learn the distribution law of the graph, and combine rule-based transformations (such as performing local or global transformation operations on the graph according to known image layout rules, geometric relationship rules, etc.), and generate new and meaningful graph structures while maintaining the characteristics of the original data. Specifically: Let the rule-based transformation function be T, and the update function of the recursive model be R. The generated graph structure is: , where is the state variable of the recursive model at time t, is the final state variable after the number of iterations.

[0065] During the model training process, various objective functions, such as the ELBO of the variational autoencoder, the minimax game objective function of the generative adversarial network, etc., and the joint loss with the text-graph relationship model, etc., need to continuously adjust the model parameters to make the generated graph structure meet the requirements of matching the text description and maintaining the characteristics of the original data, improve the understanding and modeling ability of the relationship between the graph and the text, and realize the matching of the ship's appearance and interior concept drawings with the text element descriptions.

[0066] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for designing concept maps of ship appearance and interior decoration based on the relationship between text and graphics, characterized in that, Including the following steps: S1. Conduct entity recognition and relationship extraction on the ship's exterior and interior images, specifically: S11. Extract the features of the ship's exterior and interior images based on the CNN convolutional neural network. After operations through multiple convolutional layers and pooling layers, obtain the image feature representations of the ship's exterior and interior images, and map the text into the corresponding ship's exterior and interior vector spaces through the embedding layer; S12. Process the text sequence and relationship extraction based on the RNN recurrent neural network. By aggregating operations on the last hidden state or multiple hidden states, obtain the text feature representation, and input the text feature representation corresponding to the entity into the relationship classifier to obtain the relationship category probability distribution; S13. Based on the BERT deep learning model, input the token sequence after tokenization of the text sequence and add special markers, and output the text feature representation of each token through the BERT deep learning model; S2. Based on the text graph modeling neural network architecture, splice and fuse the image feature representation and the text feature representation corresponding to the markers to generate a fused feature representation; S3. Integrate the latent space representation self-learned by the variational autoencoder and the image generation ability of the generative adversarial network, optimize the parameters of both through joint training, and generate the conceptual design drawings of the ship's exterior and interior based on the transformation or differentiability of the rules of the recursive model; The variational autoencoder described in step S3 includes an encoder and a decoder. For the input ship appearance and interior image I, the encoder maps it to the latent variable space. Let the latent variable z follow a Gaussian distribution, and the encoder outputs the mean μ and variance σ 2 , and its calculation formula is: , and the latent variable z is sampled from the distribution through the reparameterization trick: , where ε is a random variable following the standard normal distribution N(0, 1); the decoder remaps the latent variable z back to the image space and attempts to reconstruct the input image. The probability distribution of the reconstructed image is: , and the goal of the variational autoencoder is to maximize the variational lower bound of the likelihood function, and its mathematical expression is: , where is the posterior distribution of the latent variable learned by the encoder, is the generative distribution defined by the decoder, represents the KL divergence, which is used to measure the difference between two distributions, and are the parameters of the encoder and decoder respectively.

2. The method for designing the concept drawings of the ship's exterior and interior decoration based on the relationship between text and graphics according to claim 1, wherein: The method for extracting the appearance and interior image features of the ship described in step S11 is specifically as follows: Let the input ship appearance and interior image be I, and its dimension is , where H represents the image height, W represents the image width, and C represents the image channel; the convolution kernel is K, and its size is , where and are the dimensions of the convolution kernel in the height and width directions respectively; The calculation formula for the convolution operation is: , where is the value of the convolution result at the position (i, j), and b is the bias term; After the operations of multiple convolutional layers and pooling layers of the CNN convolutional neural network, the feature representations of the ship's appearance and interior images are obtained .

3. The method for designing the concept map of the ship's exterior and interior decoration based on the relationship between text and graph according to claim 2, characterized in that: The method of text mapping described in step S11 is specifically as follows: Represent the text as a sequence of word vectors {x1, x2, …, x T}, where T is the length of the text sequence, and each is a word vector with a dimension of d. The text is mapped into a corresponding ship appearance and interior fitting vector space through an embedding layer.

4. The ship exterior and interior conceptual drawing design method based on the relationship between text and graph according to claim 1, characterized in that: Step S12 specifically is: Processing the text sequence and relation extraction based on the RNN recurrent neural network, and the hidden state update formula of the RNN recurrent neural network is: , where, h t is the hidden state at time t, is the weight matrix input to the hidden layer, is the weight matrix from the hidden layer to the hidden layer, is the bias term of the hidden layer; After processing the entire text sequence, the text feature representation is obtained by using the last hidden state or aggregating multiple hidden states ; Input the text feature representation corresponding to the entity into the relationship classifier to calculate the relationship category probability distribution; Suppose the classification function for entity recognition is , where y represents the entity category label, is the text feature representation, and the softmax function is used to construct the probability distribution: , where, is the score corresponding to category k, which is calculated through the fully connected layer, D is the total number of entity categories, and s j is the score corresponding to category j.

5. The method for designing the concept map of the ship's exterior and interior decoration based on the relationship between text and graph according to claim 1, characterized in that: Step S13 is specifically: Based on the BERT deep learning model, input the token sequence after tokenization of the text sequence and add special markers, specifically: The input is represented as , where N is the number of input tokens, and each e i contains word embedding and position embedding information; The self-attention calculation of a single head in the BERT deep learning model is as follows: , where , W is the corresponding weight matrix, which is used to map the input E to the query Q, the key K, and the value V respectively, and dk is the dimension of the key vector; The multiple single heads are concatenated through the multi-head self-attention of the BERT deep learning model: , where h is the number of heads, is the weight matrix of the linear transformation of the output. After passing through multiple multi-head self-attention layers and the intermediate feed-forward neural network layer of the BERT deep learning model, the feature representation of each token is output.

6. The method for designing the concept map of the ship's exterior and interior decoration based on the relationship between text and graph as described in claim 1, wherein: The process of splicing and fusing the image feature representation and the text feature representation corresponding to the label described in step S2 is specifically as follows: Let the fused feature representation be , the image feature with a dimension of and the text feature , with a dimension of are fused: , where is the weight matrix of the image feature, with a dimension of , is the weight matrix of the text feature, with a dimension of , is the bias term of the fusion, is the expected dimension of the fused feature.

7. The ship exterior and interior conceptual drawing design method based on the relationship between text and graph according to claim 1, characterized in that: The generative adversarial network described in step S3 includes a generator and a discriminator. The goal of the generator is to generate as realistic images as possible, mapping the random noise vector z to the image space to generate images = G(z). The task of the discriminator is to distinguish between real images I and the images generated by the generator , and output a probability value representing the probability that the input image is a real image, that is, D(I) and D( ). The objective function of the generative adversarial network is expressed as: , where p data(I) is the distribution of the real images, and Pz(z) is the distribution of the noise vectors.

8. The method for designing a conceptual diagram of the appearance and interior decoration of a ship based on the relationship between text and graph according to claim 7, characterized in that: Step S3 is specifically: S31. Integrate the latent space representation self-learned by the variational autoencoder and the image generation ability of the generative adversarial network, optimize the parameters of both through joint training, and generate the conceptual design drawings of the ship's exterior and interior; S32. Use the recursive model to gradually construct the graph structure or process the sequence representation of the graph, further learn the distribution law of the graph, and combine with rule-based transformations to generate a new graph structure while maintaining the characteristics of the original data. Specifically: Let the rule-based transformation function be T, and the update function of the recursive model be R. The generated graph structure is: , where is the state variable of the recursive model at time t, is the final state variable after the number of iterations.

Images