Ship design knowledge retrieval method and system based on multimodal knowledge graph

Abstract

The present invention provides a ship design knowledge retrieval method and system based on a multimodal knowledge graph, the method comprising: collecting and constructing a ship design set from the field of ship design; forming a hierarchical multimodal knowledge graph that describes concepts, physical properties, and design specifications in the field of ship design; enhancing the association between different modal data by vector similarity calculation; integrating the hierarchical multimodal knowledge graph into a ChatGLM model; locating knowledge nodes related to user needs in the hierarchical multimodal knowledge graph; matching and locating ship design knowledge related to user query content, and feeding back the ship design knowledge obtained by intelligent semantic retrieval to the user in a hierarchical form. The present invention realizes ship design knowledge retrieval based on multimodal knowledge graph-enhanced ChatGLM, which improves the information acquisition efficiency of engineering designers and the practical application value of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of ship design, and in particular to a ship design knowledge retrieval method and system based on a multimodal knowledge graph. Background Art

[0002] With the development of artificial intelligence and big data technology, knowledge retrieval and intelligent question answering in complex fields have gradually become a hot topic in technical research. In the field of ship design, since the design process involves multiple data types and complex semantic relationships, high requirements are placed on the accuracy and intelligence of knowledge retrieval. However, traditional knowledge retrieval methods are difficult to meet this demand, resulting in many difficulties for engineers in obtaining design-related information.

[0003] At present, ship design knowledge retrieval mainly relies on traditional keyword matching and rule-based search methods. The core of the traditional retrieval method is to locate relevant documents or entries in a pre-built database through keywords entered by users. Although it is simple and easy to use, it has significant technical limitations. First, ship design involves a wide range of complex data types, and there are often implicit semantic associations between data. Keyword matching cannot understand the deep semantic relationship of these multimodal data, resulting in low accuracy of retrieval results.

[0004] Secondly, the existing rule-based retrieval system has limited semantic understanding of queries, and it is difficult to effectively analyze the user's design needs when faced with professional terms unique to the field of ship design. For example, the query raised by the user may involve specific design standards and the physical properties of structural materials, but the traditional retrieval system cannot combine the contextual semantics to give comprehensive retrieval results. In addition, such systems lack a deep understanding of the user's query contextual intent and cannot meet the user's actual needs for multi-angle design knowledge retrieval.

[0005] In recent years, some intelligent search technologies have attempted to introduce knowledge graphs and natural language processing models to improve the intelligence level of knowledge retrieval. However, in the field of ship design, the existing technologies still have the following significant problems:

[0006] Insufficient multimodal data fusion capabilities: Current knowledge retrieval systems usually only support single-modal data retrieval, and it is difficult to comprehensively process the semantic information of text, images, CAD models and simulation data, and cannot effectively integrate the associations between different modal data.

[0007] Limited ability to understand professional semantics: The application of general natural language processing models in professional fields is limited. Existing models are unable to adequately understand the unique terminology and complex semantics in the field of ship design, resulting in a lack of professional depth and contextual relevance in search results.

[0008] The accuracy and intelligence of search results are not high: the existing technology often only analyzes the user's query intent at a shallow level, and it is difficult to provide multi-angle search feedback based on complex query conditions. In addition, the existing system is prone to generate hallucination facts when the user query exceeds the system's capabilities, further reducing the reliability of the search results.

[0009] In summary, the existing technologies have obvious defects in multimodal data fusion, professional semantic understanding and intelligent retrieval, which makes it difficult to meet the needs of efficient and accurate knowledge retrieval in the field of ship design. These problems directly affect the work efficiency and information acquisition quality of design engineers in complex design scenarios, and a new method is urgently needed to effectively solve the above technical problems. Summary of the invention

[0010] The present invention provides a ship design knowledge retrieval method and system based on a multimodal knowledge graph, which is used to solve the defects of the prior art in multimodal data fusion, professional semantic understanding and intelligent retrieval, and realizes ship design knowledge retrieval based on multimodal knowledge graph enhanced ChatGLM, thereby improving the information acquisition efficiency of engineering designers and the practical application value of the system.

[0011] The present invention provides a ship design knowledge retrieval method based on a multimodal knowledge graph, comprising:

[0012] Collect and construct a multimodal ship design set D from the field of ship design, and standardize the ship design set D;

[0013] Perform deep semantic analysis on the standardized ship design set D' to generate a structured knowledge node set N, each knowledge node represents a specific concept, component, material property or design standard in ship design, establish a semantic association relationship set R between the knowledge nodes, and construct a hierarchical multimodal knowledge graph G based on the knowledge node set N and the semantic association relationship set R;

[0014] The ship design set of different modes in the hierarchical multimodal knowledge graph G is represented as an embedding vector in a unified embedding space, and a set of semantic association relationships between the embedding vectors of different modal data is constructed. , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain the hierarchical multimodal knowledge graph ;

[0015] Hierarchical Multimodal Knowledge Graph The ChatGLM model is integrated into the ChatGLM model, and the specialized terms and common expressions in the field of ship design are added to the ChatGLM model. Based on the ChatGLM model, the hierarchical multimodal knowledge graph is searched according to the user's query intention. Conduct retrieval and generate multi-angle retrieval feedback for users based on the retrieval results;

[0016] Based on the ChatGLM model, the user's natural language query content q is parsed, and the key semantic elements in the query are extracted by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity in the embedding space, and the user's query intention is identified. Based on the parsed query intention and key semantic elements, the knowledge nodes related to the user's needs are located in the hierarchical multimodal knowledge graph. ;

[0017] Acquisition and Knowledge Nodes Relevant ship design knowledge is fed back to the user in a hierarchical form.

[0018] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, a multimodal ship design set is collected and constructed from the ship design field, and the ship design set is standardized, including:

[0019] The original data in the field of ship design are classified according to the source, and a text data set T, an image data set I, a CAD model data set C and a three-dimensional simulation data set S are obtained;

[0020] Extract data from each modal data set through the data acquisition interface, and construct a ship design set D based on multiple modal data sets, D = {T, I, C, S};

[0021] The text data set T in the ship design set D is cleaned to remove non-design related information and the format is unified into a standardized structure encoded in UTF-8;

[0022] The standardization of the image data set I in the ship design set D includes size normalization and adjusting each image to a uniform resolution;

[0023] The standardization process of the CAD model data set C in the ship design set D includes converting CAD files of different formats into a unified geometric description format;

[0024] The standardized processing of the three-dimensional simulation data set S in the ship design set D includes extracting numerical calculation results and generating metadata records corresponding to the simulation scenarios;

[0025] The ship design set D' after standardized processing is expressed as D' = {T', I', C', S'}, where T' is the standardized processing result of the text data set T, I' is the standardized processing result of the image data set I, C' is the standardized processing result of the CAD model data set C, and S' is the standardized processing result of the three-dimensional simulation data set S.

[0026] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, the text data set T includes ship design specifications, design instructions and technical reports, the image data set I includes ship design sketches, pictures and structural schematics, the CAD model data set C includes a computer-aided design model that describes the structure of ship components in three-dimensional space, and the three-dimensional simulation data set S includes simulation calculation results.

[0027] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, a deep semantic analysis is performed on the standardized ship design set D' to generate a structured knowledge node set N, including:

[0028] Through the semantic embedding function, each text data in the text data set T' is mapped into a semantic feature vector to obtain a text feature set ;

[0029] Using a deep convolutional neural network as the feature extraction function, each image data in the image data set I' is extracted as a feature vector to obtain the image feature set ;

[0030] Through the geometric feature extraction function, each CAD model in the CAD model data set C' is extracted as a geometric feature vector to obtain the CAD model feature set ;

[0031] Through the simulation result feature extraction function, each simulation data in the three-dimensional simulation data set S' is extracted as a simulation feature vector to obtain a simulation feature set ;

[0032] The extracted text feature set , image feature set , CAD model feature set and simulation feature sets Mapped to a structured knowledge node set N, the mth knowledge node Defined as:

[0033] ;

[0034] in, is the unique identifier of the mth knowledge node, is the node type of the mth knowledge node, representing a concept, component, material property, or design standard. is the feature vector corresponding to the mth knowledge node, taken from the text feature set , image feature set , CAD model feature set and simulation feature sets one of the, is the attribute set of the mth knowledge node, including the semantic description and metadata information of the knowledge node.

[0035] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, a semantic association relationship set R between the knowledge nodes is established, and a hierarchical multimodal knowledge graph G is constructed according to the knowledge node set N and the semantic association relationship set R, including:

[0036] Determine the association degree between the knowledge nodes according to the feature vectors of the knowledge nodes, and generate an association relationship set R according to the association degree between the knowledge nodes:

[0037] ;

[0038] in, are the mth and nth knowledge nodes respectively, Knowledge Node The semantic relationship types between them include inclusion, similarity, and dependency. Knowledge Node The correlation between:

[0039] ;

[0040] Among them, s Knowledge Node The eigenvector of With knowledge nodes The eigenvector of The similarity function between them is calculated using cosine similarity. According to the semantic relationship type The weight factor assigned, Z is the normalization coefficient;

[0041] A hierarchical multimodal knowledge graph G is constructed according to the knowledge node set N and the semantic association relationship set R:

[0042] G = (N, R);

[0043] Among them, the hierarchical structure of the multimodal knowledge graph is based on the semantic relationship types between knowledge nodes. and relevance Organize and form a hierarchical multimodal knowledge graph that describes the concepts, physical properties and design specifications in the field of ship design.

[0044] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, a semantic association relationship set between embedded vectors of different modal data is constructed. , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain a hierarchical multimodal knowledge graph :

[0045] Calculate the semantic similarity between the embedding vectors of different modal data and obtain the similarity matrix M:

[0046] ;

[0047] in, for and The similarity score between and They represent the embedding vector of the i-th data of modality x and the embedding vector of the j-th data of modality y, respectively. is the modulus of the vector, For the modal alignment module, is the alignment factor;

[0048] Generate a set of cross-modal semantic association relations based on the similarity matrix M :

[0049] ;

[0050] in, is the threshold value, for and semantic association relationship.

[0051] The semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain a hierarchical multimodal knowledge graph :

[0052] .

[0053] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, a hierarchical multimodal knowledge graph is constructed. Integrated into the ChatGLM model, and added specialized terms and common expressions in the field of ship design to the ChatGLM model, including:

[0054] Adopting graph fusion function to layered multimodal knowledge graph Fusion with the semantic space of ChatGLM to update the parameters of the ChatGLM model :

[0055] ;

[0056] in, is the ChatGLM model parameter after fusion, are the original ChatGLM model parameters, is the fusion coefficient, which is used to control the influence of the hierarchical multimodal knowledge graph on the update of the ChatGLM model. is the graph fusion function;

[0057] The domain vocabulary embedding function is used to collect the specialized terms in the field of ship design. and a collection of common expressions Mapped to a set of embedding vectors :

[0058] ;

[0059] in, For proprietary terms or common expressions The embedding vector of It is a domain vocabulary embedding function, which is used to capture the semantic features unique to the field of ship design and integrate proprietary terms and common expressions into the embedding space of the ChatGLM model;

[0060] The original vocabulary V of the ChatGLM model is combined with the set of special terms , Common expressions collection Merge, build an enhanced vocabulary V', and use the embedding vector set Update the embedding matrix E' of the ChatGLM model.

[0061] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, the hierarchical multimodal knowledge graph is retrieved based on the ChatGLM model according to the user's query intention. Conduct a search and generate multi-angle search feedback for users based on the search results, including:

[0062] Define query functions for multi-level semantic relationships using the semantic structure of hierarchical multimodal knowledge graphs , get the retrieval result set according to the user's query intention u :

[0063] ;

[0064] in, is the search result set, is the embedding vector of the user’s query intent, Embed functions for user queries, is the i-th knowledge node The embedding vector of The user's query intention u and knowledge node The embedding vector The similarity calculation function between is the similarity threshold;

[0065] According to the search result set The hierarchical structure of the hierarchical multimodal knowledge graph generates multi-angle search feedback for users, and generates functions through the results Build feedback content:

[0066] ;

[0067] in, To provide feedback to users on search results, Knowledge Node The importance weight of Knowledge Node Detailed information about the node, including its properties, relationships, and hierarchy.

[0068] According to a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention, a user's natural language query content q is parsed based on a ChatGLM model, key semantic elements in the query are extracted by combining the semantic structure in a hierarchical multimodal knowledge graph and the semantic similarity of an embedding space, and the user's query intention is identified. Based on the parsed query intention and key semantic elements, knowledge nodes related to user needs are located in the hierarchical multimodal knowledge graph, including:

[0069] The query parsing module of the ChatGLM model converts the user’s natural language query q into a query intent embedding vector and a set of key semantic elements :

[0070]

[0071] ;

[0072] in, is the weight of the key semantic element, is the i-th key semantic element Relevance scoring in hierarchical multimodal knowledge graphs, is the correlation threshold;

[0073] Calculate the user's query intent embedding vector and knowledge nodes in hierarchical multimodal knowledge graphs The embedding vector The semantic similarity of , construct the similarity score matrix S:

[0074] ;

[0075] in, Embedding vectors for query intent With knowledge nodes The embedding vector The weighted similarity score of is the semantic alignment module, and is the weight factor;

[0076] Generate candidate node set :

[0077] ;

[0078] in, is the similarity threshold, which is used to filter knowledge nodes related to the query requirements. To represent a set of semantic elements, where the semantic elements are associated with nodes in the knowledge graph; By extracting the knowledge nodes from the knowledge graph Filter out the weighted similarity scores Greater than or equal to the similarity threshold , and the embedding vector Belongs to the semantic element set Knowledge Node constitute;

[0079] After performing an intent query on the user's semantics, a set of key semantic elements is filtered through text filters. To perform credibility scoring and filtering:

[0080] ;

[0081] ;

[0082] in, Representing semantic elements and knowledge nodes The strength of the relationship between is the credibility score threshold, is the filtered set of valid key semantic elements;

[0083] Expand the set of candidate nodes screened through the hierarchical structure of the hierarchical multimodal knowledge graph , generate a context extension node set :

[0084] ;

[0085] in, Knowledge Node and The semantic relationship type of To hierarchically correspond to the levels of knowledge nodes in the multimodal knowledge graph, so that the extended nodes are relevant to the user query at the graph level;

[0086] The candidate node set and the expanded node collection Merge to generate the final set of related nodes :

[0087] ;

[0088] in, For knowledge and The association path between them is used to preserve the extended semantic relationship.

[0089] The present invention also provides a ship design knowledge retrieval system based on a multimodal knowledge graph, comprising:

[0090] A processing module, used for collecting and constructing a multimodal ship design set D from the field of ship design, and performing standardization processing on the ship design set D;

[0091] A construction module is used to perform deep semantic analysis on the standardized ship design set D', generate a structured knowledge node set N, each knowledge node represents a specific concept, component, material property or design standard in ship design, establish a semantic association relationship set R between the knowledge nodes, and construct a hierarchical multimodal knowledge graph G based on the knowledge node set N and the semantic association relationship set R;

[0092] Integration module, used to represent the ship design set of different modes in the hierarchical multimodal knowledge graph G as embedding vectors in a unified embedding space, and to construct a set of semantic association relationships between the embedding vectors of different modal data , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain the hierarchical multimodal knowledge graph ;

[0093] The first retrieval module is used to transform the hierarchical multimodal knowledge graph The ChatGLM model is integrated into the ChatGLM model, and the specialized terms and common expressions in the field of ship design are added to the ChatGLM model. Based on the ChatGLM model, the hierarchical multimodal knowledge graph is searched according to the user's query intention. Conduct retrieval and generate multi-angle retrieval feedback for users based on the retrieval results;

[0094] The second retrieval module is used to parse the user's natural language query content q based on the ChatGLM model, extract the key semantic elements in the query by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity in the embedding space, and identify the user's query intention. Based on the parsed query intention and key semantic elements, the knowledge nodes related to the user's needs are located in the hierarchical multimodal knowledge graph. ;

[0095] Feedback module, used to obtain and knowledge nodes Relevant ship design knowledge is fed back to the user in a hierarchical form.

[0096] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the ship design knowledge retrieval method based on the multimodal knowledge graph as described above is implemented.

[0097] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the ship design knowledge retrieval methods based on a multimodal knowledge graph as described above.

[0098] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the ship design knowledge retrieval methods based on a multimodal knowledge graph as described above.

[0099] The ship design knowledge retrieval method and system based on multimodal knowledge graph provided by the present invention, by uniformly embedding the multimodal ship design set into a semantic space, and realizing cross-modal knowledge integration by establishing semantic association relationships for these heterogeneous data, can effectively capture the deep semantic relationship between different modal data, greatly improve the comprehensiveness and accuracy of retrieval, and realize the semantic alignment of multimodal data in complex query scenarios, so that users can obtain cross-modal and multi-angle ship design knowledge through natural language at one time, which significantly improves the accuracy and efficiency of information retrieval; by combining the multimodal knowledge graph with the ChatGLM model, the model's ability to understand the specific terms and complex semantics in the field of ship design is significantly enhanced; by introducing the semantic embedding of domain-specific terms and commonly used expressions, combined with the hierarchical graph structure of domain knowledge, it can deeply analyze the query requirements proposed by users, accurately capture design-related intentions, and enable users to quickly obtain accurate information that conforms to the design context. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0101] Figure 1 It is a flow chart of a ship design knowledge retrieval method based on a multimodal knowledge graph provided by the present invention;

[0102] Figure 2 It is a complete flow chart of the ship design knowledge retrieval method based on the multimodal knowledge graph provided by the present invention;

[0103] Figure 3 It is a structural schematic diagram of a ship design knowledge retrieval system based on a multimodal knowledge graph provided by the present invention;

[0104] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0105] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0106] Combine the following Figure 1 A ship design knowledge retrieval method based on a multimodal knowledge graph of the present invention is described, comprising:

[0107] Step 101, collecting and constructing a multimodal ship design set D from the field of ship design, and standardizing the ship design set D;

[0108] Step 102, performing deep semantic analysis on the standardized ship design set D' to generate a structured knowledge node set N, each knowledge node representing a specific concept, component, material property or design standard in ship design, performing semantic association analysis on the relationship between knowledge nodes, establishing a semantic association relationship set R between the knowledge nodes according to the knowledge system in the field of ship design, and forming a hierarchical multimodal knowledge graph G describing concepts, physical properties and design specifications in the field of ship design according to the knowledge node set N and the semantic association relationship set R;

[0109] Step 103: embed the multimodal knowledge graph data using multimodal knowledge graph embedding, represent different types of data in a unified embedding space, enhance the association between different modal data by vector similarity calculation, and construct a set of semantic association relationships between the embedding vectors of different modal data. , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain the hierarchical multimodal knowledge graph ;

[0110] Step 104: Hierarchical multimodal knowledge graph Integrate it into the ChatGLM model, add the specialized terms and common expressions in the field of ship design to the ChatGLM model, and intelligently guide the user's query intention based on the ChatGLM model through the semantic structure support of the hierarchical multimodal knowledge graph. In the query, the hierarchical structure of the knowledge graph is combined to generate multi-angle search feedback for users based on the search results;

[0111] Step 105: parse the user's natural language query content q based on the ChatGLM model, extract the key semantic elements in the query by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity in the embedding space, identify the user's specific design requirements and contextual query intent, and locate the knowledge nodes related to the user's needs in the hierarchical multimodal knowledge graph based on the parsed query intent and key semantic elements. ;

[0112] Step 106, obtain the knowledge node Relevant ship design knowledge is fed back to the user in a hierarchical form.

[0113] After parsing the user's query intention and key semantic elements, intelligent semantic retrieval is performed according to the semantic association relationship in the hierarchical multimodal knowledge graph, and the ship design knowledge related to the user's query content is matched and located through cross-modal association and semantic similarity in the embedding space.

[0114] The ship design knowledge obtained by intelligent semantic retrieval is fed back to users in a hierarchical form, including design parameters, design standards and historical case categories, and the hierarchical relationship of the knowledge graph is displayed through a structural relationship diagram. The complete flowchart is as follows Figure 2 shown.

[0115] This embodiment embeds the multimodal ship design set into a semantic space and realizes cross-modal knowledge integration by establishing semantic association relationships for these heterogeneous data. It can effectively capture the deep semantic relationship between different modal data, greatly improve the comprehensiveness and accuracy of retrieval, and realize the semantic alignment of multimodal data in complex query scenarios, so that users can obtain cross-modal and multi-angle ship design knowledge at one time through natural language, which significantly improves the accuracy and efficiency of information retrieval. By combining the multimodal knowledge graph with the ChatGLM model, the model's ability to understand the specific terms and complex semantics in the field of ship design is significantly enhanced. By introducing the semantic embedding of domain-specific terms and commonly used expressions, combined with the hierarchical graph structure of domain knowledge, it can deeply analyze the query requirements raised by users and accurately capture design-related intentions, so that users can quickly obtain accurate information that conforms to the design context.

[0116] On the basis of the above embodiment, in this embodiment, a multi-modal ship design set is collected and constructed from the field of ship design, and the ship design set is standardized, including:

[0117] The original data in the field of ship design are classified according to the source, and a text data set T, an image data set I, a CAD model data set C and a three-dimensional simulation data set S are obtained;

[0118] Extract data from each modal data set through the data acquisition interface, record the source and meta-information of each piece of data, and construct the ship design set D, D = {T, I, C, S};

[0119] The text data set T in the ship design set D is cleaned to remove non-design related information and the format is unified into a standardized structure encoded in UTF-8;

[0120] The standardization of the image data set I in the ship design set D includes size normalization and adjusting each image to a uniform resolution;

[0121] The standardization process of the CAD model data set C in the ship design set D includes converting CAD files of different formats into a unified geometric description format;

[0122] The standardized processing of the three-dimensional simulation data set S in the ship design set D includes extracting numerical calculation results and generating metadata records corresponding to the simulation scenarios;

[0123] The ship design set D' after standardized processing is expressed as D' = {T', I', C', S'}, where T' is the standardized processing result of the text data set T, I' is the standardized processing result of the image data set I, C' is the standardized processing result of the CAD model data set C, and S' is the standardized processing result of the three-dimensional simulation data set S.

[0124] On the basis of the above embodiments, the text data set T in this embodiment includes ship design specifications, design instructions and technical reports, the image data set I includes ship design sketches, pictures and structural schematics, the CAD model data set C includes a computer-aided design model that describes the structure of ship components in three-dimensional space, and the three-dimensional simulation data set S includes simulation calculation results.

[0125] On the basis of the above embodiment, in this embodiment, a deep semantic analysis is performed on the standardized ship design set D' to generate a structured knowledge node set N, including:

[0126] Each text data in the text data set T' is embedded by the semantic embedding function Mapped into semantic feature vectors to obtain text feature sets ;

[0127] Using deep convolutional neural network as feature extraction function, each image data in the image data set I' Extract as feature vector to get image feature set ;

[0128] Each CAD model in the CAD model data set C' is extracted by the geometric feature extraction function. Extract as geometric feature vector to obtain CAD model feature set ;

[0129] Each simulation data in the three-dimensional simulation data set S' is extracted by the simulation result feature extraction function. Extract as simulation feature vector and get simulation feature set ;

[0130] The extracted text feature set , image feature set , CAD model feature set and simulation feature sets Mapped to a structured knowledge node set N, the mth knowledge node Defined as:

[0131] ;

[0132] in, is the unique identifier of the mth knowledge node, is the node type of the mth knowledge node, representing a concept, component, material property, or design standard. is the feature vector corresponding to the mth knowledge node, taken from the text feature set , image feature set , CAD model feature set and simulation feature sets one of the, is the attribute set of the mth knowledge node, including the semantic description and metadata information of the knowledge node.

[0133] On the basis of the above embodiment, in this embodiment, a semantic association relationship set R between the knowledge nodes is established, and a hierarchical multimodal knowledge graph G is constructed according to the knowledge node set N and the semantic association relationship set R, including:

[0134] According to the knowledge system in the field of ship design, the semantic association relationship between knowledge nodes is analyzed, the association degree between the knowledge nodes is determined according to the feature vectors of the knowledge nodes, and the association relationship set R is generated according to the association degree between the knowledge nodes:

[0135] ;

[0136] in, are the mth and nth knowledge nodes respectively, Knowledge Node The semantic relationship types between them include inclusion, similarity, and dependency. Knowledge Node The correlation between:

[0137] ;

[0138] Among them, s Knowledge Node The eigenvector of With knowledge nodes The eigenvector of The similarity function between them is calculated using cosine similarity. According to the semantic relationship type The weight factor assigned, Z is the normalization coefficient;

[0139] A hierarchical multimodal knowledge graph G is constructed according to the knowledge node set N and the semantic association relationship set R:

[0140] G = (N, R);

[0141] Among them, the hierarchical structure of the multimodal knowledge graph is based on the semantic relationship types between knowledge nodes. and relevance Organize and form a hierarchical multimodal knowledge graph that describes the concepts, physical properties and design specifications in the field of ship design, and realize the semantic integration and association of multimodal data.

[0142] Based on the above embodiment, this embodiment constructs a semantic association relationship set between the embedding vectors of different modal data. , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain a hierarchical multimodal knowledge graph :

[0143] The text data, image data, CAD model data and 3D simulation data in the hierarchical multimodal knowledge graph G are embedded with modal features, and each modal data is mapped to a unified embedding space to obtain the text data embedding vector , image data embedding vector , CAD model data embedded vector and 3D simulation data embedding vector ;

[0144] Define the modality alignment function to calculate the semantic similarity between the embedding vectors of different modal data in the unified embedding space and obtain the similarity matrix M:

[0145] ;

[0146] in, for and The similarity score between and They represent the embedding vector of the i-th data of modality x and the embedding vector of the j-th data of modality y, respectively. is the modulus of the vector, For the modal alignment module, is the alignment factor;

[0147] Generate a set of cross-modal semantic association relations based on the similarity matrix M :

[0148] ;

[0149] in, is the threshold used to filter low-correlation data pairs. for and semantic association relationship.

[0150] The semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain a hierarchical multimodal knowledge graph :

[0151] .

[0152] Based on the above embodiment, this embodiment converts the hierarchical multimodal knowledge graph into Integrated into the ChatGLM model, and added specialized terms and common expressions in the field of ship design to the ChatGLM model, including:

[0153] Hierarchical Multimodal Knowledge Graph Integrated into the ChatGLM model, the graph fusion function is used to transform the hierarchical multimodal knowledge graph Fusion with the semantic space of ChatGLM to update the parameters of the ChatGLM model :

[0154] ;

[0155] in, is the ChatGLM model parameter after fusion, are the original ChatGLM model parameters, is the fusion coefficient, which is used to control the influence of the hierarchical multimodal knowledge graph on the update of the ChatGLM model. is the graph fusion function;

[0156] Incorporating a specialized terminology set from the field of ship design into the ChatGLM model and a collection of common expressions , through the domain vocabulary embedding function, the specialized terms in the field of ship design are collected and a collection of common expressions Mapped to a set of embedding vectors :

[0157] ;

[0158] in, For proprietary terms or common expressions The embedding vector of It is a domain vocabulary embedding function, which is used to capture the semantic features unique to the field of ship design and integrate proprietary terms and common expressions into the embedding space of the ChatGLM model;

[0159] The original vocabulary V of the ChatGLM model is combined with the set of special terms , Common expressions collection Merge, build an enhanced vocabulary V', and use the embedding vector set Update the embedding matrix E' of the ChatGLM model.

[0160] Based on the above embodiment, this embodiment uses the ChatGLM model to query the hierarchical multimodal knowledge graph according to the user's query intention. Conduct a search and generate multi-angle search feedback for users based on the search results, including:

[0161] Define query functions for multi-level semantic relationships using the semantic structure of hierarchical multimodal knowledge graphs , get the retrieval result set according to the user's query intention u :

[0162] ;

[0163] in, is the search result set, is the embedding vector of the user’s query intent, Embed functions for user queries, is the i-th knowledge node The embedding vector of The user's query intention u and knowledge node The embedding vector The similarity calculation function between is the similarity threshold;

[0164] According to the search result set The hierarchical structure of the hierarchical multimodal knowledge graph generates multi-angle search feedback for users, and generates functions through the results Build feedback content:

[0165] ;

[0166] in, To provide feedback to users on search results, Knowledge Node The importance weight of Knowledge Node Detailed information about the node, including its properties, relationships, and hierarchy.

[0167] On the basis of the above embodiment, in this embodiment, the user's natural language query content q is parsed based on the ChatGLM model, the key semantic elements in the query are extracted by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity of the embedding space, and the user's query intention is identified. Based on the parsed query intention and key semantic elements, the knowledge nodes related to the user's needs are located in the hierarchical multimodal knowledge graph, including:

[0168] The query parsing module of the ChatGLM model converts the user’s natural language query q into a query intent embedding vector and a set of key semantic elements :

[0169]

[0170] ;

[0171] in, is the weight of the key semantic element, is the i-th key semantic element Relevance scoring in hierarchical multimodal knowledge graphs, is the correlation threshold;

[0172] Combine hierarchical multimodal knowledge graph to calculate the user's query intention embedding vector and knowledge nodes in hierarchical multimodal knowledge graphs The embedding vector The semantic similarity of , construct the similarity score matrix S:

[0173] ;

[0174] in, Embedding vectors for query intent With knowledge nodes The embedding vector The weighted similarity score of is the semantic alignment module, and is the weight factor;

[0175] Generate candidate node set :

[0176] ;

[0177] in, is the similarity threshold, which is used to filter knowledge nodes related to the query requirements. Represents a set of semantic elements, where the semantic elements are associated with nodes in the knowledge graph. The meaning of the whole formula is to filter out such knowledge nodes from the knowledge graph , the similarity measure of these nodes Greater than or equal to the similarity threshold , and for all semantic elements related to the node All of them belong to the semantic element set. Composition Collection .

[0178] After performing an intent query on the user's semantics, a set of key semantic elements is filtered through text filters. To perform credibility scoring and filtering:

[0179] ;

[0180] ;

[0181] in, Representing semantic elements and knowledge nodes The strength of the relationship between is the credibility score threshold, is the filtered set of valid key semantic elements;

[0182] Expand the set of candidate nodes screened through the hierarchical structure of the hierarchical multimodal knowledge graph , generate a context extension node set :

[0183] ;

[0184] in, Knowledge Node and The semantic relationship type of To hierarchically correspond to the levels of knowledge nodes in the multimodal knowledge graph, so that the extended nodes are relevant to the user query at the graph level;

[0185] The candidate node set and the expanded node collection Merge to generate the final set of related nodes :

[0186] ;

[0187] in, For knowledge and The association path between them is used to preserve the extended semantic relationship.

[0188] Combine the final set of related nodes Generates a user response within limits.

[0189] The present invention adopts a multi-layer screening mechanism to parse and filter user query intentions from multiple dimensions such as semantic similarity and credibility score, solving the problem that traditional technologies cannot meet the query requirements in complex fields, allowing users to quickly obtain accurate information that meets the design context. A feedback mechanism based on intelligent filtering is designed, which dynamically filters the credibility of user query content and limits the scope of the model's answer, effectively reducing the risk of the system generating hallucinatory facts. Unlike traditional methods that are prone to inaccurate answers or beyond the scope of knowledge, the present invention can verify and expand the context of query results in combination with the hierarchical structure of the knowledge graph to ensure that the returned search content is authentic and reliable.

[0190] In order to verify the feasibility and effectiveness of the present invention in ship design knowledge retrieval, a specific scenario is used below to explain in detail the operation process and comparison results of the multimodal knowledge graph enhanced ChatGLM model in practical application.

[0191] In a large ship design company, an engineer is participating in the preliminary design of a offshore transport vessel. As the vessel needs to operate in complex sea conditions, the engineer needs to comprehensively consider the fatigue resistance of the hull material, the fluid mechanics performance and the design specifications that meet the latest standards of international classification societies. Faced with a vast amount of design documents, simulation data and historical project cases, the engineer hopes to quickly find the most suitable design parameters and reference standards. However, the traditional keyword search method cannot understand his specific needs, and the returned results are neither comprehensive nor accurate. The engineer has to screen them one by one, which greatly reduces the design efficiency.

[0192] In order to solve the above problems, the method of the present invention can be used to support ship design knowledge retrieval. By introducing multimodal knowledge graph and ChatGLM model, the company has built an intelligent retrieval system covering the field of ship design.

[0193] The engineer entered a query into the design system: "What lightweight, fatigue-resistant materials are suitable for offshore transport vessel hulls? They need to meet relevant ISO standards and provide simulation data references for fluid mechanics performance." The system immediately started working.

[0194] In the first step, the system parses the engineer's natural language query through the ChatGLM model to generate a query intent vector and a set of key semantic elements. "Offshore transport ship", "lightweight material", "fatigue resistance" and "ISO standard" are extracted as key semantic elements. By comparing with the multimodal knowledge graph, the system determines that the keywords belong to four different knowledge node categories: "design type", "material characteristics", "design requirements" and "standard specifications".

[0195] In the second step, the system calculates the semantic similarity between the engineer's query intention and each node in the knowledge graph. Through semantic embedding and modal alignment, the system locates multiple relevant nodes, including lightweight high-strength steel, aluminum alloy, composite material nodes, and ISO 12215-5 standard and fluid simulation data set nodes. The association paths between nodes are dynamically expanded to cover material performance parameters, applicability descriptions of standard specifications, and related historical design cases.

[0196] In the third step, the system filters the returned results and eliminates nodes with low relevance to the query requirements through text filters, such as material information for deep-sea transport ships and standard specifications that are not related to near-sea transport, to ensure the accuracy of the feedback content.

[0197] In the fourth step, the system generates feedback content and presents it to the engineer in a multi-level form:

[0198] List of lightweight materials: including high-strength steel (density of 7.8g / cm³, fatigue resistance of 120 MPa), aluminum alloy (density of 2.7g / cm³, fatigue resistance of 90 MPa), and carbon fiber composite materials (density of 1.8g / cm³, fatigue resistance of 150MPa).

[0199] Standard Specification: Lists specific sections of the ISO 12215-5 standard, including fatigue strength test requirements and material selection principles.

[0200] Fluid mechanics performance: Provide fluid simulation data, including the resistance curve of the aluminum alloy hull and the fluid flow field diagram.

[0201] Historical Cases: Two offshore transport ship design examples are listed, along with their main material parameters and simulation performance data.

[0202] To verify the effect of the present invention, the system conducted a comparative experiment in a real design scenario. The comparative experimental results in knowledge retrieval are shown in Table 1.

[0203] Table 1 Comparative experimental results of the proposed method and the traditional method in ship design knowledge retrieval

[0204]

[0205] The comparative test results show that under the same query conditions, the method of the present invention can significantly improve the retrieval efficiency and accuracy. For example, for the query "selection of lightweight materials for hull", only 12 of the 30 results returned by the traditional method are relevant to the query requirements, while the method of the present invention returns 23 of the 25 results within 15 seconds, and the automatically generated knowledge association graph greatly improves the user's understanding and use efficiency of the results.

[0206] Experimental results show that the present invention significantly improves the efficiency and accuracy of knowledge retrieval in complex ship design scenarios, verifies the system's advantages in multimodal knowledge fusion, professional semantic analysis and intelligent feedback, provides engineers with efficient and reliable technical support, and greatly improves work efficiency and design quality.

[0207] The ship design knowledge retrieval system based on the multimodal knowledge graph provided by the present invention is described below. The ship design knowledge retrieval system based on the multimodal knowledge graph described below and the ship design knowledge retrieval method based on the multimodal knowledge graph described above can be referenced to each other.

[0208] like Figure 3 As shown, the system includes:

[0209] The processing module 301 is used to collect and construct a multi-modal ship design set D from the field of ship design, and perform standardization processing on the ship design set D;

[0210] The construction module 302 is used to perform deep semantic analysis on the standardized ship design set D', generate a structured knowledge node set N, each knowledge node represents a specific concept, component, material attribute or design standard in ship design, establish a semantic association relationship set R between the knowledge nodes, and construct a hierarchical multimodal knowledge graph G according to the knowledge node set N and the semantic association relationship set R;

[0211] The integration module 303 is used to represent the ship design sets of different modes in the hierarchical multimodal knowledge graph G as embedded vectors in a unified embedding space, and to construct a set of semantic association relationships between the embedded vectors of different modal data. , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain the hierarchical multimodal knowledge graph ;

[0212] The first retrieval module 304 is used to convert the hierarchical multimodal knowledge graph The ChatGLM model is integrated into the ChatGLM model, and the specialized terms and common expressions in the field of ship design are added to the ChatGLM model. Based on the ChatGLM model, the hierarchical multimodal knowledge graph is searched according to the user's query intention. Conduct retrieval and generate multi-angle retrieval feedback for users based on the retrieval results;

[0213] The second retrieval module 305 is used to parse the user's natural language query content q based on the ChatGLM model, extract the key semantic elements in the query by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity of the embedding space, and identify the user's query intention, and locate the knowledge nodes related to the user's needs in the hierarchical multimodal knowledge graph based on the parsed query intention and key semantic elements. ;

[0214] Feedback module 306 is used to obtain the knowledge node Relevant ship design knowledge is fed back to the user in a hierarchical form.

[0215] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a ship design knowledge retrieval method based on a multimodal knowledge graph, the method comprising: collecting and constructing a ship design set from the field of ship design; forming a hierarchical multimodal knowledge graph describing concepts, physical properties and design specifications in the field of ship design; enhancing the association between different modal data by vector similarity calculation; integrating the hierarchical multimodal knowledge graph into the ChatGLM model; locating knowledge nodes related to user needs in the hierarchical multimodal knowledge graph; matching and locating ship design knowledge related to user query content, and feeding back the ship design knowledge obtained by intelligent semantic retrieval to the user in a hierarchical form.

[0216] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0217] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the ship design knowledge retrieval method based on a multimodal knowledge graph provided by the above methods, the method including: collecting and constructing a ship design set from the ship design field; forming a hierarchical multimodal knowledge graph that describes concepts, physical properties and design specifications in the ship design field; enhancing the association between different modal data through vector similarity calculation; integrating the hierarchical multimodal knowledge graph into the ChatGLM model; locating knowledge nodes related to user needs in the hierarchical multimodal knowledge graph; matching and locating ship design knowledge related to user query content, and feeding back the ship design knowledge obtained by intelligent semantic retrieval to the user in a hierarchical form.

[0218] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the ship design knowledge retrieval method based on a multimodal knowledge graph provided by the above-mentioned methods, the method comprising: collecting and constructing a ship design set from the field of ship design; forming a hierarchical multimodal knowledge graph that describes concepts, physical properties and design specifications in the field of ship design; enhancing the association between data of different modalities through vector similarity calculation; integrating the hierarchical multimodal knowledge graph into the ChatGLM model; locating knowledge nodes related to user needs in the hierarchical multimodal knowledge graph; matching and locating ship design knowledge related to user query content, and feeding back the ship design knowledge obtained by intelligent semantic retrieval to the user in a hierarchical form.

[0219] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0220] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship design knowledge retrieval method based on multimodal knowledge graph, characterized in that: include: Collect and construct a multimodal ship design set D from the field of ship design, and standardize the ship design set D; Perform deep semantic analysis on the standardized ship design set D' to generate a structured knowledge node set N, each knowledge node represents a specific concept, component, material property or design standard in ship design, establish a semantic association relationship set R between the knowledge nodes, and construct a hierarchical multimodal knowledge graph G based on the knowledge node set N and the semantic association relationship set R; The ship design set of different modes in the hierarchical multimodal knowledge graph G is represented as an embedding vector in a unified embedding space, and a set of semantic association relationships between the embedding vectors of different modal data is constructed. , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain the hierarchical multimodal knowledge graph ; Hierarchical Multimodal Knowledge Graph The ChatGLM model is integrated into the ChatGLM model, and the specialized terms and common expressions in the field of ship design are added to the ChatGLM model. Based on the ChatGLM model, the hierarchical multimodal knowledge graph is searched according to the user's query intention. Conduct retrieval and generate multi-angle retrieval feedback for users based on the retrieval results; Based on the ChatGLM model, the user's natural language query content q is parsed, and the key semantic elements in the query are extracted by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity in the embedding space, and the user's query intention is identified. Based on the parsed query intention and key semantic elements, the knowledge nodes related to the user's needs are located in the hierarchical multimodal knowledge graph. ; Acquisition and Knowledge Nodes Related ship design knowledge, feeding back the ship design knowledge to the user in a hierarchical form; Based on the ChatGLM model, the user's natural language query content q is parsed, and the key semantic elements in the query are extracted by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity in the embedding space, and the user's query intention is identified. Based on the parsed query intention and key semantic elements, the knowledge nodes related to the user's needs are located in the hierarchical multimodal knowledge graph, including: The query parsing module of the ChatGLM model converts the user’s natural language query q into a query intent embedding vector and a set of key semantic elements : ； in, is the weight of the key semantic element, is the i-th key semantic element Relevance scoring in hierarchical multimodal knowledge graphs, is the correlation threshold; Calculate the user's query intent embedding vector and knowledge nodes in hierarchical multimodal knowledge graphs The embedding vector The semantic similarity of , construct the similarity score matrix S: ； in, Embedding vectors for query intent With knowledge nodes The embedding vector The weighted similarity score of is the semantic alignment module, and is the weight factor; Generate candidate node set : ； in, is the similarity threshold, which is used to filter knowledge nodes related to the query requirements. Represents a set of semantic elements, where the semantic elements are associated with nodes in the knowledge graph; By extracting the knowledge nodes from the knowledge graph Filter out the weighted similarity scores Greater than or equal to the similarity threshold , and the embedding vector Belongs to the semantic element set Knowledge Node constitute; After performing an intent query on the user's semantics, a set of key semantic elements is filtered through text filters. To perform credibility scoring and filtering: ； ； in, Representing semantic elements and knowledge nodes The strength of the relationship between is the credibility score threshold, is the filtered set of valid key semantic elements; Expand the set of candidate nodes screened through the hierarchical structure of the hierarchical multimodal knowledge graph , generate a context extension node set : ； in, Knowledge Node and The semantic relationship type of To hierarchically correspond to the levels of knowledge nodes in the multimodal knowledge graph, so that the extended nodes are relevant to the user query at the graph level; The candidate node set and the expanded node collection Merge to generate the final set of related nodes : ； in, For knowledge and The association path between them is used to preserve the extended semantic relationship.

2. The ship design knowledge retrieval method based on multimodal knowledge graph according to claim 1 is characterized in that: Collect and construct a multimodal ship design set from the field of ship design, and standardize the ship design set, including: The original data in the field of ship design are classified according to the source, and a text data set T, an image data set I, a CAD model data set C and a three-dimensional simulation data set S are obtained; Extract data from each modal data set through the data acquisition interface, and construct a ship design set D based on multiple modal data sets, D = {T, I, C, S}; The text data set T in the ship design set D is cleaned to remove non-design related information and the format is unified into a standardized structure encoded in UTF-8; The standardization of the image data set I in the ship design set D includes size normalization and adjusting each image to a uniform resolution; The standardization process of the CAD model data set C in the ship design set D includes converting CAD files of different formats into a unified geometric description format; The standardized processing of the three-dimensional simulation data set S in the ship design set D includes extracting numerical calculation results and generating metadata records corresponding to the simulation scenarios; The ship design set D' after standardized processing is expressed as D' = {T', I', C', S'}, where T' is the standardized processing result of the text data set T, I' is the standardized processing result of the image data set I, C' is the standardized processing result of the CAD model data set C, and S' is the standardized processing result of the three-dimensional simulation data set S.

3. The ship design knowledge retrieval method based on multimodal knowledge graph according to claim 2 is characterized in that: The text data set T includes ship design specifications, design instructions and technical reports, the image data set I includes ship design sketches, pictures and structural schematics, the CAD model data set C includes computer-aided design models that describe the structure of ship components in three-dimensional space, and the three-dimensional simulation data set S includes simulation calculation results.

4. The ship design knowledge retrieval method based on multimodal knowledge graph according to claim 2 is characterized in that: A deep semantic analysis is performed on the standardized ship design set D' to generate a structured knowledge node set N, including: Through the semantic embedding function, each text data in the text data set T' is mapped into a semantic feature vector to obtain a text feature set ; Using a deep convolutional neural network as the feature extraction function, each image data in the image data set I' is extracted as a feature vector to obtain the image feature set ; Through the geometric feature extraction function, each CAD model in the CAD model data set C' is extracted as a geometric feature vector to obtain the CAD model feature set ; Through the simulation result feature extraction function, each simulation data in the three-dimensional simulation data set S' is extracted as a simulation feature vector to obtain a simulation feature set ; The extracted text feature set , image feature set , CAD model feature set and simulation feature sets Mapped to a structured knowledge node set N, the mth knowledge node Defined as: ； in, is the unique identifier of the mth knowledge node, is the node type of the mth knowledge node, representing a concept, component, material property, or design standard. is the feature vector corresponding to the mth knowledge node, taken from the text feature set , image feature set , CAD model feature set and simulation feature sets one of the, is the attribute set of the mth knowledge node, including the semantic description and metadata information of the knowledge node.

5. The ship design knowledge retrieval method based on multimodal knowledge graph according to claim 4 is characterized in that: Establishing a semantic association relationship set R between the knowledge nodes, and constructing a hierarchical multimodal knowledge graph G according to the knowledge node set N and the semantic association relationship set R, including: Determine the association degree between the knowledge nodes according to the feature vectors of the knowledge nodes, and generate an association relationship set R according to the association degree between the knowledge nodes: ； in, are the mth and nth knowledge nodes respectively, Knowledge Node The semantic relationship types between them include inclusion, similarity, and dependency. Knowledge Node The correlation between: ； in, Knowledge Node The eigenvector of With knowledge nodes The eigenvector of The similarity function between them is calculated using cosine similarity. According to the semantic relationship type The weight factor assigned, Z is the normalization coefficient; A hierarchical multimodal knowledge graph G is constructed according to the knowledge node set N and the semantic association relationship set R: G = (N, R); Among them, the hierarchical structure of the multimodal knowledge graph is based on the semantic relationship types between knowledge nodes. and relevance Organize and form a hierarchical multimodal knowledge graph that describes the concepts, physical properties and design specifications in the field of ship design.

6. The ship design knowledge retrieval method based on multimodal knowledge graph according to claim 1 is characterized in that: Construct a set of semantic association relationships between embedding vectors of different modal data , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain a hierarchical multimodal knowledge graph : Calculate the semantic similarity between the embedding vectors of different modal data and obtain the similarity matrix M: ； in, for and The similarity score between and They represent the embedding vector of the i-th data of modality x and the embedding vector of the j-th data of modality y, respectively. is the modulus of the vector, For the modal alignment module, is the alignment factor; Generate a set of cross-modal semantic association relations based on the similarity matrix M : ； in, is the threshold value, for and The semantic relationship of The semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain a hierarchical multimodal knowledge graph : 。 7. The ship design knowledge retrieval method based on multimodal knowledge graph according to claim 1 is characterized in that: Hierarchical Multimodal Knowledge Graph Integrated into the ChatGLM model, and added specialized terms and common expressions in the field of ship design to the ChatGLM model, including: Using graph fusion function to transform hierarchical multimodal knowledge graph Fusion with the semantic space of ChatGLM to update the parameters of the ChatGLM model : ； in, is the ChatGLM model parameter after fusion, are the original ChatGLM model parameters, is the fusion coefficient, which is used to control the influence of the hierarchical multimodal knowledge graph on the update of the ChatGLM model. is the graph fusion function; The domain vocabulary embedding function is used to collect the specialized terms in the field of ship design. and a collection of common expressions Mapped to a set of embedding vectors : ； in, For specialized terms or common expressions The embedding vector of It is a domain vocabulary embedding function, which is used to capture the semantic features unique to the field of ship design and integrate proprietary terms and common expressions into the embedding space of the ChatGLM model; The original vocabulary V of the ChatGLM model is combined with the set of special terms , Common expressions collection Merge, build an enhanced vocabulary V', and use the embedding vector set Update the embedding matrix E' of the ChatGLM model.

8. The ship design knowledge retrieval method based on multimodal knowledge graph according to claim 1 is characterized in that: Based on the ChatGLM model, the hierarchical multimodal knowledge graph is constructed according to the user's query intention. Conduct a search and generate multi-angle search feedback for users based on the search results, including: Define query functions for multi-level semantic relationships using the semantic structure of hierarchical multimodal knowledge graphs , get the retrieval result set according to the user's query intention u : ； in, is the search result set, is the embedding vector of the user’s query intent, Embed functions for user queries, is the i-th knowledge node The embedding vector of The user's query intention u and knowledge node The embedding vector The similarity calculation function between is the similarity threshold; According to the search result set The hierarchical structure of the hierarchical multimodal knowledge graph generates multi-angle search feedback for users, and generates functions through the results Build feedback content: ； in, To provide feedback to users on search results, Knowledge Node The importance weight of Knowledge Node Detailed information about the node, including its properties, relationships, and hierarchy.

9. A ship design knowledge retrieval system based on multimodal knowledge graph, characterized in that: The ship design knowledge retrieval method based on a multimodal knowledge graph applied to any one of claims 1 to 8 comprises: A processing module, used for collecting and constructing a multimodal ship design set D from the field of ship design, and performing standardization processing on the ship design set D; A construction module is used to perform deep semantic analysis on the standardized ship design set D', generate a structured knowledge node set N, each knowledge node represents a specific concept, component, material property or design standard in ship design, establish a semantic association relationship set R between the knowledge nodes, and construct a hierarchical multimodal knowledge graph G based on the knowledge node set N and the semantic association relationship set R; Integration module, used to represent the ship design set of different modes in the hierarchical multimodal knowledge graph G as embedding vectors in a unified embedding space, and to construct a set of semantic association relationships between the embedding vectors of different modal data , the semantic association relationship set Integrate into the hierarchical multimodal knowledge graph G to obtain the hierarchical multimodal knowledge graph ; The first retrieval module is used to transform the hierarchical multimodal knowledge graph The ChatGLM model is integrated into the ChatGLM model, and the specialized terms and common expressions in the field of ship design are added to the ChatGLM model. Based on the ChatGLM model, the hierarchical multimodal knowledge graph is searched according to the user's query intention. Conduct retrieval and generate multi-angle retrieval feedback for users based on the retrieval results; The second retrieval module is used to parse the user's natural language query content q based on the ChatGLM model, extract the key semantic elements in the query by combining the semantic structure in the hierarchical multimodal knowledge graph and the semantic similarity in the embedding space, and identify the user's query intention. Based on the parsed query intention and key semantic elements, the knowledge nodes related to the user's needs are located in the hierarchical multimodal knowledge graph. ; Feedback module, used to obtain and knowledge nodes Relevant ship design knowledge is fed back to the user in a hierarchical form.

Images