Maritime accident knowledge graph quality evaluation and optimization method based on large model

Abstract

The invention discloses a maritime accident knowledge graph quality evaluation and optimization method based on a large model, which comprises the following steps: firstly, acquiring a maritime accident report, screening maritime accident report content, taking accident summary, accident causes and accident training as texts, and then extracting a knowledge graph from the screened texts according to an established knowledge graph extraction mode. And performing entity recognition and relation extraction by adopting a triple extraction large model, and constructing a maritime accident knowledge graph. The entity redundancy, the relation redundancy, the structure redundancy and the reconstruction text accuracy serve as evaluation indexes, quality evaluation is conducted on the constructed maritime accident knowledge graph, corresponding optimization measures are selected according to the difference between an evaluation value and a target value, and iterative optimization is conducted on the maritime accident knowledge graph. According to the method, comparable and reproducible quality diagnosis is realized, optimization measures are automatically retrieved based on evaluation results, a closed-loop mechanism of'evaluation-optimization-reevaluation 'is formed, and the optimization effect can be quantified.

Description

Technical Field

[0001] This application belongs to the field of knowledge graph evaluation and optimization technology, and in particular relates to a method for quality evaluation and optimization of maritime accident knowledge graphs based on large models. Background Technology

[0002] Maritime accidents are influenced by multiple factors, including severe weather, vessel defects, human error, and mismanagement, exhibiting a high degree of uncertainty and typically causing multi-dimensional impacts on life, property, and the environment. Maritime accident information is usually stored in the form of accident investigation reports, characterized by sparse and non-uniform temporal distribution, high dimensionality, and unstructured nature. While these reports are rich in information, their narrative nature and inclusion of numerous domain-specific terms increase the difficulty of data utilization.

[0003] Knowledge graphs can transform unstructured text into structured semantic networks, which can be used to characterize accident processes, identify causal chains, and apply to the quantitative assessment of maritime safety risks. However, existing knowledge graphs for the maritime domain often lack systematic quality evaluation and improvement mechanisms, resulting in problems such as redundant entities and relationships, structurally invalid connections, and inconsistent expressions of key facts, affecting their reliability and usability in large-scale corpora and downstream applications.

[0004] Existing knowledge graph evaluation technologies typically assess the accuracy of extraction modules using conventional methods, such as recall, precision, and F1 scores for sentences and fragments. At the graph optimization level, external knowledge bases are often used during the construction process for entity linking and alias resolution, or manual sampling is performed based on templates. These approaches have two significant shortcomings in terms of the systematic nature and operable improvement mechanisms for quality assessment: First, the evaluation metrics are insufficient. Existing evaluation technologies focus on sentence-level or fragment-level extraction accuracy, lacking quantitative metrics for graph structure and application-level evaluations such as the matching degree between the graph and maritime safety analysis scenarios. Second, the evaluation and optimization methods are highly dependent on manual intervention. Data annotation and verification require personnel with maritime business knowledge and the ability to interpret technical language, making it difficult to update the knowledge graph in real time.

[0005] Therefore, to apply knowledge graphs to large-scale maritime accident corpora, it is urgent to establish a quantifiable, configurable, and re-evaluable assessment system tailored to the characteristics of this field, and to form a corresponding iterative improvement path and version-based governance framework. Summary of the Invention

[0006] The purpose of this application is to provide a method for quality assessment and optimization of maritime accident knowledge graphs based on large models, aiming to solve the problems in the existing technology, such as the reliance on a large amount of manual annotation for maritime accident prediction, high cost and difficulty in expansion; lack of quantitative quality assessment methods for knowledge graphs in the field of maritime accidents; and lack of operable graph iterative optimization mechanism.

[0007] To achieve the above objectives, the technical solution of this application is as follows:

[0008] A method for quality assessment and optimization of maritime accident knowledge graphs based on large models, including:

[0009] Obtain maritime accident reports, filter the content of maritime accident reports, and include the accident summary, causes of the accident, and lessons learned as the main text;

[0010] Based on the established knowledge graph extraction model, a large triplet extraction model is used to perform entity recognition and relation extraction from the selected text to construct a maritime accident knowledge graph.

[0011] The quality of the constructed maritime accident knowledge graph is evaluated using entity redundancy, relation redundancy, structural redundancy, and reconstructed text accuracy as evaluation indicators.

[0012] Based on the gap between the assessed value and the target value, corresponding optimization measures are selected to iteratively optimize the maritime knowledge graph.

[0013] Preferably, the formula for calculating the entity redundancy is as follows:

[0014]

[0015] in, Indicates entity redundancy. Let i represent the set of candidate same-reference entities. Represents the set of candidate referential entities Number of entities in the middle.

[0016] Preferably, if the similarity between the entity embeddings of two entities is greater than a threshold, they are considered as candidate entities with the same reference.

[0017] Preferably, the formula for calculating relation redundancy is as follows:

[0018]

[0019] in, This represents the number of self-loop relations in a maritime accident knowledge graph. This indicates the number of repeated parallel edges. This represents the total number of relationships in the maritime accident knowledge graph.

[0020] Preferably, the formula for calculating structural redundancy is as follows:

[0021]

[0022] in, Indicates structural redundancy. This represents the overall information entropy of the maritime accident knowledge graph. This represents the graph information entropy of the main chain.

[0023] As a preferred approach, a set of relationships related to event development and accident causes is selected from the maritime accident knowledge graph. ,Depend on The subgraphs connected by the relationships in the graph are called the main graph, and their graph information entropy is the graph information entropy of the main graph.

[0024] Preferably, the formula for calculating the accuracy of the reconstructed text is as follows:

[0025]

[0026] in, This indicates the accuracy of the reconstructed text, where n represents the total number of sentences in the original accident report. This represents the maximum similarity between the j-th sentence of the original report and the restored text.

[0027] This application proposes a method for quality assessment and optimization of maritime accident knowledge graphs based on a large-scale model. Through automatic assessment and optimization driven by the large-scale model, it significantly reduces reliance on manual annotation and expert review, enabling rapid adaptation and incremental updates to large-scale corpora of maritime accident reports. It proposes a unified standard for graph quality indicators tailored to maritime accident scenarios, achieving comparable and reproducible quality diagnosis. Based on the assessment results, it automatically retrieves optimization measures, forming a closed-loop mechanism of "assessment—optimization—reassessment," and the optimization effect can be quantified. Attached Figure Description

[0028] Figure 1 Methodology and Optimization Process for Maritime Accident Knowledge Graph Based on Large Model Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] One embodiment of this application, such as Figure 1 As shown, a method for quality assessment and optimization of maritime accident knowledge graphs based on large models is proposed, including:

[0031] Step S1: Obtain the maritime accident report, filter the content of the maritime accident report, and include the accident summary, the cause of the accident, and the lessons learned from the accident as the main text.

[0032] First, obtain maritime accident reports (e.g., maritime accident investigation reports related to extreme weather issued by a country's maritime accident investigation bureau from 1990 to 2023). Each accident corresponds to one report. Using keyword and tag filtering, the accident summary, accident causes, and accident lessons are selected as the main text and finally saved as a txt file.

[0033] Step S2: Based on the established knowledge graph extraction mode, the triplet extraction model is used to perform entity recognition and relationship extraction from the selected text to construct a maritime accident knowledge graph.

[0034] A knowledge graph extraction schema is the foundation for building a knowledge graph. It defines entities, attributes, relationships, and the structure between them. This embodiment sets the application scenario of the knowledge graph as the causal analysis of maritime accidents related to extreme weather. The goal is to use a knowledge graph to structurally represent all risk factors and their potential relationships recorded in reports of maritime accidents related to extreme weather. The knowledge graph extraction schema, as shown in Table 1, includes four concepts: ships, weather, hazardous events, and accidents. Each concept contains entity types and entity attributes, and relationships exist between entity types.

[0035] Table 1

[0036]

[0037]

[0038] Then, based on the knowledge graph extraction patterns in Table 1, the triplet extraction model (Qwen2.5-7B-Instruct) was used for entity recognition and relation extraction. Each task was implemented using one-shot prompts. Examples of prompts for named entity recognition are shown in Table 2, and the prompts for the remaining steps adopted the same structure as in Table 2.

[0039] Table 2

[0040]

[0041] After entity recognition and relation extraction, a maritime accident knowledge graph can be constructed, which will not be elaborated here.

[0042] Step S3: Use entity redundancy, relation redundancy, structural redundancy, and reconstructed text accuracy as evaluation indicators to conduct a quality assessment of the constructed maritime accident knowledge graph.

[0043] In this embodiment, four evaluation metrics are set for the structural redundancy and accuracy of the knowledge graph: entity redundancy. Relationship redundancy Structural redundancy Reconstructing text accuracy The specific calculation methods are as follows:

[0044] Entity redundancy : Refers to the proportion of multiple references to the same real entity object in the knowledge graph.

[0045] In this embodiment, the cosine similarity of entity embeddings is first calculated for entities in the graph, and then the similarity is used to... A threshold of 0.9 is used to determine whether two entities are candidate entities that refer to each other. If the similarity is greater than the threshold, they are considered candidate entities and added to the candidate entity set. Where i represents the set of the i-th candidate referential entities, if This is then denoted as a redundant entity set. In this embodiment, text embedding is performed on the entity names to obtain the entity embeddings. Text embedding can be achieved using a large embedding model (bge-m3), which will not be elaborated here.

[0046] Entity redundancy The calculation method is as follows:

[0047] ;

[0048] in, Let i represent the set of candidate same-reference entities. Represents the set of candidate referential entities Number of entities in the middle This represents the size of the set or the number of elements in the set.

[0049] Relationship redundancy This refers to the proportion of invalid edges in the total number of relations in a knowledge graph.

[0050] The set of relationships in a knowledge graph is For the first entity Second Entity Relationship The relationship is a point from the first entity to the second entity. Invalid edges are counted for the following two types: the first type is the first entity in the triplet. Second Entity They are the same entity, i.e., the number of self-loop relations. The second category is for ordered entity pairs ( ) ( There are repeated parallel edges between them, let's call them... The number of repeated parallel edges is denoted as . Relationship redundancy The calculation method is as follows:

[0051] ;

[0052] in, Indicates relation redundancy. This represents the number of self-loop relations in a maritime accident knowledge graph. This indicates the number of repeated parallel edges. This represents the total number of relationships in the maritime accident knowledge graph.

[0053] Structural redundancy (SR) refers to the proportion of the subgraph information entropy corresponding to minor information in the knowledge graph of an accident report to the total information entropy of the entire accident graph.

[0054] In this embodiment, a set of relationships related to event development and accident causes is selected from the maritime accident knowledge graph. ={"cause", "exhibit a response", "execute"}, by The subgraph formed by the connections between the main graphs is called the main chain, and its graph information entropy is denoted as . Information outside the main chain is secondary information. The graph information entropy of the knowledge graph corresponding to the entire accident report is denoted as... . This refers to the subgraph information entropy corresponding to secondary information.

[0055] Structural redundancy The calculation method is as follows:

[0056]

[0057] in, Indicates structural redundancy. This represents the overall information entropy of the maritime accident knowledge graph. This represents the graph information entropy of the main chain.

[0058] The definition and calculation method of graph information entropy are as follows:

[0059]

[0060] in, The information entropy of a graph or subgraph. This represents the total number of nodes in the graph or subgraph. Representing the The normalization degree of each node. For the first The degree of each node.

[0061] In this embodiment, the knowledge graph subgraph corresponding to each incident document is first calculated. The average of all structural redundancies is taken as the final value. .

[0062] Reconstructed text precision (RTA) refers to extracting triples from each accident report, semantically reconstructing them into a complete description of the maritime accident process, then evaluating the similarity between the reconstructed text and the original document sentence by sentence, and finally integrating the results to obtain the reconstructed text precision.

[0063] In this embodiment, the accident text reconstruction is achieved using the specified large-scale text reconstruction model (DeepSeek-R1-Distill-Qwen-7B). The large-scale model selected has reasoning capabilities, enabling it to logically expand knowledge graph triples into accident documents. One-shot prompts are used, and examples of text reconstruction prompts are shown in Table 3. To eliminate the influence of narrative word order, the fidelity of the reconstructed text to the original text is compared sentence by sentence. The two texts are divided into clauses according to punctuation marks, and the set of clauses in the reconstructed text is denoted as . The set of clauses in the original text is denoted as Using the embedding model (bge-m3) specified in steps S01-2, all restored text sentences and original text sentences are converted into text embedding vectors of dimension 1024. , The semantic similarity of each sentence in the original text is compared with that of all sentences in the restored text. For each original clause... The maximum similarity between them in the restored text is defined as:

[0064]

[0065] The final average similarity, or the accuracy of the restored text, is:

[0066]

[0067] in, This indicates the accuracy of the reconstructed text, where n represents the total number of sentences in the original accident report. This represents the maximum similarity between the j-th sentence of the original report and the restored text.

[0068] In this embodiment, the accuracy of the restored text is first calculated for each accident report, and the average, maximum, minimum and standard deviation are recorded.

[0069] Table 3

[0070]

[0071] Quality assessment begins with setting target values ​​for the indicators. Optionally, the performance of the historical knowledge graph on each indicator can be benchmarked, and historical scores can be set as target values. If no historical scores exist, initial target values ​​can be manually set, including the redundancy indicator. , and The value ranges from 0 to 1; a lower value indicates lower redundancy. A target can be set. , , Precision index The range is 0 to 1, with higher values ​​indicating higher accuracy in text reconstruction. A target value can be set. At the same time, a tolerance level for degradation is set for all evaluation indicators, which is the maximum acceptable percentage of degradation in score. .

[0072] Then, an evaluation process is performed to assess the quality of the constructed knowledge graph. Optionally, structural redundancy is considered. and the accuracy of reconstructed text Evaluations and results are saved at the granular level of each incident report.

[0073] Step S4: Based on the difference between the evaluated value and the target value, select the corresponding optimization measures to optimize the maritime knowledge graph.

[0074] This embodiment sets the initial target value for the redundancy index: , , ,right The following are requirements for incident documentation: , , At the same time, degradation tolerance thresholds are set for all indicators, among which... , and The degradation (increase) does not exceed 0.1. The degradation (degradation) does not exceed 0.1.

[0075] The evaluation was conducted, and the quality of the constructed knowledge graph was assessed according to the four indicators mentioned above. A typical evaluation result is shown in Table 4.

[0076] Table 4

[0077]

[0078] After obtaining the evaluation results, this embodiment compares them with the set target values. If the target value is not met, measures are selected from the quality optimization measure library to update the map.

[0079] Optionally, the ratio of the difference between the evaluated value and the target value can be used as the criterion. If the optimization target is not achieved, the 1-2 indicators with the largest difference from the target value are selected as the optimization targets for this round. In this embodiment, by comparing the evaluation results obtained in Table 4 with the set target values, it can be found that two indicators have not been met: entity redundancy (ER) is too high and the minimum accuracy of reconstructed text (RTA) is too low. This round of optimization aims to improve these two indicators.

[0080] In this embodiment, six types of optimization measures are set: A. Pattern modification; B. Entity standardization; C. Relationship merging; D. Time alignment; E. Slot template and mandatory field validation; F. Hint and decoding strategy optimization.

[0081] Comparing the evaluated values ​​with the target values, potential optimization scenarios fall into two categories: single-indicator optimization and dual-indicator combination optimization. Single-indicator optimization scenarios include: excessive entity redundancy, excessive relation redundancy, excessive structural redundancy, and insufficient accuracy in text reconstruction. Dual-indicator combination optimization scenarios include: double redundancy of entities and relations, double redundancy of entities and structure, entity redundancy with insufficient reconstruction accuracy, double redundancy of relations and structure, relation redundancy with insufficient reconstruction accuracy, and structural redundancy with insufficient reconstruction accuracy. The corresponding measures for single-indicator optimization are shown in Table 5. For dual-indicator combination optimization, measures are combined based on the measures corresponding to single-indicator optimization.

[0082] Table 5

[0083]

[0084] Each type of optimization measure has a meta-model, which defines and stores the attributes of the measure, serving as the basis for compiling optimization measures. The meta-model includes the measure's identifier, category, triggering conditions, required resources, adjustable parameters, execution steps, potential impact, review criteria, rollback conditions, and examples. Table 6 shows an example of an optimization measure meta-model using redundant entity merging as an example. All optimization measures are written into the maritime accident knowledge graph quality optimization measure library in meta-model form.

[0085] Table 6

[0086]

[0087] In this embodiment, for ER optimization, the redundancy defects of entities are first located by sorting them according to the size of the redundant entity set. It is found that the redundant entity set contains two types of defects: attribute variants that share core concepts, and synonymous and paraphrased expressions. For TRA optimization, the sample with the lowest text reconstruction accuracy is selected, and its main defects are summarized: factual distortion and missing important details. The defect location and examples described in this step are shown in Table 7.

[0088] Table 7

[0089]

[0090] Based on the triggering conditions and defect types, corresponding measures are selected from the optimization measure library to generate the optimization list for this round, as shown in Table 8.

[0091] Table 8

[0092]

[0093] Following the optimization measures in Table 8, the knowledge graph extraction mode and the knowledge graph were optimized, and then a quality review was conducted. The difference in indicators after the review is shown in Table 9.

[0094] Table 9

[0095]

[0096] Based on the review results, no metrics showed degradation exceeding the tolerance threshold, and no version rollback was necessary. The optimized knowledge graph was logged and versioned. Since the metrics have not yet reached the phased improvement target after this round of optimization, iterative optimization will continue until the target is achieved, at which point a version record will be made.

[0097] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for quality assessment and optimization of maritime accident knowledge graphs based on large models, characterized in that, The method for quality assessment and optimization of maritime accident knowledge graphs based on large models includes: Obtain maritime accident reports, filter the content of maritime accident reports, and include the accident summary, causes of the accident, and lessons learned as the main text; Based on the established knowledge graph extraction model, a large triplet extraction model is used to perform entity recognition and relation extraction from the selected text to construct a maritime accident knowledge graph. The quality of the constructed maritime accident knowledge graph is evaluated using entity redundancy, relation redundancy, structural redundancy, and reconstructed text accuracy as evaluation indicators. Based on the gap between the assessed value and the target value, corresponding optimization measures are selected to iteratively optimize the maritime knowledge graph.

2. The method for quality assessment and optimization of maritime accident knowledge graph based on a large model as described in claim 1, characterized in that, The formula for calculating entity redundancy is as follows: in, Indicates entity redundancy. Let i represent the set of candidate same-reference entities. Represents the set of candidate referential entities Number of entities in the middle.

3. The method for quality assessment and optimization of maritime accident knowledge graph based on a large model as described in claim 2, characterized in that, If the similarity between the entity embeddings of two entities is greater than a threshold, then they are candidate entities that refer to each other.

4. The method for quality assessment and optimization of maritime accident knowledge graph based on a large model as described in claim 1, characterized in that, The formula for calculating relation redundancy is as follows: in, This represents the number of self-loop relations in a maritime accident knowledge graph. This indicates the number of repeated parallel edges. This represents the total number of relationships in the maritime accident knowledge graph.

5. The method for quality assessment and optimization of maritime accident knowledge graph based on a large model as described in claim 1, characterized in that, The formula for calculating structural redundancy is as follows: in, Indicates structural redundancy. This represents the overall information entropy of the maritime accident knowledge graph. This represents the graph information entropy of the main chain.

6. The method for quality assessment and optimization of maritime accident knowledge graph based on a large model as described in claim 5, characterized in that, Select sets of relationships related to event development and accident causes from the maritime accident knowledge graph. ,Depend on The subgraphs connected by the relationships in the graph are called the main graph, and their graph information entropy is the graph information entropy of the main graph.

7. The method for quality assessment and optimization of maritime accident knowledge graph based on a large model as described in claim 1, characterized in that, The formula for calculating the accuracy of the reconstructed text is as follows: in, This indicates the accuracy of the reconstructed text, where n represents the total number of sentences in the original accident report. This represents the maximum similarity between the j-th sentence of the original report and the restored text.

Images