A port facility management and maintenance large model professional knowledge base data labeling method and system

Abstract

The present application provides a port facility management and maintenance large model professional knowledge base data labeling method and system, the method comprises the following steps: collecting multi-source heterogeneous data set, constructing port digital twin three-dimensional model, generating three-dimensional damage body with attribute labeling, fine-tuning with port field corpus. Using expert collaborative review platform to correct and mark, obtain standardized and structured port facility management and maintenance professional knowledge base labeling data. The present application solves the problem of traditional single mode data labeling by collecting data through multi-modal perception network and generating three-dimensional damage body combined with algorithm; constructs and fine-tunes intelligent labeling model, realizes accurate correlation identification, and overcomes the semantic gap of general model; combined with expert platform correction and marking, forming a quantitative evaluation closed loop, reducing labor cost and improving consistency; through structured conversion and association index construction, realize knowledge reasoning guided labeling, sedimentation field rules, improve the processing efficiency of high difficulty samples, meet the core needs of port facility management and maintenance.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, and in particular to a data annotation method and system for a professional knowledge base of a large-scale model for port facility management and maintenance. Background Technology

[0002] Currently, data annotation in port facility maintenance primarily relies on a combination of manual experience and basic automated tools. Traditional methods often employ single-modal data (such as structural damage images or vibration sensor data) for independent analysis, manually identifying damage types and annotating maintenance needs. Some automated systems attempt to integrate multi-source data (such as images, sensor data, and maintenance logs), but these often remain at the level of simple data stitching, lacking deep fusion and semantic association capabilities. For example, 3D model construction typically only achieves geometric shape mapping, failing to effectively integrate dynamic attributes (such as damage evolution trends) and static attributes (such as facility material characteristics). Intelligent annotation models often adopt general architectures without fine-tuning instructions and optimizing parameters specific to the port domain, resulting in limited multimodal understanding capabilities and difficulty in accurately identifying the 3D attribute relationships between facility types, damage types, and maintenance needs. Summary of the Invention

[0003] This invention aims to at least solve the technical problem of the difficulty in accurately identifying the three-dimensional attribute association of facility type, damage type and maintenance needs in the prior art, and innovatively proposes a data annotation method and system for a professional knowledge base of port facility management and maintenance large model.

[0004] To achieve the above-mentioned objectives of this invention, this invention provides a data annotation method for a professional knowledge base of a large-scale model for port facility maintenance, the method comprising:

[0005] S1. Deploy a multimodal sensing network in port facilities to collect structural damage images, vibration sensor data, temperature stress monitoring data, and maintenance log text data to form a spatiotemporally synchronized multi-source heterogeneous dataset.

[0006] S2. Based on the multi-source heterogeneous dataset, construct a port digital twin 3D model. Use point cloud registration algorithm and semantic segmentation model to map the damage detection results in the structural damage image to the port digital twin 3D model to generate a 3D damage body with attribute annotation.

[0007] S3. Construct an intelligent annotation model based on the three-dimensional damage body, and use a port domain corpus to fine-tune instructions and parameters to form an intelligent annotation model with multimodal understanding capabilities.

[0008] S4. Utilize the multimodal understanding capability of the intelligent annotation model to automatically annotate facility types, damage types, and maintenance requirements, and combine this with an expert collaborative review platform to perform semi-automatic annotation correction and uncertainty marking, thereby obtaining standardized and structured port facility maintenance professional knowledge base annotation data.

[0009] S5. The annotation results are quantitatively evaluated by using consistency verification, integrity check and domain adaptability verification to obtain the evaluation results;

[0010] S6. Based on the evaluation results, the unqualified labeled data is returned to the intelligent labeling model for secondary fine-tuning training, and the labeling rules and domain knowledge constraints are supplemented by the manual correction opinions of the expert collaborative review platform for secondary labeling; after the secondary labeling is completed, the quantitative evaluation is re-executed; for the labeled data that still cannot pass the evaluation after multiple corrections, it is marked as a high-difficulty sample and included in the expert knowledge base for special labeling and rule accumulation.

[0011] S7. The qualified labeled data will be structurally transformed according to the entity-relationship-attribute triplet specification of the knowledge graph, and an association index with the port facility management and maintenance model will be established.

[0012] On the other hand, the present invention also provides a data annotation system for a professional knowledge base of a large-scale model for port facility maintenance, the system comprising:

[0013] processor;

[0014] Memory used to store processor-executable instructions;

[0015] The processor is configured to implement the port facility maintenance large model professional knowledge base data annotation method when executing the executable instructions.

[0016] The beneficial effects of this invention are as follows: This invention collects structural damage images, vibration sensor data, temperature stress monitoring data, and maintenance log text data through a multimodal sensing network, forming a spatiotemporally synchronized multi-source heterogeneous dataset. Combining point cloud registration algorithms and semantic segmentation models, it generates attribute-annotated 3D damage volumes, solving the problems of inaccurate damage type identification and broken logic in maintenance requirement reasoning in traditional single-modal data annotation. By constructing an intelligent annotation model including a 3D spatial encoder, attribute reasoning module, and uncertainty quantization unit, and using a low-rank adapter for efficient parameter fine-tuning, it achieves accurate correlation and identification of facility type, damage type, and 3D attributes of maintenance requirements. It overcomes the semantic gap of general models in complex port scenarios; combines an expert collaborative review platform for semi-automatic correction and uncertainty labeling, and forms a quantitative evaluation closed loop through consistency verification, integrity check and domain adaptability verification, which significantly reduces the cost of manual correction and improves the consistency of annotation results; finally, through the structured transformation of knowledge graph entity-relation-attribute triples and the construction of hash association index, it realizes knowledge reasoning-guided annotation of expert knowledge graph and large model, effectively precipitates domain rules and improves the processing efficiency of high-difficulty samples, and overall meets the core requirements of port facility management for high accuracy, high consistency and strong domain adaptability of annotation data.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0019] Figure 1 This is a flowchart of a method for annotating data in a professional knowledge base for a large-scale model of port facility maintenance, according to the present invention. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] Example 1

[0022] like Figure 1 As shown, a data annotation method for a professional knowledge base of a large-scale model for port facility maintenance is described, the method comprising:

[0023] S1. Deploy a multimodal sensing network in port facilities to collect structural damage images, vibration sensor data, temperature stress monitoring data, and maintenance log text data to form a spatiotemporally synchronized multi-source heterogeneous dataset.

[0024] In step S1, it is necessary to elaborate on the deployment of the multimodal sensing network, which must be combined with the actual layout and environmental characteristics of the port facilities to ensure that sensor nodes cover key monitoring areas while avoiding blind spots or redundancy in the data acquisition process. The acquisition of structural damage images relies on high-resolution industrial cameras. By setting fixed shooting angles and frequencies, the consistency and comparability of image data are ensured. Vibration sensors and temperature stress monitoring equipment need to be installed at key stress-bearing parts of the facility, such as bridge bearings, wharf pile foundations, or crane connection points, to capture subtle physical change signals. The acquisition of maintenance log text data can be achieved through a digital system interface, combining historical records with real-time updates to form a complete text data stream. Furthermore, to achieve spatiotemporal synchronization of multi-source heterogeneous data, a unified timestamp protocol needs to be designed, and edge computing devices should be used to perform preliminary cleaning and format conversion of the raw data, thereby improving subsequent processing efficiency and reducing transmission load.

[0025] S2. Based on the multi-source heterogeneous dataset, construct a port digital twin 3D model. Use point cloud registration algorithm and semantic segmentation model to map the damage detection results in the structural damage image to the port digital twin 3D model to generate a 3D damage body with attribute annotation.

[0026] S3. Construct an intelligent annotation model based on the three-dimensional damage body, and use a port domain corpus to fine-tune instructions and parameters to form an intelligent annotation model with multimodal understanding capabilities.

[0027] S4. Utilize the multimodal understanding capability of the intelligent annotation model to automatically annotate facility types, damage types, and maintenance requirements, and combine this with an expert collaborative review platform to perform semi-automatic annotation correction and uncertainty marking, thereby obtaining standardized and structured port facility maintenance professional knowledge base annotation data.

[0028] S5. The annotation results are quantitatively evaluated by using consistency verification, integrity check and domain adaptability verification to obtain the evaluation results;

[0029] S6. Based on the evaluation results, the unqualified labeled data is returned to the intelligent labeling model for secondary fine-tuning training, and the labeling rules and domain knowledge constraints are supplemented by the manual correction opinions of the expert collaborative review platform for secondary labeling; after the secondary labeling is completed, the quantitative evaluation is re-executed; for the labeled data that still cannot pass the evaluation after multiple corrections, it is marked as a high-difficulty sample and included in the expert knowledge base for special labeling and rule accumulation.

[0030] In step S6, it is necessary to explain in detail that for the labeled data that fails the evaluation, a second round of fine-tuning training is first performed using the intelligent labeling model. During this process, the labeling rules and domain knowledge constraints are dynamically supplemented and optimized using correction opinions provided by the expert collaborative review platform (such as newly added domain rules and labeling constraints). Specifically, the correction opinions are transformed into structured rules and embedded into the inference module of the intelligent labeling model to enhance its ability to understand complex scenarios. Simultaneously, for high-difficulty samples that still fail to meet the evaluation criteria after multiple corrections, the system automatically marks them and archives them in the expert knowledge base. These high-difficulty samples will serve as specialized research objects, undergoing in-depth analysis by domain experts, and the expert knowledge graph will be continuously updated based on real-time labeling results, thus forming a closed-loop optimization mechanism. Furthermore, to ensure a balance between labeling efficiency and quality, the system also introduces an adaptive weight adjustment strategy, dynamically allocating computing resources based on data complexity during the secondary labeling process, prioritizing the labeling tasks of key attributes, and ultimately achieving comprehensive optimization and standardized output of the labeled data.

[0031] S7. The qualified labeled data will be structurally transformed according to the entity-relationship-attribute triplet specification of the knowledge graph, and an association index with the port facility management and maintenance model will be established.

[0032] The principle of a data annotation method for a professional knowledge base of a large-scale port facility maintenance model in this embodiment is as follows: First, a multimodal perception network is deployed to comprehensively collect multi-source heterogeneous data of port facilities, including structural damage images, vibration sensor data, temperature stress monitoring data, and maintenance log text data, forming a spatiotemporally synchronized dataset. After preliminary cleaning and format conversion, the damage detection results are mapped to the port digital twin 3D model using point cloud registration algorithms and semantic segmentation models, generating a 3D damage volume with attribute annotations. Based on this 3D damage volume, an intelligent annotation model is constructed, and its instructions and parameters are fine-tuned through a port domain corpus to enable it to have multimodal understanding capabilities. Subsequently, the intelligent annotation model is used to automatically annotate facility types, damage types, and maintenance requirements, while a semi-automatic correction and uncertainty labeling are performed in conjunction with an expert collaborative review platform, ultimately obtaining standardized and structured annotated data. This process not only achieves accurate identification of complex scenarios, but also significantly improves the quality and domain adaptability of the annotated data through expert knowledge graph-guided annotation and rule accumulation.

[0033] The port facility maintenance model is a large model constructed in steps S1 to S7. When the intelligent annotation model is used for annotation, the reasoning ability of the expert knowledge graph can be combined to effectively improve the accuracy and consistency of the annotation.

[0034] As an optional embodiment of the present invention, optionally, in step S2, a port digital twin 3D model is constructed based on the multi-source heterogeneous dataset, and the damage detection results in the structural damage image are mapped to the port digital twin 3D model using a point cloud registration algorithm and a semantic segmentation model to generate a 3D damage body with attribute annotations, including:

[0035] S201. Based on the multi-source heterogeneous dataset, laser point cloud data, UAV aerial survey orthophotos and BIM model framework, an initial three-dimensional model is constructed using three-dimensional reconstruction algorithm and multi-view stereo matching algorithm.

[0036] In step S201, the specific method for constructing the initial 3D model in this embodiment is as follows: A 3D reconstruction algorithm (such as Structure from Motion, SfM) combined with multi-view stereo matching technology is used to first perform multi-level fusion processing on laser point cloud data, UAV aerial survey orthophotos, and the basic framework of the BIM model. By extracting the geometric features of the laser point cloud and the texture information of the UAV imagery, and combining it with the structured semantic information provided by the BIM model, the basic framework of the initial 3D model is generated. During this process, a multi-view stereo matching algorithm (such as dense point cloud generation technology) is used to perform consistency calibration on image data from different perspectives to ensure high-precision alignment of geometric and texture information.

[0037] S202. Based on the initial 3D model, the RANSAC algorithm and FPFH feature descriptor are used to perform coarse registration of the point cloud. The initial transformation matrix between the source point cloud and the target point cloud in the initial 3D model is calculated to perform large-scale spatial alignment and obtain the coarse registration result.

[0038] In step S202: First, using existing FPFH feature descriptors (which can describe the local geometric information of the point cloud and capture the relationship between points and their neighbors), local geometric features of the source and target point clouds are extracted to generate feature description vectors. Then, based on the existing RANSAC algorithm, the best matching point pair is found in the feature description vectors using a random sampling consensus strategy, and an initial transformation matrix is ​​calculated. This matrix is ​​used to achieve large-scale spatial alignment between the source and target point clouds, thereby obtaining coarse registration results. This process effectively handles noise and outliers in the point cloud data, ensuring the robustness and accuracy of the coarse registration results.

[0039] S203. Based on the coarse registration result, perform fine registration of the point cloud using the ICP algorithm to obtain the fine registration result;

[0040] In step S203, the coarse registration result is refined using the Iterative Nearest Point (ICP) algorithm to further optimize the spatial alignment accuracy of the point cloud. Specifically, the relative positions of the source and target point clouds are first initialized based on the coarse registration result. Then, the ICP algorithm is used to calculate the Euclidean distance between point pairs, find the nearest neighbor pair, and adjust the transformation matrix to gradually reduce the alignment error. In each iteration, the system removes abnormal point pairs whose distance exceeds a preset threshold to reduce noise interference and dynamically updates the transformation matrix until the convergence condition is met or the maximum number of iterations is reached. The final fine registration result significantly improves the geometric consistency of the 3D model.

[0041] S204. Based on the fine registration result, a semantic segmentation model is used to identify the damage region in the structural damage image, and a pixel-level segmentation mask and damage type are output.

[0042] In step S203, the input to the semantic segmentation model is the 3D model data after fine registration processing and the structural damage image. Using a semantic segmentation model (such as a Convolutional Neural Network (CNN) or U-Net architecture), the model can perform pixel-level classification of damaged regions in the image and output a segmentation mask containing the damage type. The segmentation mask labels each pixel with its corresponding damage category, such as cracks, corrosion, or deformation, and uses color coding for visualization. Furthermore, to improve segmentation accuracy, the model introduces a multi-scale feature fusion mechanism, which enhances the ability to identify subtle damage by weighted integration of feature maps at different resolutions. The final generated segmentation mask not only possesses high resolution but also accurately reflects the spatial distribution and morphological characteristics of the damage.

[0043] S205. Based on the segmentation mask, the image pixel coordinates are converted into three-dimensional model coordinates through camera calibration parameters, and dynamic and static attributes are injected to generate a three-dimensional damaged body with attribute annotations.

[0044] In step S205, the pixel coordinates in the structural damage image are mapped to the global coordinate system of the 3D model using camera calibration parameters, thereby achieving spatial localization of the damaged area. Dynamic attributes include the damage development trend, real-time monitoring data, and related time-series information, while static attributes cover fixed information such as damage type, location, and geometric features. These attributes are injected into the 3D damage volume through semantic association rules, forming a labeled entity with rich semantic information. The final generated 3D damage volume can not only intuitively display the spatial distribution and morphological characteristics of the damage.

[0045] As an optional embodiment of the present invention, optionally, the expression for calculating the initial transformation matrix between the source point cloud and the target point cloud in the initial three-dimensional model in step S202 is:

[0046] , , , ;

[0047] in, Represents the transformation matrix. Represents an orthogonal rotation matrix. This represents the displacement from the centroid of the source point cloud to the centroid of the target point cloud. This represents a 1×3 zero vector, used for filling translation terms in homogeneous transformation matrices. , and Represents the covariance matrix The SVD decomposition results, This represents the transpose of a vector. This indicates the centroid of the target point cloud sample after synchronization. This indicates the centroid of the source point cloud sample after synchronization. This represents the number of corresponding point pairs used to estimate the transformation matrix. This indicates that after synchronization, the centroid source point will be reached. This indicates that the target centroid will be reached after synchronization.

[0048] As an optional embodiment of the present invention, optionally, in step S3, an intelligent annotation model is constructed based on the three-dimensional damage body, and the instructions and parameters are fine-tuned using a port domain corpus to form an intelligent annotation model with multimodal understanding capabilities, including:

[0049] S301. Based on the three-dimensional damaged body, a convolutional neural network is used as the basic model, and a three-dimensional spatial encoder, an attribute reasoning module and an uncertainty quantization unit are added to the basic model to form an initial intelligent annotation model.

[0050] In step S301, the initial intelligent annotation model is constructed using a convolutional neural network as its core architecture, and a 3D spatial encoder is introduced to extract spatial features of the 3D damaged body. The 3D spatial encoder combines 3D coordinate information with damage attributes to generate a high-dimensional representation vector containing spatial distribution features. Simultaneously, the attribute inference module employs a multilayer perceptron structure, deriving the latent semantic relationships of the damaged body through joint modeling of static and dynamic attributes. Furthermore, an uncertainty quantization unit, designed based on Bayesian neural network principles, is used to evaluate the model's confidence level in different annotation tasks. The entire model framework is trained end-to-end to ensure feature transfer and collaborative optimization between modules.

[0051] S302. Construct a port domain corpus based on the initial intelligent annotation model, and preprocess the port domain corpus to obtain a training dataset; the port domain corpus includes maintenance logs, annotated images, and expert experience text;

[0052] In step S302, the construction of the port domain corpus first extracts relevant information from multi-source heterogeneous data, including maintenance logs, labeled images, and expert experience texts. After cleaning and format standardization, this data is divided into training, validation, and test sets. In the preprocessing stage, maintenance logs undergo word segmentation, stop word removal, and word vector conversion using natural language processing techniques to form structured data suitable for model training. Labeled images utilize image enhancement techniques (such as rotation, scaling, and brightness adjustment) to expand sample diversity, and corresponding label files are generated using annotation tools. Expert experience texts undergo semantic parsing and keyword extraction, transforming them into a rule-based knowledge base. The final training dataset not only covers annotation needs across various scenarios but also possesses high domain adaptability.

[0053] S303. Based on the training dataset, design a three-level instruction template for the initial intelligent annotation model, including recognition, classification and reasoning, and implement instruction fine-tuning through a supervised fine-tuning process. Use an optimizer for hierarchical training to generate a fine-tuned intelligent annotation model.

[0054] In step S303, to achieve instruction fine-tuning, a three-level instruction template is first designed, corresponding to the recognition, classification, and inference tasks, respectively. The recognition instruction template is used to extract key attributes from the input data, such as damage type or location information; the classification instruction template classifies the data according to the extracted attributes to ensure the accuracy of the annotation results; the inference instruction template, by combining the domain knowledge base, derives potential semantic relationships and annotation constraints. In the supervised fine-tuning process, a hierarchical training strategy is adopted to gradually optimize the model parameters. Specifically, an optimizer (such as AdamW) is used to dynamically adjust the learning rate of different modules, prioritizing the updating of network layer parameters closely related to the task, while freezing some weights of the basic model to prevent overfitting. In addition, to improve the model's generalization ability, mixed precision training technology is introduced during training, effectively reducing computational resource consumption and accelerating the convergence process.

[0055] After fine-tuning the instructions, efficient parameter fine-tuning is further implemented. In this stage, the key parameter matrix of the model is compressed using a low-rank decomposition method, updating only the low-rank portion after decomposition, thus significantly reducing the storage and computational overhead required for fine-tuning. Simultaneously, combined with a domain adaptability verification mechanism, the model's performance in port facility maintenance scenarios is periodically evaluated, and the fine-tuning strategy is dynamically adjusted based on the evaluation results. The final fine-tuned intelligent annotation model not only possesses multimodal understanding capabilities but also accurately captures semantic information in complex scenarios.

[0056] S304. Add a low-rank adapter to the weight matrix bypass of the fine-tuned intelligent annotation model, and fuse the parameters with the activation function through two fully connected layers to obtain the intelligent annotation model after parameter fusion.

[0057] The introduction of the low-rank adapter in step S304 aims to further improve the model's parameter efficiency and domain adaptability. Specifically, by adding a low-rank adapter as a bypass to the weight matrix of the fine-tuning intelligent annotation model, local adjustments to key parameters can be achieved without updating the entire weight matrix. This method significantly reduces computational complexity and storage requirements while preserving the model's high performance in specific tasks. The design of two fully connected layers enhances the expressive power of the low-rank adapter, where the first fully connected layer maps the input features to a low-dimensional space, and the second layer restores them to the original dimension. Activation functions (such as ReLU or GELU) are inserted between the two fully connected layers to introduce non-linear characteristics and enhance the model's ability to learn complex patterns. After parameter fusion, the intelligent annotation model can not only process multimodal data more efficiently but also exhibit stronger robustness and adaptability in port facility management scenarios.

[0058] S305. The intelligent annotation model based on the fusion of the parameters is verified and optimized using a four-dimensional indicator system to obtain an intelligent annotation model with multimodal understanding capabilities; the four-dimensional indicator system includes accuracy, recall, F1 score and inference speed.

[0059] The expression for an intelligent annotation model with multimodal understanding capabilities is:

[0060]

[0061] in, This indicates that the model output annotation results include three-dimensional attributes such as facility type, damage type, and maintenance requirements. This represents a parameter-efficient fine-tuning function based on a low-rank adapter. This represents a multimodal input tensor containing 3D damage features from images, text, and point clouds. This represents the activation function. Represents the weight matrix of the basic model. and This represents a low-rank adapter matrix, where efficient parameter fine-tuning is achieved using the LoRA technique. This represents the bias vector, which is added to the adapter output through the parameter fusion unit;

[0062] In step S305, based on the intelligent annotation model after parameter fusion, a four-dimensional indicator system is used for verification and optimization. Accuracy is used to evaluate the correctness of the model's annotation results; recall measures the model's ability to cover the target damaged area; the F1 score comprehensively considers accuracy and recall to balance their performance; and inference speed reflects the model's efficiency in practical applications. Through a comprehensive evaluation of these four dimensions, the model's performance is further optimized to ensure its robustness and adaptability in complex scenarios. The final intelligent annotation model not only possesses multimodal understanding capabilities but can also efficiently handle the diverse task requirements in the field of port facility maintenance.

[0063] As an optional embodiment of the present invention, optionally, in step S4, the multimodal understanding capability of the intelligent annotation model is used to automatically annotate the facility type, damage type, and maintenance requirements, and semi-automatic annotation correction and uncertainty marking are performed in conjunction with an expert collaborative review platform to obtain standardized and structured port facility maintenance professional knowledge base annotation data, including:

[0064] S401. Based on the intelligent annotation model, the multi-modal perception network collects multi-source heterogeneous data in real time and automatically annotates it to obtain preliminary three-dimensional attribute annotation results including facility type, damage type, and maintenance requirements.

[0065] In step S401, the intelligent annotation model generates preliminary 3D attribute annotation results through automatic analysis of multi-source heterogeneous data. These results cover key information such as facility type, damage type, and maintenance requirements, and are stored in a structured form for subsequent processing. To ensure the accuracy and consistency of the annotation results, the system dynamically adjusts the annotation strategy based on the quality and complexity of the input data. For example, when faced with high-noise or low-resolution data, the model prioritizes a conservative annotation mode to avoid incorrect annotations due to misjudgment. Furthermore, by introducing a real-time feedback mechanism, the model can continuously optimize its predictive capabilities based on historical annotation data, thereby improving overall annotation efficiency.

[0066] S402. Use an expert collaborative review platform to correct the preliminary three-dimensional attribute annotation results to form semi-automatic corrected annotation data;

[0067] In step S402, the introduction of the expert collaborative review platform aims to combine the advantages of human experience with those of intelligent models to further improve the accuracy and reliability of the labeled data. During the correction process, the platform displays the preliminary 3D attribute labeling results to domain experts and provides an intuitive visualization interface for review and adjustment. Experts can add, delete, or modify the labeling results according to actual needs, while also marking areas or attributes with uncertainties, forming semi-automatically corrected labeling data. This collaborative mode not only effectively reduces the workload of manual review but also continuously optimizes the performance of the intelligent labeling model through expert feedback. In addition, to improve review efficiency, the platform also supports batch processing and can dynamically allocate resources according to task priority to ensure that critical tasks can be responded to in a timely manner. The final semi-automatically corrected labeling data retains the efficiency of the intelligent model while incorporating the professional judgment of experts.

[0068] S403. Based on the semi-automatic correction annotation data, the uncertainty quantification module is used to evaluate the confidence level of the annotation results, mark the low confidence area and generate uncertainty labels;

[0069] In step S403, the introduction of the uncertainty quantification module aims to assess the confidence level of the semi-automatically corrected annotation data, thereby identifying low-confidence regions and generating corresponding uncertainty labels. Specifically, this module, based on Bayesian inference principles, calculates the confidence interval for each annotation result by analyzing the model's prediction distribution during the annotation process. For regions or attributes with low confidence, the system automatically marks them as uncertainty labels and uses them as the focus of subsequent optimization. This process not only helps improve the reliability of the annotation data but also provides clear guidance for expert review. Furthermore, through statistical analysis of the uncertainty labels, weaknesses of the model in specific scenarios can be further identified. The final generated annotation data shows significant improvements in both accuracy and reliability.

[0070] S404. Combine the uncertainty labels to manually review the low-confidence areas, and supplement the rule base with expert knowledge to optimize the decision boundary of the intelligent labeling model, thereby obtaining standardized and structured port facility maintenance professional knowledge base labeling data.

[0071] In step S404, the process of manually reviewing low-confidence regions using uncertainty labels begins with domain experts conducting a detailed review of the low-confidence portions of the annotation results. Based on their experience and supplementary rule base content, experts correct any omissions or misjudgments by the model and re-input the corrected data into the system to optimize the decision boundaries of the intelligent annotation model. During this process, the rule base continuously improves its coverage and accuracy by absorbing expert knowledge and new findings from real-world cases. Simultaneously, for problem areas confirmed after review, the system automatically adjusts relevant parameters and updates the training set to ensure the model can more accurately handle similar scenarios in subsequent tasks. The final generated annotation data not only achieves standardization and structure but also possesses high domain adaptability.

[0072] As an optional embodiment of the present invention, optionally, in step S5, consistency verification, integrity check, and domain adaptability verification are used to quantitatively evaluate the annotation results, and the evaluation results include:

[0073] S501. Based on the annotation results, using consistency verification rules, by comparing the annotation data of the same facility at different time points, detect whether there are conflicts or abnormal fluctuations in the annotation attributes, and generate a consistency verification report.

[0074] In step S501, the consistency verification rules are designed to ensure the stability and logical coherence of the annotation results over time. By comparing and analyzing the annotation data of the same facility at different time points, the system can identify potential attribute conflicts or abnormal fluctuations. For example, if the damage type of a facility changes unreasonably between adjacent time points, the system will mark it as suspicious data and generate a detailed consistency verification report. This report not only lists all detected anomalies but also provides possible cause analysis and correction suggestions for subsequent processing.

[0075] S502. Perform a completeness check on the annotation results to verify whether each annotated entity contains complete attribute information, including facility type, damage type and maintenance requirements, and mark the annotation data with missing attributes to obtain the completeness check results.

[0076] In step S502, the integrity check process focuses on ensuring that no attribute information for each labeled entity is missing. The system checks the labeled data one by one to confirm whether it covers necessary attributes such as facility type, damage type, and maintenance requirements. For labeled data lacking any attribute, the system automatically marks it and generates an integrity check result report. This report lists all incomplete labeled entities in detail and provides supplementary suggestions for future improvement. This mechanism effectively improves the overall quality of labeled data.

[0077] S503. Construct a domain adaptability verification model based on the port domain knowledge base. By calculating the semantic similarity between the annotation results and the domain knowledge, evaluate whether the annotation data conforms to the domain specifications and output the domain adaptability score.

[0078] In step S503, the construction of the domain adaptability verification model aims to ensure that the labeled data maintains a high degree of consistency with the professional standards and actual needs of port facility maintenance. By introducing a domain knowledge base, the system can calculate the semantic similarity between the labeled results and domain knowledge, thereby assessing whether the labeled data conforms to domain standards. Specifically, the model first performs semantic parsing on the labeled results, extracting key attributes and relationships, and matching them with regularized entries in the domain knowledge base. During the matching process, indicators such as cosine similarity or Jaccard similarity coefficient are used to quantify the degree of consistency between the labeled results and domain knowledge. For labeled data with low semantic similarity, the system further analyzes the sources of deviation and generates a detailed domain adaptability scoring report. This report not only includes the scoring results but also provides specific optimization suggestions to guide subsequent data correction and model improvement work. Ultimately, through this verification mechanism, the professionalism and applicability of the labeled data can be significantly improved.

[0079] S504. Based on the consistency verification report, integrity check results and domain adaptability score, a quantitative evaluation result is formed, and the evaluation result is classified into three categories of labeled data: qualified, needing correction, and unqualified.

[0080] In step S504, the system classifies the labeled data according to strict standards, based on the consistency verification report, integrity check results, and domain suitability score. Specifically, data that performs well in all three assessments is directly classified as qualified; data with minor issues that can be improved with simple corrections is marked as requiring correction; and data with serious deviations in any one assessment is deemed unqualified. This classification method not only helps to clarify the quality level of the labeled data but also provides a clear direction for subsequent optimization work. Furthermore, the system generates a detailed assessment summary report based on the classification results, listing the specific distribution of each data category and corresponding improvement suggestions, thereby supporting efficient data management and model iteration.

[0081] As an optional embodiment of the present invention, optionally, in step S7, the qualified labeled data is structurally transformed according to the entity-relationship-attribute triplet specification of the knowledge graph, and an association index with the port facility management and maintenance model is established, including:

[0082] S701. Based on the qualified labeled data, perform structured transformation according to entity, relation and attribute triples to generate facility entities, damage entities and core relations, forming a triple dataset that conforms to the knowledge graph specification.

[0083] In step S701, the system performs deep analysis on the qualified labeled data, transforming it into triples that conform to knowledge graph specifications. Specifically, facility entities are defined as core components of port facilities, damage entities describe various problems that may occur during facility use, and core relationships express the correlation between facilities and damage. For example, a wharf facility and its crack damage are connected through an "existence" relationship, forming a complete triple entry. During this process, the system automatically identifies and extracts key information from the labeled data and maps it to a preset triple template to ensure the uniformity and standardization of the data structure. Furthermore, to enhance the semantic richness of the triple dataset, the system also expands the entities and relationships by incorporating a domain knowledge base, adding necessary contextual information to enhance its expressive power in practical applications.

[0084] S702. Based on the triplet dataset, the knowledge graph storage structure is obtained by storing it in the form of an entity-relation nested dictionary.

[0085] In step S702, the system stores the generated triplet dataset in the form of an entity-relation nested dictionary, thereby constructing an efficient and flexible knowledge graph storage structure. In this structure, each facility entity serves as the primary key, and its corresponding damaged entities and core relationships are organized in a nested dictionary manner, forming a hierarchical data representation. For example, all damaged entities and their associated relationships under a certain facility entity will be categorized into a sub-dictionary of that facility, facilitating rapid retrieval and updates. Simultaneously, to support multi-dimensional query needs, the system also indexes key fields in the nested dictionaries, further improving data access efficiency. Furthermore, by introducing a version control mechanism, the knowledge graph storage structure can record the historical state of each update, ensuring data traceability and consistency. The final storage structure possesses excellent scalability.

[0086] S703. Based on the knowledge graph storage structure, a unique triplet identifier is generated using a hash algorithm. Based on the unique identifier and the port facility management and maintenance model, an association index containing timestamps and source information is constructed.

[0087] In step S703, the system generates a unique identifier for each triple using an existing hash algorithm, ensuring the uniqueness and identifiability of the data within the knowledge graph. This identifier not only distinguishes different triple entries but also serves as the core key value of the association index, deeply binding it to the port facility management and maintenance model. During this process, the system embeds timestamps and source information into the association index to record the generation time and original source of each data entry. This design effectively supports dynamic data updates and source tracing queries, while also enhancing the model's ability to process historical data. Furthermore, by combining timestamp information, the system can also perform time-series analysis of facility status changes. The resulting association index structure ensures efficient data management.

[0088] As an optional embodiment of the present invention, the method may further include:

[0089] S9. Construct an expert knowledge graph based on expert experience;

[0090] S10. When annotating the intelligent annotation model, knowledge reasoning is used to guide the annotation based on the expert knowledge graph.

[0091] In step S10, the intelligent annotation model achieves accurate reasoning and efficient guidance for complex scenarios through deep interaction with the expert knowledge graph. During the annotation process, the system dynamically adjusts the annotation strategy based on the domain rules and experience data in the expert knowledge graph, taking into account the specific needs of the current task. For example, when encountering new facilities or rare damage types, the model can perform analogy analysis based on similar cases in the knowledge graph and generate reasonable annotation suggestions. Furthermore, by introducing a knowledge reasoning mechanism, the system can discover potential implicit relationships during the annotation process, further improving the comprehensiveness and accuracy of the annotation results. This reasoning-guided approach based on the expert knowledge graph not only enhances the professional adaptability of the intelligent annotation model but also significantly reduces the frequency of manual intervention, thereby improving overall annotation efficiency. Ultimately, the annotation results are effectively guaranteed in terms of both professionalism and consistency.

[0092] As an optional embodiment of the present invention, optionally, constructing an expert knowledge graph based on expert experience includes:

[0093] S901. Collect expert experience documents in the field of port facility maintenance, including historical maintenance records, fault diagnosis reports and technical manuals, to form a raw dataset;

[0094] In step S901, the system collects expert experience documents in the field of port facility maintenance through multiple channels, covering various formats such as historical maintenance records, fault diagnosis reports, and technical manuals, forming a rich raw dataset. These documents not only record the operational procedures and decision-making basis of field experts in actual work, but also include a large number of case analyses in real-world scenarios. To ensure the comprehensiveness and authority of the data, the system prioritizes materials from leading industry institutions or senior experts and rigorously evaluates the timeliness and applicability of the documents. In addition, the system performs format standardization processing on data from different sources to facilitate subsequent information extraction and integration. The final raw dataset is both highly professional and reflects the latest development trends in the field of port facility maintenance.

[0095] S902. Preprocess the original dataset, extract key entities and relationships, use natural language processing technology to identify domain terms and their association rules, and generate a preliminary knowledge graph framework.

[0096] In step S902, the system performs deep preprocessing on the original dataset, extracting key entities and relationships, and uses natural language processing techniques to identify domain terms and their association rules, thereby generating a preliminary knowledge graph framework. Specifically, the system first extracts core concepts and attributes related to port facility maintenance from the documents using techniques such as word segmentation, part-of-speech tagging, and named entity recognition. Then, using dependency parsing and semantic role labeling, it further mines the logical relationships and contextual associations between these entities. For example, the system can identify the causal relationship between a certain fault type and its corresponding maintenance method, and use this as an important edge in the knowledge graph. Furthermore, to improve the accuracy of terminology recognition, the system also combines a domain dictionary and rule base to correct and supplement the extraction results. The final preliminary knowledge graph framework contains not only rich entities and relationships.

[0097] S903. Based on the preliminary knowledge graph framework, semantic enhancement is performed using a graph embedding algorithm, the relation weights between entities are optimized, and multiple rounds of iterative verification are conducted to obtain the verified expert knowledge graph.

[0098] In step S903, the system uses a graph embedding algorithm to semantically enhance the preliminary knowledge graph framework, optimizing relation weights by calculating the similarity and association strength between entities. This process not only reveals implicit semantic connections but also effectively addresses relation bias caused by data sparsity or noise interference. In multi-round iterative verification, the system evaluates the performance of the knowledge graph in conjunction with real-world application scenarios and dynamically adjusts relation weights and structural layout based on feedback. For example, for some frequently occurring but semantically ambiguous relations, the system introduces additional contextual information for clarification to improve accuracy and usability. Simultaneously, to ensure the stability and reliability of the knowledge graph, a consistency check is performed after each iteration to eliminate potential logical conflicts or redundant connections. The final generated expert knowledge graph possesses high semantic richness and domain adaptability.

[0099] S904. Inject the newly added expert experience and real-time annotation results into the expert knowledge graph for updating.

[0100] In step S904, the system injects newly added expert experience and real-time annotation results into the expert knowledge graph through a dynamic update mechanism, ensuring that it remains up-to-date and highly practical. During this process, the system automatically identifies and filters new experience or annotation data with significant value, transforming them into standardized knowledge entries. For example, when a maintenance method for a new facility is repeatedly verified as effective, the system will include it in the knowledge graph and establish associations with relevant entities. Simultaneously, to avoid information redundancy or conflicts, the system performs rigorous quality checks on newly added content, including semantic consistency checks and domain adaptability assessments. For content that does not meet the standards, the system generates detailed correction suggestions, which are then submitted for manual review and confirmation. Ultimately, through this continuous updating approach, the expert knowledge graph can continuously expand its coverage and enhance its support for complex scenarios.

[0101] Example 2

[0102] A data annotation system for a professional knowledge base of port facility management and maintenance large-scale models includes:

[0103] processor;

[0104] Memory used to store processor-executable instructions;

[0105] The processor is configured to implement a data annotation method for a large-scale professional knowledge base of port facility management and maintenance models when executing executable instructions.

[0106] It should be noted that the computer device includes a processor, a memory, and may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0107] The processor controls the overall operation of the computer device to complete all or part of the steps in the aforementioned method for annotating the professional knowledge base of the port facility management and maintenance model.

[0108] Memory is used to store various types of data to support the operation of the computer device. This data may include, for example, instructions for any application or method used to operate on the computer device, as well as application-related data. Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0109] The multimedia component may include a screen and an audio component, wherein the screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals; for example, the audio component may include a microphone for receiving external audio signals, the received audio signals may be further stored in memory or transmitted via a communication component; the audio component may also include at least one speaker for outputting audio signals.

[0110] I / O interfaces provide interfaces between the processor and other interface modules, such as keyboards, mice, buttons, etc.; these buttons can be virtual buttons or physical buttons.

[0111] The communication component is used for wired or wireless communication between the computer device and other devices; wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G or 5G, or one or more combinations thereof, and the corresponding communication component may include: Wi-Fi module, Bluetooth module, NFC module, mobile communication module.

[0112] As a preferred embodiment, the computer device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the aforementioned port facility maintenance large model professional knowledge base data annotation method.

[0113] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A data annotation method for a professional knowledge base of a large-scale model for port facility management and maintenance, characterized in that, The method includes: A multimodal sensing network is deployed in port facilities to collect structural damage images, vibration sensor data, temperature stress monitoring data, and maintenance log text data, forming a spatiotemporally synchronized multi-source heterogeneous dataset. Based on the aforementioned multi-source heterogeneous dataset, a port digital twin 3D model is constructed. The point cloud registration algorithm and semantic segmentation model are used to map the damage detection results in the structural damage image to the port digital twin 3D model, generating a 3D damage body with attribute annotations. A smart annotation model is constructed based on the three-dimensional damage body, and the instructions and parameters are fine-tuned using a port domain corpus to form a smart annotation model with multimodal understanding capabilities. The intelligent annotation model is used to automatically annotate facility types, damage types and maintenance requirements, and combined with an expert collaborative review platform for semi-automatic annotation correction and uncertainty marking, to obtain standardized and structured port facility maintenance professional knowledge base annotation data. The annotation results are quantitatively evaluated using consistency checks, integrity checks, and domain adaptability verification to obtain the evaluation results. Based on the evaluation results, the unqualified labeled data is returned to the intelligent labeling model for secondary fine-tuning and training. The labeling rules and domain knowledge constraints are supplemented by the manual correction opinions of the expert collaborative review platform for secondary labeling. After the secondary labeling is completed, the quantitative evaluation is re-executed. For labeled data that still cannot pass the evaluation after multiple corrections, it is marked as a high-difficulty sample and included in the expert knowledge base for special labeling and rule accumulation. The qualified labeled data will be structurally transformed according to the entity-relationship-attribute triplet specification of knowledge graph, and an association index with the port facility management and maintenance model will be established.

2. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 1, characterized in that, Generating 3D damage volumes with attribute annotations includes: Based on the aforementioned multi-source heterogeneous dataset, laser point cloud data, UAV aerial survey orthophotos, and BIM model framework, an initial 3D model is constructed using 3D reconstruction algorithms and multi-view stereo matching algorithms. Based on the initial 3D model, the RANSAC algorithm and FPFH feature descriptors are used for coarse registration of the point cloud. Large-scale spatial alignment is performed by calculating the initial transformation matrix between the source point cloud and the target point cloud in the initial 3D model to obtain the coarse registration result. Based on the coarse registration result, the ICP algorithm is used to perform fine registration of the point cloud to obtain the fine registration result; Based on the fine registration results, a semantic segmentation model is used to identify the damaged regions in the structural damage image and output a pixel-level segmentation mask and damage type. Based on the segmentation mask, the image pixel coordinates are converted into three-dimensional model coordinates through camera calibration parameters, and dynamic and static attributes are injected to generate a three-dimensional damaged body with attribute annotations.

3. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 2, characterized in that, The expression for calculating the initial transformation matrix between the source point cloud and the target point cloud in the initial 3D model is as follows: ， ， ， ； in, Represents the transformation matrix. Represents an orthogonal rotation matrix. This represents the displacement from the centroid of the source point cloud to the centroid of the target point cloud. This represents a 1×3 zero vector, used for filling translation terms in homogeneous transformation matrices. , and Represents the covariance matrix The SVD decomposition results, This represents the transpose of a vector. This indicates the centroid of the target point cloud sample after synchronization. This indicates the centroid of the source point cloud sample after synchronization. This represents the number of corresponding point pairs used to estimate the transformation matrix. This indicates that after synchronization, the centroid source point will be reached. This indicates that the target centroid will be reached after synchronization.

4. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 1, characterized in that, The formation of an intelligent annotation model with multimodal understanding capabilities in step S3 includes: Based on the three-dimensional damaged body, a convolutional neural network is used as the basic model, and a three-dimensional spatial encoder, an attribute reasoning module, and an uncertainty quantization unit are added to the basic model to form an initial intelligent annotation model. A port domain corpus is constructed based on the initial intelligent annotation model, and the port domain corpus is preprocessed to obtain a training dataset; the port domain corpus includes maintenance logs, annotated images, and expert experience text; Based on the training dataset, a three-level instruction template including recognition, classification and reasoning is designed for the initial intelligent annotation model. The instruction is fine-tuned through a supervised fine-tuning process. An optimizer is used for hierarchical training to generate a fine-tuned intelligent annotation model. A low-rank adapter is added to the weight matrix of the fine-tuned intelligent annotation model, and the parameters are fused with the activation function through two fully connected layers to obtain the intelligent annotation model after parameter fusion. The intelligent annotation model based on the fusion of the parameters is validated and optimized using a four-dimensional indicator system to obtain an intelligent annotation model with multimodal understanding capabilities; the four-dimensional indicator system includes accuracy, recall, F1 score and inference speed.

5. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 1, characterized in that, Standardized, structured professional knowledge base data for port facility maintenance includes: Based on the intelligent annotation model, the multi-modal sensing network collects multi-source heterogeneous data in real time and automatically annotates it to obtain preliminary three-dimensional attribute annotation results including facility type, damage type, and maintenance requirements. The preliminary 3D attribute annotation results are corrected using an expert collaborative review platform to form semi-automatic corrected annotation data; Based on the semi-automatic correction annotation data, an uncertainty quantification module is used to evaluate the confidence level of the annotation results, mark low-confidence areas, and generate uncertainty labels. By combining the aforementioned uncertainty labels, low-confidence areas are manually reviewed, and the decision boundary of the intelligent labeling model is optimized by supplementing the rule base with expert knowledge, thus obtaining standardized and structured port facility maintenance professional knowledge base labeling data.

6. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 1, characterized in that, The evaluation results include: Based on the annotation results, consistency verification rules are used to compare the annotation data of the same facility at different time points to detect whether there are conflicts or abnormal fluctuations in the annotation attributes, and a consistency verification report is generated. The annotation results are subjected to a completeness check to verify whether each annotated entity contains complete attribute information, including facility type, damage type and maintenance requirements, and the annotation data with missing attributes are marked to obtain the completeness check results; A domain adaptability verification model is built based on a port domain knowledge base. By calculating the semantic similarity between the annotation results and domain knowledge, the model evaluates whether the annotation data conforms to the domain specifications and outputs a domain adaptability score. Based on the consistency verification report, integrity check results, and domain adaptability score, a quantitative evaluation result is formed, and the evaluation result is classified into three categories of labeled data: qualified, needing correction, and unqualified.

7. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 1, characterized in that, The qualified labeled data will be structurally transformed according to the entity-relation-attribute triplet specification of knowledge graphs, and an association index with the port facility management and maintenance model will be established, including: Based on the qualified labeled data, the structure is transformed according to entity, relation and attribute triples to generate facility entities, damage entities and core relations, forming a triple dataset that conforms to the knowledge graph specification. Based on the triplet dataset, a knowledge graph storage structure is obtained by storing it in the form of an entity-relation nested dictionary. Based on the knowledge graph storage structure, a unique triplet identifier is generated using a hash algorithm. Based on the unique identifier and the port facility management and maintenance model, an associated index containing timestamps and source information is constructed.

8. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 1, characterized in that, The method further includes: Constructing an expert knowledge graph based on expert experience; When the intelligent annotation model performs annotation, it uses the expert knowledge graph to guide the annotation process through knowledge reasoning.

9. The data annotation method for a professional knowledge base of a large-scale port facility management and maintenance model as described in claim 8, characterized in that, Building expert knowledge graphs based on expert experience includes: Collect expert experience documents in the field of port facility maintenance, including historical maintenance records, fault diagnosis reports and technical manuals, to form a raw dataset; The original dataset is preprocessed to extract key entities and relationships. Natural language processing techniques are used to identify domain terms and their association rules to generate a preliminary knowledge graph framework. Based on the aforementioned preliminary knowledge graph framework, semantic enhancement is performed using graph embedding algorithms, the relation weights between entities are optimized, and multiple rounds of iterative verification are conducted to obtain the verified expert knowledge graph. The newly added expert experience and real-time annotation results are injected into the expert knowledge graph for updating.

10. A data annotation system for a professional knowledge base of a large-scale model for port facility management and maintenance, characterized in that, The system includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the port facility maintenance large model professional knowledge base data annotation method according to any one of claims 1 to 9 when executing the executable instructions.

Images