Ship name standardization system and method based on large model and multistage fault-tolerant matching

Abstract

The invention relates to the technical field of data processing and matching, in particular to a ship name standardization system and method based on a large model and multistage fault-tolerant matching. The method comprises the following steps: firstly, receiving a natural language text through a data acquisition module, extracting an original ship name character string by utilizing a large model, and associating the original ship name character string with a basic data dictionary through an entity standardization service module by adopting a multi-stage matching algorithm; a result processing module calculates a confidence score of a matching result to obtain standardized ship name output; according to the system, the ship name dictionary is maintained through the dynamic dictionary updating module, cross validation is carried out by using the multi-modal verification module in combination with the ship image recognition result and the AIS trajectory data, finally, the accurate standardized ship name is obtained, and the ship name recognition and matching precision and efficiency are improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing and matching, and in particular to a ship name standardization system and method based on large model and multi-level fault-tolerant matching. Background Technology

[0002] In the shipping, port, and logistics sectors, extracting ship names from natural language text and associating them with basic system data is a crucial step in achieving business automation. With the continuous development of the shipping industry, the volume of related business has increased significantly, resulting in a massive increase in natural language text data from various sources such as port operation reports, AIS messages, and business emails. This data contains a wealth of ship name information, and accurately extracting and associating these ship names is of great significance for improving the efficiency of shipping operations, reducing costs, and promoting the digital transformation of shipping. For example, in port operations, accurate ship name identification enables automated entry of operation reports, improving the accuracy and timeliness of information processing; in parsing ship AIS messages, precise association with ship names helps to grasp real-time ship dynamic information and ensure navigational safety.

[0003] In the past, various methods were commonly used to address the problem of extracting ship names from natural language text and associating them with the system's basic data. In SOF (State of Goods) document recognition scenarios, traditional manual data entry was initially employed. Later, OCR technology combined with AI algorithms was introduced, enabling the rapid and accurate extraction of key information from documents. When processing transportation contracts, ship names and voyage information were previously extracted manually from contract text. Now, intelligent information extraction systems are used, which, by setting specific keywords and rules, can quickly extract and categorize key information such as ship names from offer information. Traditional matching algorithms often rely on simple string matching, lacking contextual understanding capabilities, and the ship name dictionary is difficult to update, typically requiring manual updates.

[0004] However, existing technologies employ diverse forms of ship name representation, including abbreviations, mixed use of Chinese and English, and misspellings. Traditional simple string matching algorithms have poor fault tolerance and cannot effectively handle these complex variations. Furthermore, existing technologies lack contextual understanding capabilities, failing to effectively utilize textual context information for ambiguity resolution. In addition, ship name dictionaries are difficult to update automatically to adapt to new naming rules and new vessels. These issues result in low ship name recognition accuracy, a large workload for manual verification, and severely hinder the progress of shipping digital transformation. Summary of the Invention

[0005] To address the problems existing in ship name recognition and matching in the prior art, this application provides a ship name standardization system and method based on a large model and multi-level fault-tolerant matching, employing the following technical solution: In a first aspect, this application provides a ship name standardization system based on a large model and multi-level fault-tolerant matching, characterized in that it includes: The data acquisition module is used to receive natural language text from various sources; The large model information extraction module is used to extract the original ship name string from the natural language text; The entity standardization service module is used to associate the original ship name with the basic data dictionary based on the original ship name string through a multi-level matching algorithm; The results processing module is used to calculate the confidence scores of matching results at different levels, and to obtain standardized ship names based on the confidence scores. The dynamic dictionary update module is used to automatically update and maintain the ship name dictionary based on the standardized ship name output. The multimodal verification module is used to perform cross-verification by combining the ship name dictionary, ship image recognition results and AIS trajectory data to obtain the finally verified standardized ship name.

[0006] By adopting the above technical solution, the accurate identification and standardization of ship names are significantly challenged due to the diverse ways ship names are expressed in the maritime industry, with the same ship potentially existing in multiple variant forms, such as mixed Chinese and English, simplified and traditional Chinese conversion, and transliteration variations. Furthermore, in actual business scenarios, ship names are often interspersed within unstructured text. This firstly, the data acquisition module receives natural language text from various channels, and then uses a large-scale model information extraction module to extract the original ship name string from the text. The extracted ship name string then enters the entity standardization service module, which uses a multi-level matching algorithm to associate the original ship name with the basic data dictionary. The system's result processing module calculates the confidence score of different levels of matching results and outputs the standardized ship name based on the confidence score. The system automatically updates and maintains the ship name dictionary through a dynamic dictionary update module. Based on this, the system's multimodal verification module performs cross-validation by combining the ship name dictionary, ship image recognition results, and AIS trajectory data to obtain the final verified standardized ship name. Through this multi-module collaborative approach, efficient and accurate ship name standardization processing is achieved, improving the data processing efficiency of maritime operations.

[0007] Optionally, the large model information extraction module uses a pre-trained language model based on the Transformer architecture and fine-tunes it for text data in the shipping field to identify and extract ship name entities from the text.

[0008] By adopting the above technical solutions, the pre-trained language model can make full use of the language knowledge learned from large-scale general corpora. By fine-tuning the model for text data in the shipping field, the model can better understand and recognize shipping professional terms and expressions, reducing the cost of manual rule maintenance. Moreover, the solution has strong generalization ability and can accurately identify ship name expressions based on contextual semantics even when faced with unfamiliar expressions.

[0009] Optionally, the multi-level matching algorithm in the entity standardization service module includes at least: The exact matching unit is used for string matching of the original ship name string based on the hash index; A fuzzy matching unit is used to perform fault-tolerant matching on the original ship name string based on edit distance or Jaro-Winkler similarity. The Pinyin / Translation Matching Unit is used to perform Chinese-English transliteration and Pinyin variant matching on the original ship name string; The vector retrieval unit is used to perform semantic similarity matching on the original ship name string in the vector space.

[0010] By adopting the above technical solutions, this application achieves multi-level matching through precise matching, fuzzy matching, pinyin / transliteration matching, and vector retrieval. The precise matching unit enables fast querying through hash indexes; the fuzzy matching unit effectively handles minor differences caused by input errors, aliases, or abbreviations through edit distance or Jaro-Winkler similarity algorithms; the transliteration conversion unit specifically solves the matching problem of multilingual expressions of ship names, such as mixed Chinese and English and pinyin variants; the vector retrieval unit captures deep semantic relationships between ship names through semantic similarity matching, handling complex variant cases that cannot be covered by other units; this not only improves the system's fault tolerance but also ensures the accuracy and completeness of the matching results.

[0011] Optionally, the system further includes an adaptive matching order optimization mechanism, used to dynamically adjust the execution order of the precise matching unit, the fuzzy matching unit, the pinyin / transliteration matching unit, and the vector retrieval unit based on the characteristics of the original ship name string and the system load.

[0012] By adopting the above technical solution, this application dynamically adjusts the execution order of matching units by analyzing ship name characteristics and system load, reducing the resource waste caused by fixed sequential execution, improving system processing efficiency, and at the same time, prioritizing matching methods with lower computational overhead when the system load is high, thus ensuring the stable operation of the system.

[0013] Optionally, the dynamic dictionary update module includes an automatic alias discovery mechanism, which analyzes historical matching records, identifies potential ship name alias relationships through clustering algorithms, and updates the ship name dictionary after manual review.

[0014] By adopting the above technical solution, the dynamic dictionary update module automatically analyzes historical matching records to discover potential ship name alias relationships, and combines this with manual review to ensure the accuracy of the update. This not only reduces the large amount of workload of purely manual dictionary maintenance, but also ensures the timely update of the dictionary and data quality, enabling the ship name dictionary to continuously adapt to new ship name expressions.

[0015] Optionally, a two-way feedback mechanism is established between the large model information extraction module and the entity standardization service module; The large model information extraction module is used to pre-sort the candidate ship names that match the original ship name string and assign initial weights to each candidate ship name. The matching results of the entity standardization service module are fed back to the large model information extraction module for incremental learning and optimization.

[0016] By adopting the above technical solution, the two-way feedback mechanism pre-assigns weights to candidate ship names through a large model for initial screening, and feeds back the final matching results for model optimization, forming a closed-loop learning process. On the one hand, the pre-sorting reduces the search space for subsequent matching, and on the other hand, the actual matching results are used to continuously improve the model performance, so that the system can continuously improve its accuracy during use.

[0017] Optionally, the result processing module is configured as follows: When the confidence score is greater than or equal to the first threshold, the standardized ship name is output directly; When the confidence score is less than the first threshold and greater than or equal to the second threshold, the multimodal verification module is triggered to perform cross-validation. When the confidence score is less than the second threshold, the result is marked as requiring manual review.

[0018] By adopting the above technical solution, the result processing module uses a multi-level threshold diversion strategy, taking corresponding processing measures according to different ranges of confidence scores: high confidence results are output directly, medium confidence results are cross-validated with multimodal data to improve accuracy, and low confidence results are manually reviewed; this ensures both processing efficiency and the reliability of standardized results, achieving a good balance between automated processing and quality control.

[0019] Secondly, this application provides a ship name standardization method based on a large model and multi-level fault-tolerant matching, including the following steps: It can receive natural language text from various sources; The original ship name string was extracted from the natural language text using a large model; Based on the original ship name string, the original ship name is associated with the basic data dictionary through a multi-level matching algorithm; Calculate the confidence score of matching results at different levels, and obtain the standardized ship name output based on the confidence score; Based on the standardized ship name output, the ship name dictionary is automatically updated and maintained; By combining the ship name dictionary, ship image recognition results, and AIS trajectory data for cross-validation, the final verified standardized ship names are obtained.

[0020] Optionally, the original ship name string can be extracted from the natural language text using a large model, specifically including the following steps: A pre-trained language model based on the Transformer architecture is used; Fine-tuning of the model for text data in the shipping industry; The fine-tuned model was used to identify and extract ship name entities from the natural language text.

[0021] Optionally, the original ship name can be associated with the basic data dictionary using a multi-level matching algorithm, specifically including the following steps: The original ship name string is matched using a precise matching step based on a hash index. The original ship name string is subjected to fault-tolerant matching through a fuzzy matching step, based on edit distance or Jaro-Winkler similarity. The original ship name string is transliterated into Chinese and English and matched with pinyin variants through a transliteration conversion step. The original ship name string is semantically similar to each other in the vector space through a vector retrieval step.

[0022] In summary, this application includes at least one of the following beneficial technical effects: This application receives natural language text through a data acquisition module, extracts the original ship name string using a large model, and associates it with the basic data dictionary using a multi-level matching algorithm by the entity standardization service module; the result processing module calculates the confidence score of the matching result to obtain the standardized ship name output; the system maintains the ship name dictionary through a dynamic dictionary update module, and uses a multimodal verification module to perform cross-validation by combining ship image recognition results and AIS trajectory data, finally obtaining accurate standardized ship names, thus improving the accuracy and efficiency of ship name recognition and matching; This application achieves multi-level matching through precise matching, fuzzy matching, pinyin / transliteration matching, and vector retrieval. The precise matching unit enables fast querying via hash index; the fuzzy matching unit effectively handles minor differences caused by input errors, aliases, or abbreviations using edit distance or Jaro-Winkler similarity algorithms; the transliteration unit specifically addresses the matching problem of multilingual expressions of ship names, such as mixed Chinese and English usage and pinyin variations; and the vector retrieval unit captures deep semantic relationships between ship names through semantic similarity matching, handling complex variations that other units cannot cover. This not only improves the system's fault tolerance but also ensures the accuracy and completeness of the matching results. This application dynamically adjusts the execution order of matching units by analyzing ship name characteristics and system load, reducing resource waste caused by fixed-order execution and improving system processing efficiency. At the same time, it can prioritize matching methods with lower computational overhead when the system load is high, ensuring stable system operation. Attached Figure Description

[0023] Figure 1 This is a diagram of the ship name standardization system architecture based on a large model and multi-level fault-tolerant matching in this application embodiment; Figure 2 This is an internal structure diagram of the entity standardization service module in an embodiment of this application; Figure 3 This is a flowchart illustrating the ship name standardization process in an embodiment of this application; Figure 4 This is a schematic diagram of the dynamic dictionary update process in an embodiment of this application; Figure 5 This is a flowchart illustrating the ship name standardization method based on a large model and multi-level fault-tolerant matching in an embodiment of this application. Figure 6 This is an internal structural diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0025] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0026] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0027] Firstly, this application provides a ship name standardization system based on a large model and multi-level fault-tolerant matching, including a data acquisition module, a large model information extraction module, an entity standardization service module, a result processing module, a dynamic dictionary update module, and a multimodal verification module. The data acquisition module receives natural language text from various sources. The large model information extraction module extracts the original ship name strings from these texts. The entity standardization service module associates the original ship name strings with the basic data dictionary using a multi-level matching algorithm. The result processing module calculates the confidence scores of the matching results at different levels and obtains the standardized ship name output based on the confidence scores. The dynamic dictionary update module automatically updates and maintains the ship name dictionary based on the standardized ship name output. The multimodal verification module performs cross-validation by combining the ship name dictionary, ship image recognition results, and AIS trajectory data to obtain the finally verified standardized ship name. Through this module setup and coordination, the system can comprehensively utilize multiple technical means to effectively handle the problem of diverse ship name expressions, improve the accuracy of ship name recognition, and reduce the workload of manual review.

[0028] Reference Figure 1 The system's workflow is as follows: The text input layer (i.e., the data acquisition module) receives text data from various sources, including port operation reports, AIS messages, and business emails. Subsequently, the large-scale model information extraction module performs preliminary identification and extraction of ship names from the text, handling any possible mixed Chinese and English characters, abbreviations, or typos. The extracted original ship name strings enter the entity standardization service module, which uses various strategies such as exact matching, fuzzy matching, pinyin / transliteration matching, vector retrieval, contextual analysis, and multimodal verification to perform multi-level fault-tolerant matching between the original ship names and the basic dictionary. The matching results are then fed into the result processing module for confidence assessment and comprehensive processing, ultimately outputting standardized ship names. Simultaneously, the dynamic dictionary update module automatically identifies new ship names and implements version management through a closed-loop manual review process, enabling continuous dictionary optimization. The entire architecture's modules work closely together, forming a complete closed loop from text input to standardized output, effectively improving the accuracy of ship name recognition and the system's adaptability.

[0029] Specifically, the data acquisition module includes a data receiving unit and a data preprocessing unit. The data receiving unit can receive natural language text from various sources, such as SOF documents, business emails, ship information, draft survey reports, loading and unloading operation records, port operation daily reports, cargo handover lists, stowage plans, and cargo hold stowage diagrams, via network interfaces, file import interfaces, etc. This unit supports both batch import and real-time API calls to meet different business needs. The data preprocessing unit performs operations such as noise reduction and format standardization on the received text, removing noisy data, standardizing the text encoding to UTF-8, and normalizing punctuation and spaces. For example, it processes redundant spaces and non-standard punctuation marks to make the text easier to process later. This unit can also use data cleaning algorithms and text processing tools to implement these functions and can be flexibly configured and adjusted according to actual needs.

[0030] The large-scale model information extraction module includes a pre-trained model unit and a feature extraction and processing unit. The pre-trained model unit employs a pre-trained language model based on the Transformer architecture, such as the BERT-base-Chinese pre-trained model. In a specific embodiment of this application, the model has 12 Transformer layers, 768 hidden layers, 12 attention heads, and 110M parameters. It supports processing multiple character types, including Chinese, English, and numbers, and has been pre-trained and optimized for special vocabulary in the shipping field, including all ship names and their English names on the platform. Of course, other similar pre-trained models, such as the GPT series models, can also be used, as long as they can be fine-tuned for shipping-related text to recognize ship name entities. The feature extraction and processing unit performs preprocessing on the input text, including word segmentation, case conversion, and special character processing. It extracts contextual features from the text, such as keywords before and after ship names and punctuation marks. It uses an attention mechanism to capture long-distance dependencies in the text and supports batch processing to improve processing efficiency. For example, given the input text "Today, the 'COSCO Shipping Star' ship arrived at the port," this unit will segment the text, extract the ship name "COSCO Shipping Star," and analyze the contextual information such as the keywords "ship" and "arrived at the port."

[0031] The entity standardization service module includes a precise matching unit, a fuzzy matching unit, a pinyin / transliteration matching unit, a vector retrieval unit, and a context analysis unit. The precise matching unit performs string matching on the original ship name string based on a hash index, supports case-insensitive matching (achieved through lowercase conversion), and can handle common punctuation differences such as spaces, hyphens, and forward slashes. It uses a hash table index, with a lookup complexity of O(1), enabling fast and accurate matching of completely identical ship names. The fuzzy matching unit performs fault-tolerant matching on the original ship name string based on edit distance or Jaro-Winkler similarity, assigning different weights to character differences at different positions; for example, the difference between the first and last characters of the ship name has a higher weight than the middle characters. The pinyin / transliteration matching unit converts Chinese ship names to pinyin, supporting full pinyin and initial abbreviations. English ship names are transliterated, supporting different transliteration rules and handling special cases such as polyphonic characters and neutral tones. It supports different pinyin systems, such as Hanyu Pinyin and Wade-Giles. The vector retrieval unit maps ship names to a vector space, uses pre-trained word or sentence vector models to calculate cosine similarity between vectors, performs semantic-level matching, supports batch retrieval and efficient indexing, and improves matching speed. The context analysis unit identifies suffixes such as "wheel" and "number" in the text, improving the accuracy of ship name recognition. It combines contextual features such as ship type and cargo information for semantic understanding, assigns weight values ​​to the extracted ship names, and prioritizes matching candidate results with higher weights. For cases where multiple ships share the same name, it further differentiates them by combining other contextual information, such as tonnage and route.

[0032] Reference Figure 2 The entity standardization service module first receives the "original ship name input" through the "matching process controller" and coordinates multiple internal matching units to work together. The core of the module comprises five matching units: precise matching, fuzzy matching, pinyin / transliteration matching, vector retrieval, and context analysis. During processing, each unit can call algorithms from the "algorithm library" and frequently access the core data source, the "ship name dictionary." The matching process controller schedules these units to identify the original ship name sequentially or in parallel. Specifically, the precise matching unit first performs a fast and accurate comparison; if unsuccessful, it sequentially initiates fault-tolerant strategies such as fuzzy matching and pinyin / transliteration matching; the vector retrieval unit performs supplementary matching at the semantic level; and the context analysis unit is involved throughout, providing auxiliary decision-making information for the matching results. The candidate results from all units are finally converged to the "result scoring unit" for comprehensive evaluation and ranking, thereby outputting the most likely "standardized ship name."

[0033] To further improve system efficiency, the entity standardization service module is also equipped with an adaptive matching order optimization mechanism. By analyzing the characteristics of the original input ship name string (such as whether it contains Chinese characters or special symbols) and the current system load, the execution order of the five matching units is dynamically adjusted through the matching process controller. For example, when a purely Chinese ship name is detected as input, the pinyin matching unit can be initiated first; when the system load is high, the precise matching unit with lower computational overhead is called first. This intelligent execution order scheduling strategy ensures optimal utilization of system resources and improves the processing efficiency of ship name standardization.

[0034] Furthermore, the system establishes a two-way feedback mechanism between the large model information extraction module and the entity standardization service module. The large model information extraction module can pre-sort and weight candidate ship names, for example, assigning higher initial weights to more likely candidate ship names based on contextual features and historical matching data. Simultaneously, the final matching results generated by the entity standardization service module are fed back to the large model information extraction module for incremental learning and optimization of the model, thereby continuously improving the model's ability to recognize ship name entities.

[0035] The results processing module includes a confidence score calculation unit and an output decision unit. The confidence score calculation unit calculates a confidence score based on the characteristics and matching status of matching results at different levels. The output decision unit directly outputs the standardized ship name when the confidence score is greater than or equal to the first threshold; when the confidence score is less than the first threshold but greater than or equal to the second threshold, it triggers the multimodal validation module to perform cross-validation; when the confidence score is less than the second threshold, the result is marked as requiring manual review.

[0036] like Figure 3 As shown, the result processing module receives candidate results from the "multi-level matching processing" stage. The confidence calculation unit integrates features such as matching level, similarity score, and contextual consistency to calculate a quantified confidence score for each candidate result. Subsequently, the output decision unit performs a three-level decision based on preset thresholds: if the score is higher than 95%, it is determined to be "high confidence," and the result is automatically confirmed and directly output as a standardized ship name; if the score is between 80% and 95%, it is determined to be "medium confidence," and the system will trigger the multimodal verification module to perform cross-validation, using external information such as images and AIS trajectories for secondary confirmation; if the score is lower than 80%, it is determined to be "low confidence," and the result will be marked as requiring manual review and enter the manual review queue.

[0037] Reference Figure 4, the dynamic dictionary update module constructs an automated closed-loop workflow from data input to continuous optimization. The system receives data from multiple channels such as standardized results, manual annotations, and external databases. Subsequently, it enters the data collection and processing stage, where the data is cleaned and standardized. The processed data flows to the core alias discovery and analysis link, and potential alias relationships (such as the association between "COSCO STAR" and "中远之星") are automatically identified through algorithms such as clustering. The identified candidate aliases will enter the conflict detection and resolution step to ensure that the new aliases do not conflict with the existing dictionary data. The candidate entries that pass the detection do not take effect directly but must be submitted to the manual review and confirmation link, and are finally reviewed by experts to ensure the accuracy of the update. After passing the review, the update instruction is sent to the dictionary update execution unit to perform addition, deletion, and modification operations on the dictionary. All update operations are managed by a version management and rollback mechanism to ensure that dictionary changes are traceable and incorrect updates can be quickly restored. Finally, the system will continuously optimize based on the usage effect. By monitoring indicators such as the matching success rate of new entries, the update quality is evaluated, and the data is fed back to further optimize the alias discovery algorithm, thus forming a self-improving intelligent evolution system.

[0038] Among them, the dynamic dictionary update module adopts a segmented index and cache system to achieve efficient management and query of massive ship name data. Through the "index selector", according to the type of query request (such as ship name keywords and context), the request is intelligently routed to the optimal index path. Specifically, the system adopts a three-level index to cooperate: the first-level index is a hash index, which uses open addressing to solve conflicts, and its O(1) time complexity provides faster speed for the exact matching of English ship names and standard ship names. The second-level index is a B+ tree index, whose node size is optimized to match the disk page and supports prefix compression, and is designed specifically for the fuzzy matching and range query of Chinese ship names (such as finding all ship names starting with "中远"), and the search complexity is stable at O(log n). The third-level index is a vector index (ANN / LSH), which maps ship names to vectors through algorithms such as locality-sensitive hashing and performs approximate nearest neighbor search, thereby supporting semantic similarity retrieval and effectively serving alias discovery. To maximize performance, the system also配备了多层内存缓存机制:使用LRU策略的热点数据缓存保障高频船名的瞬时响应;中间结果缓存则复用匹配过程中的计算结果,提升批处理效率;同时,采用写回(Write-back)策略的缓存与动态字典更新模块协同,在保证数据一致性的同时大幅降低了写操作的开销。

[0039] Note: There seems to be an incomplete part in the translation of . The text "配备了多层内存缓存机制:使用LRU策略的热点数据缓存保障高频船名的瞬时响应;中间结果缓存则复用匹配过程中的计算结果,提升批处理效率;同时,采用写回(Write-back)策略的缓存与动态字典更新模块协同,在保证数据一致性的同时大幅降低了写操作的开销。" is not fully translated. Please check and provide the complete correct content for a more accurate translation.Furthermore, the automatic alias discovery mechanism analyzes historical matching records and uses clustering algorithms to group ship name expressions, thereby identifying potential ship name alias relationships. For example, when the system discovers that ship names such as "COSCO SHIPPING AQUARIUS", "COSCO SHIPPING Aquarius", and "COSCO Aquarius" often appear in similar contexts and have a high matching correlation, they will be identified as potential alias relationships. These discovered alias relationships are not directly updated to the ship name dictionary but are only officially incorporated after manual review and confirmation, ensuring the accuracy of the dictionary data.

[0040] The multimodal verification module includes an image recognition verification unit and an AIS trajectory data verification unit. The image recognition verification unit combines the ship image recognition results for verification, such as recognizing the ship name images on the bow and stern of the ship and comparing them with the ship names in the system. The AIS trajectory data verification unit uses AIS trajectory data for cross-checking, such as information on position, course, draft, etc., to determine whether the ship name matches this data. For example, for a recognized ship name, the image recognition verification unit can recognize the ship name on the ship image. If it is consistent with the ship name recognized by the system, the credibility of the verification is increased; the AIS trajectory data verification unit can judge whether it conforms to the records in the system based on the AIS trajectory data of the ship corresponding to this ship name, such as current position, course, etc.

[0041] The implementation principle of this embodiment is as follows: Through the collaborative work of each module, the context understanding ability of the large model and the advantages of the multi-level fault-tolerant matching algorithm are fully utilized. The data collection module widely collects natural language texts from various sources, providing a rich data basis for subsequent processing. The large model information extraction module can accurately extract ship name entities from the text and perform well even when faced with complex texts and diverse ship name expressions. The entity standardization service module realizes multi-level matching from exact matching to fuzzy matching and semantic matching through the combination of multiple matching units, greatly improving the accuracy of the association between ship names and the basic data dictionary. The result processing module makes reasonable decisions based on the confidence score, ensuring the reliability of the output results. The dynamic dictionary update module can update the ship name dictionary in a timely manner to adapt to new naming rules and newly added ships. The multimodal verification module further improves the accuracy and reliability of ship name recognition by combining multi-faceted information such as image recognition and AIS trajectory data for cross-verification. Compared with the prior art, this system effectively solves problems such as diverse ship name expressions, single matching algorithm, lack of context understanding, and difficulty in dictionary update, significantly improving the accuracy of ship name recognition, reducing the workload of manual review, and providing strong support for the digital transformation of shipping.

[0042] Secondly, this application provides a ship name standardization method based on large model and multi-level fault-tolerant matching. The ship name standardization method based on large model and multi-level fault-tolerant matching of this application will be described below in conjunction with the above-mentioned ship name standardization system based on large model and multi-level fault-tolerant matching.

[0043] Reference Figure 5 A ship name standardization method based on a large model and multi-level fault-tolerant matching includes the following steps: S510: Receives natural language text from various sources; S520: Extract the original ship name string from natural language text using a large model; S530. Based on the original ship name string, associate the original ship name with the basic data dictionary using a multi-level matching algorithm; S540. Calculate the confidence score of matching results at different levels, and obtain the standardized ship name output based on the confidence score. S550. Automatically update and maintain the ship name dictionary based on standardized ship name output; S560. By combining the ship name dictionary, ship image recognition results and AIS trajectory data for cross-validation, the final verified standardized ship name is obtained.

[0044] In one embodiment, extracting the original ship name string from natural language text using a large model specifically includes the following steps: A pre-trained language model based on the Transformer architecture is used; Fine-tuning of the model for text data in the shipping industry; The finely tuned model was used to identify and extract ship name entities from natural language text.

[0045] In one embodiment, the original ship name is associated with the basic data dictionary using a multi-level matching algorithm, specifically including the following steps: The original ship name string is matched using a precise matching step based on a hash index. The original ship name string is matched with tolerance through a fuzzy matching step, based on edit distance or Jaro-Winkler similarity. The original ship name string is transliterated into Chinese and English and matched with pinyin variants through a transliteration conversion step. The original ship name strings are semantically similar to each other in the vector space through a vector retrieval step. In one embodiment, the execution order of the precise matching step, the fuzzy matching step, the transliteration conversion step, and the vector retrieval step is dynamically adjusted based on the characteristics of the original ship name string and the system load.

[0046] In one embodiment, the automatic updating and maintenance of the ship name dictionary based on standardized ship name output specifically includes: Analyze historical matching records; Potential ship name alias relationships are identified using clustering algorithms; Updated to the ship name dictionary after manual review.

[0047] In one embodiment, the method further includes pre-sorting and assigning initial weights to the candidate ship names matching the original ship name string using a large model; and feeding the entity standardization results back to the large model for incremental learning and optimization of the large model.

[0048] In one embodiment, the standardized ship name output is obtained based on the confidence score, specifically including: When the confidence score is greater than or equal to the first threshold, the standardized ship name is output directly; Multimodal cross-validation is triggered when the confidence score is less than the first threshold and greater than or equal to the second threshold. When the confidence score is less than the second threshold, the result is marked as requiring manual review.

[0049] In one embodiment, this application provides an electronic device, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a ship name standardization method based on a large model and multi-level fault-tolerant matching.

[0050] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0051] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0052] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0053] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A ship name standardization system based on a large model and multi-level fault-tolerant matching, characterized in that, include: The data acquisition module is used to receive natural language text from various sources; The large model information extraction module is used to extract the original ship name string from the natural language text; The entity standardization service module is used to associate the original ship name with the basic data dictionary based on the original ship name string through a multi-level matching algorithm; The results processing module is used to calculate the confidence scores of matching results at different levels, and to obtain standardized ship names based on the confidence scores. The dynamic dictionary update module is used to automatically update and maintain the ship name dictionary based on the standardized ship name output. The multimodal verification module is used to perform cross-verification by combining the ship name dictionary, ship image recognition results and AIS trajectory data to obtain the finally verified standardized ship name.

2. The ship name standardization system based on large model and multi-level fault-tolerant matching according to claim 1, characterized in that, The large model information extraction module uses a pre-trained language model based on the Transformer architecture and fine-tunes it for text data in the shipping field to identify and extract ship name entities from the text.

3. The ship name standardization system based on large model and multi-level fault-tolerant matching according to claim 1, characterized in that, The multi-level matching algorithm in the entity standardization service module includes at least the following: The exact matching unit is used for string matching of the original ship name string based on the hash index; A fuzzy matching unit is used to perform fault-tolerant matching on the original ship name string based on edit distance or Jaro-Winkler similarity. The Pinyin / Translation Matching Unit is used to perform Chinese-English transliteration and Pinyin variant matching on the original ship name string; The vector retrieval unit is used to perform semantic similarity matching on the original ship name string in the vector space.

4. The ship name standardization system based on large model and multi-level fault-tolerant matching according to claim 3, characterized in that, The system also includes an adaptive matching order optimization mechanism, which dynamically adjusts the execution order of the precise matching unit, the fuzzy matching unit, the pinyin / transliteration matching unit, and the vector retrieval unit based on the characteristics of the original ship name string and the system load.

5. The ship name standardization system based on large model and multi-level fault-tolerant matching according to claim 1, characterized in that, The dynamic dictionary update module includes an automatic alias discovery mechanism, which analyzes historical matching records, identifies potential ship name alias relationships through clustering algorithms, and updates the ship name dictionary after manual review.

6. The ship name standardization system based on large model and multi-level fault-tolerant matching according to claim 1, characterized in that, A two-way feedback mechanism is established between the large model information extraction module and the entity standardization service module. The large model information extraction module is used to pre-sort the candidate ship names that match the original ship name string and assign initial weights to each candidate ship name. The matching results of the entity standardization service module are fed back to the large model information extraction module for incremental learning and optimization.

7. The ship name standardization system based on large model and multi-level fault-tolerant matching according to claim 1, characterized in that, The result processing module is configured as follows: When the confidence score is greater than or equal to the first threshold, the standardized ship name is output directly; When the confidence score is less than the first threshold and greater than or equal to the second threshold, the multimodal verification module is triggered to perform cross-validation. When the confidence score is less than the second threshold, the result is marked as requiring manual review.

8. A ship name standardization method based on a large model and multi-level fault-tolerant matching, characterized in that, Includes the following steps: It can receive natural language text from various sources; The original ship name string was extracted from the natural language text using a large model; Based on the original ship name string, the original ship name is associated with the basic data dictionary through a multi-level matching algorithm; Calculate the confidence score of matching results at different levels, and obtain the standardized ship name output based on the confidence score; Based on the standardized ship name output, the ship name dictionary is automatically updated and maintained; By combining the ship name dictionary, ship image recognition results, and AIS trajectory data for cross-validation, the final verified standardized ship names are obtained.

9. The ship name standardization method based on large model and multi-level fault-tolerant matching according to claim 8, characterized in that, Extracting the original ship name string from the natural language text using a large model includes the following steps: A pre-trained language model based on the Transformer architecture is used; Fine-tuning of the model for text data in the shipping industry; The fine-tuned model was used to identify and extract ship name entities from the natural language text.

10. The ship name standardization method based on large model and multi-level fault-tolerant matching according to claim 8, characterized in that, The original ship names are associated with the basic data dictionary through a multi-level matching algorithm, which includes the following steps: The original ship name string is matched using a precise matching step based on a hash index. The original ship name string is subjected to fault-tolerant matching through a fuzzy matching step, based on edit distance or Jaro-Winkler similarity. The original ship name string is transliterated into Chinese and English and matched with pinyin variants through a transliteration conversion step. The original ship name string is semantically similar to each other in the vector space through a vector retrieval step.

Images