A port multi-source heterogeneous data sharing method and system based on a data middle platform

Abstract

This invention belongs to the technical field of intelligent port management, specifically relating to a method and system for sharing multi-source heterogeneous data in ports based on a data platform. The method involves steps such as data acquisition, aggregation and classification, structured data conversion, data storage management, semantic tagging, and sharing and publishing. By determining the data structure type, the method employs field mapping or semantic extraction strategies to uniformly convert structured and unstructured data into a format recognizable by the platform, and implements a tag-driven data sharing mechanism. The system includes modules for data acquisition, classification, processing, verification, and interface publishing, supporting data traceability, tag consistency verification, and on-demand sharing strategies. This invention improves the efficiency and accuracy of multi-source data fusion and sharing in ports, and is suitable for data connectivity and business collaboration needs between multiple departments.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent port management, specifically relating to a method and system for sharing multi-source heterogeneous data in ports based on a data middle platform. Background Technology

[0002] In the process of port intelligent development, the integration and utilization of data resources is increasingly becoming an important manifestation of core competitiveness. As the scope of port operations continues to expand, involving multiple fields such as ship scheduling, cargo loading and unloading, equipment monitoring, safety management, and energy consumption monitoring, massive amounts of data are generated. These data come from diverse sources and are in various formats, covering structured, semi-structured, and unstructured data, exhibiting typical "multi-source heterogeneous" characteristics.

[0003] In existing technologies, some ports have attempted to use data warehouses, information integration buses, or platform interfaces to achieve data transmission and sharing between different business systems. However, traditional data warehouses have bottlenecks in processing real-time and unstructured data, making it difficult to support dynamic and high-frequency port operations. While information buses can solve the system silo problem to some extent, they suffer from high interface coupling, high maintenance costs, and a lack of unified data governance capabilities. Some point-to-point integration methods based on customized interfaces lack standardization mechanisms, easily leading to a fragmented sharing pattern, which is not conducive to the formation of data assets. Furthermore, with the continuous enrichment of data sources and the constant changes in the data types and interface standards of newly connected systems, traditional methods are insufficient in terms of adaptability, scalability, and data quality assurance, making it difficult to meet the urgent need for efficient sharing of multi-source data in smart port construction.

[0004] To adapt to the new requirements of smart ports in terms of unified data management and efficient cross-system circulation, it is urgent to build a data sharing mechanism with a data platform at its core, improve the integration capability and sharing efficiency of multi-source heterogeneous data, and promote the overall business collaboration level of the port. Summary of the Invention

[0005] To address the above problems, the present invention aims to propose a method for sharing multi-source heterogeneous data in ports based on a data middle platform, comprising the following steps: S1. Data Acquisition: Based on a preset list of end nodes, acquire multi-source data from various production systems inside and outside the port. The end nodes include ship dynamic monitoring system, container loading and unloading scheduling system, cargo storage management system, port security monitoring system, and customs clearance system. S2. Data Collection and Classification: The collection engine is used to verify, synchronize and unify the timestamps of the multi-source data. The collected data is classified according to the preset classification criteria, which include four dimensions: data source type, data time attribute, data structure format and usage label. S3. Structured Transformation: Determine the structure type of the collected data. If it is structured data, perform field mapping and standard transformation. If it is unstructured data, perform word segmentation and semantic tag extraction, and uniformly transform it into formatted data that the middle platform can recognize. S4. Data entry into the data platform: After the collected data has been cleaned and standardized, it is stored in the data storage module of the data platform and its data source, timestamp and data provider identification information are recorded. S5. Tagging Processing: Based on the port business directory and existing rule set, semantic tags are extracted and multi-dimensional verification is performed on the inbound data to form a multi-tag data set that can be shared. S6. Shared Release: Based on the usage records and weighting strategies of external call interfaces, the tagged data is released to the outside world in the form of interfaces, supporting subscription delivery and on-demand call methods, and the configuration information of the middle platform data call interface is synchronously distributed to the access party.

[0006] As a preferred technical solution, the structured processing of the acquired data includes: S31. For structured data, obtain its field names and compare them with the preset standard field set; S32. Determine whether the field name is consistent with or can be mapped to a standard field; S33. If they match, store them directly; if they can be mapped, execute the preset field mapping rules and store them; if they cannot be mapped, perform temporary storage of the newly added field and store the field in the structured buffer pool. S34. For unstructured data, after content segmentation and semantic tag extraction, it is uniformly converted into a structured format.

[0007] As a preferred technical solution, step S33 includes: S331. When a field name is inconsistent with a standard field, first extract its prefix, core words, and suffix information according to the composition of the field name, and then perform a three-stage matching with the registered field rule table. S332. If a field name contains an item with a structure similar to part of the rule table, mark the field as a "suspected mapping field" and record its similarity range and difference segment with the target field. S333. Construct a field context structure graph based on the relationship between adjacent fields in the record where the field is located. If the frequency of the field appearing in combination with known fields in the context exceeds a preset threshold, promote it to "field to be confirmed" and temporarily store the field and the entire context structure in the structured buffer pool. S334. For fields that do not meet the above conditions, mark them directly as "unmapped fields" and enter the buffer pool in the original field format. The mapping path will be completed through manual approval later.

[0008] As a preferred technical solution, step S34 includes: S341. Perform content block-based splitting operation on unstructured data, divide the data text into three categories: title paragraph, body paragraph and footnote paragraph, and extract the first and last keywords, time sentences and operation verb sequences of each paragraph to construct a preliminary semantic unit set. S342. Classify and organize each semantic unit according to its location and correlation density, and prioritize the retention of semantic units that appear repeatedly in multiple paragraphs or have clear time indications, and mark them as core expression units. S343. Based on the port business domain rule base, the core expression unit is compared with the preset tag set. When there are multiple matching tags, the final tag is determined according to the priority of the source paragraph and the keyword hit weight. S344. Organize the tags and their corresponding content fragments into a key-value structure, and number and identify them in conjunction with the source information, generation time and identification order of the original text, and uniformly transform them into a structured data format with traceability.

[0009] As a preferred technical solution, step S5 includes the following tagging process: Call the existing semantic annotation rule library to perform content recognition and tag matching on the input data; For structured data, labels are determined based on the association between field names and their uses, and consistency checks are performed in conjunction with the range of field values. For unstructured data, tags are extracted based on file name, text content, and associated time information, and their consistency with the tags of the same source data is verified. Based on the call logs and historical rule records, missing tags are filled in or conflicting tags are identified, forming a multi-tag data set that can be shared.

[0010] As a preferred technical solution, the data storage module includes a time field, which is used to record the entry time, update time and retrieval time of each data.

[0011] As a preferred technical solution, the external calling interface includes a data subscription mode and a call return mode. The former supports pushing data according to a predetermined period, while the latter supports calling on demand and returning a specified data set.

[0012] As a preferred technical solution, the configuration information for calling the interface includes the interface address, access token, call frequency limit, data format requirements, and error handling mechanism.

[0013] This invention also provides a system for sharing multi-source heterogeneous data in ports based on a data middle platform, used to implement the method, including: The first acquisition module is used to acquire port multi-source data based on a preset list of end nodes; The second acquisition module is used to collect and process the acquired multi-source port data. The classification module is used to classify the multi-source data after data aggregation into subsets of data to be processed according to preset standards; The judgment module is used to determine the structure type of each subset of data to be processed. An execution module is used to perform standardization processing according to the structure type and store the data in the data warehouse module; The verification module is used to generate labeled data based on a multi-dimensional verification mechanism. The interface module is used to publish tagged data to the outside world in the form of an interface based on the usage records and weighting strategy of external interface calls.

[0014] As a preferred technical solution, the first acquisition module includes a data acquisition submodule and a synchronization and time synchronization submodule. The former acquires data based on the list of end nodes, while the latter performs time synchronization processing based on the port's unified time server.

[0015] As a preferred technical solution, the classification module supports multi-level classification based on data structure information, metadata, timestamps, and usage tags.

[0016] As a preferred technical solution, the execution module includes a structured data processing submodule and an unstructured conversion submodule. The former performs field alignment and formatting processing, while the latter performs word segmentation and tag extraction and converts the data into a structured format.

[0017] The beneficial effects of this invention are as follows: This invention introduces a structured and unstructured data fusion processing mechanism, constructing a dynamic transformation framework compatible with field mapping rules, semantic extraction processes, and contextual relationship analysis. This overcomes the problems of limited structure types and ineffective integration of unstructured data in existing technologies. In particular, step S33 employs a combination of field context structure diagrams and three-segment field name analysis, ensuring that unknown fields are no longer simply discarded but are instead entered into a buffer pool as "fields to be confirmed" or "suspected mapping fields" for further comparison. This significantly improves the accuracy of data mapping and field identification efficiency between heterogeneous port systems.

[0018] In terms of unstructured data processing, the content block splitting and semantic tag aggregation strategy designed in this invention can extract semantic units closely related to port operations based on business context, and achieve dynamic tag generation by combining tag priority and hit weight. This mechanism avoids the one-sided word segmentation and fixed template recognition path of unstructured data in traditional solutions, making the system more adaptable and semantically linked, especially suitable for the standardized processing needs of semi-structured documents such as ship notices, operation notices, and safety inspection notices.

[0019] Furthermore, this invention constructs an interface publishing strategy based on call records and tag quality feedback mechanisms. By dynamically adjusting data sharing priorities through subscription call weights, it achieves precise alignment between shared data and real business needs. The system introduces tag consistency verification and tag conflict resolution mechanisms when publishing interfaces, effectively preventing misuse of business data due to tag confusion. This enhances the controllability, traceability, and compliance of data sharing, adapting to the refined management requirements of multi-departmental and cross-system collaborative supervision. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0021] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention. Example 1

[0022] To further achieve unified management and flexible sharing of various heterogeneous data in ports, this implementation method provides a port multi-source heterogeneous data sharing method based on a data middle platform. It is applicable to multiple business scenarios covering port logistics, operations, safety, customs clearance, etc., and can effectively solve problems such as scattered data sources, diverse formats, inconsistent standards, and inefficient sharing.

[0023] This method is based on a unified data platform architecture, and its process design revolves around six core stages: "collection, aggregation, structuring, storage, tagging, and sharing." It also constructs multiple traceable and scalable data processing and interface mechanisms, such as... Figure 1 As shown, the specific steps include the following: S1. Data Acquisition: The system first constructs a list of information endpoints covering all port operations. This list includes the name of each system, responsible unit, business domain, interface protocol, data item list, field definition descriptions, and data collection cycle requirements. Endpoints include: Vessel dynamic monitoring system: including AIS automatic identification system and VTS traffic monitoring platform, providing information such as vessel name, IMO number, berthing and departure time, speed, heading, and anchoring status; Container handling scheduling systems, such as TOS (Terminal Operating System), provide structured records such as work instructions, crane status, container number trajectory, and work shifts. Warehouse management systems, such as WMS systems, provide records of goods entering and leaving the warehouse, warehouse space occupancy, batch numbers, and owner information. Security monitoring systems include video surveillance (image streams, alarm events), facial recognition, access control records, etc., and are mostly in unstructured formats. Customs clearance and declaration systems include single window platforms, electronic port systems, port access systems, etc., involving fields such as customs declaration number, mode of transport, supervision site code, and cargo owner tax number.

[0024] Data acquisition methods include real-time push, polling retrieval, file crawling, and asynchronous subscription. Each acquisition method is configured with a corresponding acquisition adapter and protocol converter. The system appends a unique source system identifier, data type identifier, data access time, and receiving device number to each acquisition record to ensure the consistency and traceability of the data source identity and time.

[0025] S2. Data Collection and Classification: The system transmits the collected raw data to the preprocessing module of the data platform for aggregation and preliminary multi-dimensional organization. Aggregation operations include: data format recognition and conversion (e.g., XML to JSON); timestamp unification and time zone correction; primary key detection and deduplication; and structural anomaly detection and automatic repair (e.g., padding missing fields with zeros).

[0026] After initial data cleaning, the system categorizes the data according to the following four dimensions: 1) Data source type: Classified by the business system to which it belongs, such as "VTS system", "customs declaration platform", "video stream acquisition gateway", etc.; 2) Time attributes: such as real-time data (second level), batch data (daily level), historical stock data (monthly level), event-triggered data, etc.; 3) Structure format: including purely structured (such as database table records), semi-structured (such as logs, XML segments), and unstructured (such as images, scanned documents, audio, etc.); 4) Data usage tags: Divided according to business application layer, such as for “ship scheduling analysis”, “on-site security linkage”, “energy efficiency assessment”, “compliance audit”, etc.

[0027] The classification results are written to the data index table, with each data entry accompanied by a unique classification code, which is used by the structured conversion module to refer to the classification path and execute personalized processing logic.

[0028] S3, Structured Transformation: This step proceeds to either structured data processing or unstructured data processing based on the aforementioned classification results.

[0029] Structured data processing: S31: Extract the data field name set, such as {“ETA”,“Berth_ID”,“Cargo_Type”}, and load the preset middle platform standard field dictionary (including field English abbreviations, Chinese explanations, type definitions, unit ranges, etc.). S32: Compare the collected fields with the standard field dictionary one by one to determine whether they are completely consistent, whether the rules can be mapped, or whether they do not match. S33: If the fields are completely identical or can be mapped, then standardized naming and type conversion are performed before data entry; if the fields do not match, then the following unique field identification mechanism is used: S331: Extract field features according to the three-part structure of "prefix-core word-suffix" (e.g., the field "plan_arr_time" can be broken down into "plan" + "arr" + "time"). S332: Compare the three-segment structure of the field with the standard field lexicon for similar structures. If a match is found, mark it as a "suspected mapped field". S333: Analyze the context combination of this field in its record (such as whether it appears together with "ship_id" and "cargo_name"), construct the field context graph, and if the combination frequency exceeds the set threshold, it is promoted to "field to be confirmed" and temporarily stored in the buffer pool. S334: If there is no contextual support or the semantic structure is unclear, mark it as "unmapped field", put it into the buffer pool and mark it for manual review or subsequent rule updates.

[0030] S34 Unstructured Data Processing: S341: Parse the original text content and divide it into heading paragraphs, body paragraphs, and footnote paragraphs according to structure, and extract keywords (such as "ship name", "time", "operation type"), time statements, action verbs, and proper nouns; S342: Analyze the occurrence density, relevance, and order of keywords in different paragraphs, and mark consistently high-frequency words across paragraphs as "core expression units"; S343: Compare the core expression units with the port business rule base one by one, and generate semantic tags based on the priority and weight of the hit keywords; S344: Organize semantic tags and their corresponding fragment content into a key-value structure, and add meta-information such as source file name, collection time, paragraph position, and processing batch number to generate structured data records with traceability capabilities.

[0031] S4. Data entry into the data platform: The structured data is sent to the data warehouse module in the middle platform. The fields of the inbound record include: standard field structure, field mapping strategy number, data source system identifier, inbound timestamp, archive tag, tag set, version control number, call frequency counter, etc.

[0032] The data warehousing module divides multiple datasets according to business domains, such as "ship status dataset", "operation scheduling dataset" and "security event dataset", and configures a hot and cold data partitioning mechanism.

[0033] The hot data area retains data that has been accessed most frequently recently and is stored in high-performance storage. The cold data area retains data that has been accessed less frequently in the past, archives it regularly, and restores it as needed. Each data entry supports version snapshot recording and rollback functionality, and supports data error correction and audit tracing.

[0034] S5. Tagging Processing: To improve query efficiency, controllability, and compatibility with upstream and downstream systems during data sharing, the system performs multi-dimensional semantic tagging on the data.

[0035] Structured fields: directly match tag items by field name, such as "vessel_name" matching "ship name"; Field values: Based on the data value rule library, determine whether the field value is valid or abnormal, and assign auxiliary labels such as "abnormal value", "field missing", "timeout"; Unstructured fields: Tags are generated based on keywords in the text, source paths, file naming conventions, etc., and cross-validation is performed using tags from the same source data to improve tag consistency; If a label is missing or there is a conflict, the system can automatically fill in or replace the conflicting label based on the call log, context label matrix and rule revision records, and finally form a complete, unique and controllable multi-label data set.

[0036] S6, Shared Publishing: Once tagged data is entered into the sharing and publishing module, it supports two data sharing methods: 1) Subscription delivery mode: The business system configures subscription conditions (such as data domain, update frequency, field range), and the system pushes qualified data to the target API endpoint at a set period (such as every 30 minutes); subscription expiration reminders and change updates are supported.

[0037] 2) Call return mode: The caller requests a specific dataset based on the authorized interface. The system supports functions such as field filtering, tag filtering, pagination query, aggregation statistics, and format conversion (such as JSON to CSV) to meet flexible calling needs.

[0038] The shared interface configuration includes: interface path, call token, call frequency limit, request parameter description, error code description, rate limiting strategy (such as QPS limit, concurrency limit) and access audit log.

[0039] Each call is written to a shared call log table, recording information such as caller ID, time, request conditions, number of returned records, interface status code, and tag usage, which serves as the basis for subsequent call billing, permission assessment, behavior analysis, and security auditing.

[0040] In summary, this method constructs a data sharing platform solution for port scenarios with horizontal integration, vertical penetration, traceability, and scalability through unified collection of multi-source data, standardized transformation of structural semantics, semantic enhancement tagging, and dynamic permission sharing mechanism. It has good adaptability for promotion and engineering implementation. Example 2

[0041] To more clearly illustrate the technical implementation process of the "System for Sharing Multi-Source Heterogeneous Data in Ports Based on Data Platform" proposed in this invention, the following device-type embodiments are given in conjunction with the specific application scenarios of this invention, in order to further explain the specific implementation methods of each functional module in the access, conversion, processing and sharing of heterogeneous data in ports.

[0042] In this embodiment, the data sharing system is deployed based on the port operation and management platform, adopting a microservice architecture and embedded in the core module of the port data platform. Each functional unit of the system runs as a service container in a distributed cluster environment, supporting elastic expansion and independent module upgrades. The system establishes a bidirectional communication channel with various port production systems through securely authenticated API interfaces, supporting data transmission based on SSL / TLS encryption protocols. The main data processing engine is based on a multi-threaded asynchronous scheduling mechanism to ensure real-time performance and stability under high-concurrency data writing and interface publishing scenarios.

[0043] Implementation of the first acquisition module: The primary function of the first acquisition module is to acquire multi-source heterogeneous port data based on a pre-defined list of endpoints. In this embodiment, this module specifically includes the following two sub-modules: Data Acquisition Submodule: This submodule is pre-configured with a set of end-node identification tables, including access addresses, interface keys, transmission protocol types (RESTful, MQTT, FTP, etc.), and data frequency parameters for the ship dynamic monitoring system, container loading and unloading scheduling system, cargo warehousing management system, port security monitoring system, and customs clearance system. Based on the access policies configured in the end-node identification tables, the data acquisition submodule periodically or in real-time retrieves various data packets and marks them with source system identifiers and timestamps.

[0044] Synchronization Time Synchronization Submodule: To ensure the integration of multi-source data on a unified timeline, this submodule synchronizes with the port's central clock server using the NTP protocol. All data is forcibly rewritten with standard timestamps before being stored in the database to ensure the accuracy of time correlation in subsequent analysis and tagging processes.

[0045] Implementation of the second acquisition module and classification module: The second acquisition module is responsible for the initial aggregation of the collected raw data. It employs the Extract phase of the ETL (Extract-Transform-Load) framework, parsing data packets through a unified temporary caching mechanism. The module performs content format validation, character encoding standardization, anomaly field identification, and data packet integrity comparison on various types of source data, and sends qualified data to the classification module.

[0046] Classification Module: This module introduces a multi-level classification engine to finely categorize data according to structure type, metadata, timestamps, and usage tags. The system's built-in data classification rules include, but are not limited to, multiple dimensions such as: structured / unstructured, static / dynamic data, real-time / batch processing, business / regulatory purposes, and form-based / document-based. Based on these rules, the module tags the data, forming multiple subsets of data to be processed, which are then either processed through structured methods or transformed from unstructured data.

[0047] Implementation of the judgment module and the execution module: Judgment Module: In this embodiment, the judgment module embeds an intelligent recognition engine that automatically identifies the data type based on features such as the data file extension, field characteristics, data density, and format style. If the data is structured (such as JSON, CSV, or XML format), it enters the structured path; if the data is unstructured (such as TXT, PDF, DOC, or image OCR results), it enters the unstructured path.

[0048] Execution Module: This module includes a structured data processing submodule and an unstructured data transformation submodule. The structured data processing submodule first obtains the field names from the structured data and compares them with a preset standard field set. During the comparison process, the system supports three strategies: direct matching, fuzzy matching, and rule mapping. If a field can be mapped, a standard field conversion operation is performed according to the field mapping table; if the field is unknown, it is temporarily stored in the structured buffer pool and marked as "field to be confirmed" or "unmapped field". In addition, this submodule also supports prefix and suffix parsing of field names, context graph construction, and clustering of similar fields.

[0049] Unstructured Transformation Submodule: This submodule first performs content segmentation, dividing the text content into title segments, body segments, and footnote segments, and extracting basic features such as keywords, time expressions, and verb phrases. Next, it calls the semantic analysis engine to perform vector encoding and semantic tag comparison on each segment, filtering out high-weight core semantic units and mapping them to a port business semantic database. Finally, it combines the extracted tags and content to construct structured key-value pair data, labeling the data source, generation time, and processing path.

[0050] Implementation of the data warehouse module: In this embodiment, after the structured and unstructured transformed data undergo standardization processing, they are uniformly stored in the data warehouse module of the middle platform. This module adopts a hybrid architecture of time-series and document-based databases. Structured data is written to the main database in a table structure, which includes data content fields, tag fields, time fields, data source identifiers, access control fields, etc. Unstructured data is stored in NoSQL databases as documents, which retain the original fragments, parsing results, semantic tags and mapping paths. All data entering the database is recorded with three timestamp fields: entry time, update time, and external call time, which are used to support version management, data traceability, and access logs.

[0051] In addition, the system is equipped with a data redundancy verification mechanism and a secure sandbox environment to prevent unauthorized data injection and label contamination.

[0052] Implementation of the verification module: This module is primarily used for tag validation and multi-dimensional consistency assessment to ensure data quality and tag usability. Specific steps include: For structured data, call the field label mapping table to confirm the usage matching relationship between field names and labels; By combining the field value range and unit standards, perform a comparison of data validity and label consistency. For unstructured data, cross-validate the extracted tags based on parameters such as file name, text content, and generation time to see if they are consistent with the tags of the same source data; Introduce a historical call log and rule record library to automatically complete missing tags or determine the priority of conflicting tags; After successful verification, the system packages each tag and its associated data into a "multi-tag data set" and marks it as "publishable," thus entering the interface publishing preparation process.

[0053] Implementation of the interface module: This module is responsible for securely sharing "publishable" data sets with external systems via a pre-configured interface protocol. In this embodiment, the module supports the following two main modes: Data subscription mode: The access party submits a subscription request based on its own business needs, and the system pushes data with matching tags to the access party's interface address according to the set push period (such as daily or hourly).

[0054] Call return mode: The access party sends a parameterized call request via API, and the system returns a set of data that meets the conditions in real time, and records the call frequency, result status and error log.

[0055] The system will generate an interface configuration information document for each external interface, which includes: Interface address and access token for authentication; call frequency limit and data capacity limit; supported data formats (JSON / XML / CSV, etc.); error handling strategy (such as exception code definition, retry mechanism, etc.); logging level and backtracking window period.

[0056] To prevent data leakage and interface abuse, the system has set up a dynamic access token update mechanism to periodically change interface credentials and manage access subjects through a whitelist mechanism.

[0057] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for port multi-source heterogeneous data sharing based on a data platform, characterized in that, Includes the following steps: S1. Data Acquisition: Based on a preset list of end nodes, acquire multi-source data from various production systems inside and outside the port. The end nodes include ship dynamic monitoring system, container loading and unloading scheduling system, cargo storage management system, port security monitoring system, and customs clearance system. S2. Data Collection and Classification: The collection engine is used to verify, synchronize and unify the timestamps of the multi-source data. The collected data is classified according to the preset classification criteria, which include four dimensions: data source type, data time attribute, data structure format and usage label. S3. Structured Transformation: Determine the structure type of the collected data. If it is structured data, perform field mapping and standard transformation. If it is unstructured data, perform word segmentation and semantic tag extraction, and uniformly transform it into formatted data that the middle platform can recognize. S4. Data entry into the data platform: After the collected data has been cleaned and standardized, it is stored in the data storage module of the data platform and its data source, timestamp and data provider identification information are recorded. S5. Tagging Processing: Based on the port business directory and existing rule set, semantic tags are extracted and multi-dimensional verification is performed on the inbound data to form a multi-tag data set that can be shared. S6. Shared Release: Based on the usage records and weighting strategies of external call interfaces, the tagged data is released to the outside world in the form of interfaces, supporting subscription delivery and on-demand call methods, and the configuration information of the middle platform data call interface is synchronously distributed to the access party.

2. The method for port multi-source heterogeneous data sharing based on data platform according to claim 1, characterized in that, The structured processing of the collected data includes: S31. For structured data, obtain its field names and compare them with the preset standard field set; S32. Determine whether the field name is consistent with or can be mapped to a standard field; S33. If they match, store them directly; if they can be mapped, execute the preset field mapping rules and store them; if they cannot be mapped, perform temporary storage of the newly added field and store the field in the structured buffer pool. S34. For unstructured data, after content segmentation and semantic tag extraction, it is uniformly converted into a structured format.

3. The method for port multi-source heterogeneous data sharing based on data platform according to claim 1, characterized in that, In step S5, the tagging process includes: Call the existing semantic annotation rule library to perform content recognition and tag matching on the input data; For structured data, labels are determined based on the association between field names and their uses, and consistency checks are performed in conjunction with the range of field values. For unstructured data, tags are extracted based on file name, text content, and associated time information, and their consistency with the tags of the same source data is verified. Based on the call logs and historical rule records, missing tags are filled in or conflicting tags are identified, forming a multi-tag data set that can be shared.

4. The method for port multi-source heterogeneous data sharing based on data platform according to claim 1, characterized in that, The data storage module includes a time field, which is used to record the entry time, update time, and retrieval time of each data.

5. The method for port multi-source heterogeneous data sharing based on data hub according to claim 1, characterized in that, The external call interface includes a data subscription mode and a call return mode. The former supports pushing data according to a predetermined period, while the latter supports calling on demand and returning a specified data set.

6. The method for port multi-source heterogeneous data sharing based on data hub according to claim 1, characterized in that, The configuration information for calling the interface includes the interface address, access token, maximum call frequency, data format requirements, and error handling mechanism.

7. A system for port multi-source heterogeneous data sharing based on a data middle platform, for implementing the method according to any one of claims 1-6, characterized in that, include: The first acquisition module is used to acquire port multi-source data based on a preset list of end nodes; The second acquisition module is used to collect and process the acquired multi-source port data. The classification module is used to classify the multi-source data after data aggregation into subsets of data to be processed according to preset standards; The judgment module is used to determine the structure type of each subset of data to be processed. An execution module is used to perform standardization processing according to the structure type and store the data in the data warehouse module; The verification module is used to generate labeled data based on a multi-dimensional verification mechanism. The interface module is used to publish tagged data to the outside world in the form of an interface based on the usage records and weighting strategy of external interface calls.

8. The system for port multi-source heterogeneous data sharing based on data hub according to claim 7, characterized in that, The first acquisition module includes a data acquisition submodule and a time synchronization submodule. The former acquires data based on the list of end nodes, while the latter performs time synchronization processing based on the port's unified time server.

9. The system for sharing multi-source heterogeneous port data based on a data middle platform according to claim 7, characterized in that, The classification module supports multi-level classification based on data structure information, metadata, timestamps, and usage tags.

10. The system for sharing multi-source heterogeneous port data based on a data middle platform according to claim 7, characterized in that, The execution module includes a structured data processing submodule and an unstructured conversion submodule. The former performs field alignment and formatting processing, while the latter performs word segmentation and tag extraction and converts the data into a structured format.

Images