Multi-source heterogeneous data-based Yangtze River shipping data medium-station construction method and system

Abstract

The invention discloses a method and a system for constructing Yangtze River shipping data based on multi-source heterogeneous data. The method comprises the following steps: step 101, acquiring heterogeneous data of Yangtze River shipping of a plurality of data platforms, extracting business processes in the heterogeneous data, and extracting semantic triples (business entities, attributes and connection relationships) from the business processes; step 102, performing normalization processing on fields representing the same service in the heterogeneous data, and setting a unified naming rule to perform unified naming on the normalized fields to form a standard word list; and step 103, mapping fields in the standard word list according to a semantic triple form, finally generating a plurality of semantic triples of the heterogeneous data, and forming a data middle table of the Yangtze River shipping data.

Description

Technical Field

[0001] The present invention belongs to the technical field of Yangtze River shipping data middle platform construction, and more specifically, relates to a Yangtze River shipping data middle platform construction method and system based on multi-source heterogeneous data. Background Art

[0002] The "data middle platform" has become a popular concept in recent years within enterprise digital transformation. It serves as an intermediate platform that helps enterprises centrally manage and share data resources. Simply put, the data middle platform acts as an internal "data supermarket" or "data operating system," integrating, processing, and sharing data across various business systems, supporting rapid response and innovation across various front-end business systems.

[0003] However, there is currently no data middle platform for Yangtze River shipping data. In order to improve the efficiency of using shipping data, there is an urgent need for a technical solution that can build a Yangtze River shipping data middle platform for multi-source heterogeneous data, thereby solving the problem of low efficiency in processing multi-source heterogeneous data. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a method for constructing a Yangtze River shipping data center based on multi-source heterogeneous data, including:

[0005] Step 101: Obtain heterogeneous data of Yangtze River Shipping from multiple data platforms, extract business processes from the heterogeneous data, and extract semantic triples of (business entity, attribute, connection relationship) from the business processes;

[0006] Step 102: normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary;

[0007] Step 103, mapping the fields in the standard vocabulary in the form of the semantic triples, and finally generating multiple groups of semantic triples of the heterogeneous data, and forming a data middle platform for the Yangtze River shipping data.

[0008] Furthermore, it also includes: setting a data query interface, when a user queries through an entity, outputting the attributes and connection relationships corresponding to the query entity to the user.

[0009] Furthermore, fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

[0010] Furthermore, the step 102 further includes: extracting metadata of the heterogeneous data, concatenating relevant information of fields in the metadata, and generating a description text of the fields;

[0011] Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

[0012] Furthermore, the semantic triples of (business entity, attribute, connection relationship) extracted from the business process in step 101 include: extracting semantic events in the heterogeneous data, binding participating entities and their roles to each semantic event, and forming semantic triples of (business entity (role), attribute, connection relationship) corresponding to the semantic event.

[0013] The present invention also proposes a Yangtze River shipping data middle platform construction system based on multi-source heterogeneous data, including:

[0014] A semantic triple extraction module is used to obtain heterogeneous data of Yangtze River Shipping from multiple data platforms, extract business processes from the heterogeneous data, and extract semantic triples (business entities, attributes, connection relationships) from the business processes;

[0015] A standardization module is used to normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary;

[0016] Construct a data middle platform module to map the fields in the standard vocabulary in the form of the semantic triples, and finally generate multiple groups of semantic triples of the heterogeneous data to form a data middle platform for the Yangtze River shipping data.

[0017] Furthermore, it also includes: setting a data query interface, when a user queries through an entity, outputting the attributes and connection relationships corresponding to the query entity to the user.

[0018] Furthermore, fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

[0019] Furthermore, the standardization module further includes: extracting metadata of the heterogeneous data, concatenating relevant information of fields in the metadata, and generating description text of the fields;

[0020] Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

[0021] Furthermore, the semantic triples extracted from the business process in the semantic triple extraction module include: extracting semantic events from the heterogeneous data, binding participating entities and their roles to each semantic event, and forming (business entities (roles), attributes, connection relationships) corresponding to the semantic events.

[0022] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0023] Through the above technical solutions, the present invention forms a data center for Yangtze River shipping data, which is convenient for querying relevant shipping data and improving data query and usage efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the method of embodiment 1 of the present invention;

[0025] Figure 2 is a system structure diagram of embodiment 2 of the present invention;

[0026] Figure 3 It is an overall schematic diagram of Example 1 of the present invention. DETAILED DESCRIPTION

[0027] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] The method provided by the present invention can be implemented in the following terminal environment, wherein the terminal may include one or more of the following components: a processor, a storage medium, and a display screen. The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0029] A processor can include one or more processing cores. It connects various components within the terminal using various interfaces and circuits. It executes instructions, programs, code sets, or instruction sets stored in storage media, and accesses data stored in storage media to perform various terminal functions and process data.

[0030] The storage medium may include a random access memory (RAM) or a read-only memory (ROM). The storage medium may be used to store instructions, programs, codes, code sets, or instructions.

[0031] The display is used to show the user interface of each application.

[0032] In addition, those skilled in the art will appreciate that the structure of the terminal described above does not limit the terminal. The terminal may include more or fewer components, or a combination of certain components, or a different arrangement of components. For example, the terminal may also include a radio frequency circuit, an input unit, a sensor, an audio circuit, a power supply, and other components, which will not be described in detail here.

[0033] Example 1

[0034] like Figure 1 and 3 As shown, this embodiment proposes a method for constructing a Yangtze River shipping data center based on multi-source heterogeneous data, including:

[0035] Step 101: Obtain heterogeneous data on Yangtze River shipping from multiple data platforms, extract business processes (such as ship scheduling, berthing operations, logistics and transportation, route planning, etc.) from the heterogeneous data, extract semantic triples of (business entity, attribute, connection relationship) from the business processes, and organize the semantic triples into a concept map. For example, business entities can be ships, ports, routes, crew members, cargo, voyages, etc., and connection relationships can be "ship belongs to the company", "ship berths at the port", "voyage is executed in a time period", etc., as shown in the following table;

[0036]

[0037] Preferably, the heterogeneous data of Yangtze River shipping on multiple data platforms can be AIS (automatic identification system) data, radar and satellite remote sensing data, IoT sensor data (terminal, ship, waterway equipment), shipping company ERP / TMS system data and meteorological, hydrological, and traffic supervision data, etc.

[0038] Specifically, the semantic triples of (business entity, attribute, connection relationship) extracted from the business process in step 101 include: extracting semantic events in the heterogeneous data, binding participating entities and their roles to each semantic event, and forming semantic triples of (business entity (role), attribute, connection relationship) corresponding to the semantic event.

[0039] Preferably, ship berthing, route execution, cargo loading and unloading, etc. are modeled as semantic events, and each event is bound to the participating entities and their roles (such as "ship [executor] berthing event", "port [place of occurrence]").

[0040] Step 102: normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary;

[0041] Preferably, information such as field names, field descriptions, types, and physical units are extracted from various heterogeneous data sources, and fields expressing the same business concept in different systems are normalized, such as "mmsi", "vessel_id", and "ship_code" are unified into "ShipID", and a unified naming rule is established, such as using camel case naming for fields and using international standards for units (such as speed is unified into km / h), to form a complete standard vocabulary.

[0042] Specifically, the step 102 further includes: extracting metadata of the heterogeneous data, concatenating relevant information of fields in the metadata, and generating a description text of the fields;

[0043] Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

[0044] Preferably, meta-information such as field name (name), field comment (comment), field type, table name, module name, etc. in each system is collected, and the field-related information is spliced ​​into "description text" for the model to understand, such as: field name: mmsi; comment: ship unique identification code; table: vessel_info → construct description: mmsi-ship unique identification code (table: vessel_info), then remove special characters, unify the language style, such as all lowercase, simplified verb form, etc., use the BERT model to generate a fixed-dimensional embedding vector (such as 768 dimensions) for each field description, and calculate the cosine similarity for each pair of field vectors. The higher the value, the closer the semantics.

[0045] Step 103, map the fields in the standard vocabulary in the form of the semantic triples, and finally generate multiple groups of semantic triples of the heterogeneous data, and form a data center for the Yangtze River shipping data. The specific mapping relationship is shown in the table below.

[0046]

[0047]

[0048] Preferably, the fields in the standard vocabulary are classified into "classes (entities)" (such as ships) and "attributes" (such as captain, draft depth), and their hierarchical structure is defined (such as ships are a type of transportation tool), and the connection relationship between classes (entities) is defined: such as "ships perform voyages", "voyages pass through routes", and "ships dock at ports".

[0049] Specifically, it also includes: setting a data query interface, and when a user queries through an entity, outputting the attributes and connection relationships corresponding to the query entity to the user.

[0050] Optimally, build entity recognition and normalization services: identify "Jianghai No. 1" and "JH-1" as the same ship, and provide a unified data access API to support business systems to directly access semantic data;

[0051] Specifically, fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

[0052] Example 2

[0053] like Figure 2 As shown, this embodiment proposes a Yangtze River shipping data middle platform construction system based on multi-source heterogeneous data, including:

[0054] A semantic triple extraction module is used to obtain heterogeneous data of Yangtze River Shipping from multiple data platforms, extract business processes from the heterogeneous data, and extract semantic triples (business entities, attributes, connection relationships) from the business processes;

[0055] Specifically, the semantic triples of (business entity, attribute, connection relationship) extracted from the business process in step 101 include: extracting semantic events in the heterogeneous data, binding participating entities and their roles to each semantic event, and forming semantic triples of (business entity (role), attribute, connection relationship) corresponding to the semantic event.

[0056] A standardization module is used to normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary;

[0057] Specifically, the step 102 further includes: extracting metadata of the heterogeneous data, concatenating relevant information of fields in the metadata, and generating a description text of the fields;

[0058] Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

[0059] Construct a data middle platform module to map the fields in the standard vocabulary in the form of the semantic triples, and finally generate multiple groups of semantic triples of the heterogeneous data to form a data middle platform for the Yangtze River shipping data.

[0060] Specifically, it also includes: setting a data query interface, and when a user queries through an entity, outputting the attributes and connection relationships corresponding to the query entity to the user.

[0061] Specifically, fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

[0062] Example 3

[0063] An embodiment of the present invention also proposes a storage medium storing a plurality of instructions, wherein the instructions are used to implement the method for constructing a Yangtze River shipping data middle platform based on multi-source heterogeneous data.

[0064] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0065] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: Step 101, obtaining heterogeneous data of Yangtze River Shipping from multiple data platforms, extracting business processes from the heterogeneous data, and extracting semantic triples of (business entity, attribute, connection relationship) from the business processes;

[0066] Specifically, the semantic triples of (business entity, attribute, connection relationship) extracted from the business process in step 101 include: extracting semantic events in the heterogeneous data, binding participating entities and their roles to each semantic event, and forming semantic triples of (business entity (role), attribute, connection relationship) corresponding to the semantic event.

[0067] Step 102: normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary;

[0068] Specifically, the step 102 further includes: extracting metadata of the heterogeneous data, concatenating relevant information of fields in the metadata, and generating a description text of the fields;

[0069] Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

[0070] Step 103, mapping the fields in the standard vocabulary in the form of the semantic triples, and finally generating multiple groups of semantic triples of the heterogeneous data, and forming a data middle platform for the Yangtze River shipping data.

[0071] Specifically, it also includes: setting a data query interface, and when a user queries through an entity, outputting the attributes and connection relationships corresponding to the query entity to the user.

[0072] Specifically, fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

[0073] Example 4

[0074] An embodiment of the present invention also proposes an electronic device, including a processor and a storage medium connected to the processor, wherein the storage medium stores multiple instructions, which can be loaded and executed by the processor to enable the processor to execute the method for constructing a Yangtze River shipping data middle platform based on multi-source heterogeneous data.

[0075] Specifically, the electronic device of this embodiment may be a computer terminal, which may include: one or more processors, and a storage medium.

[0076] Among them, the storage medium can be used to store software programs and modules, such as a method for constructing a Yangtze River shipping data middle platform based on multi-source heterogeneous data in an embodiment of the present invention, and corresponding program instructions / modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the storage medium, that is, realizing the above-mentioned method for constructing a Yangtze River shipping data middle platform based on multi-source heterogeneous data. The storage medium may include high-speed random storage media, and may also include non-volatile storage media, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state storage media. In some instances, the storage medium may further include a storage medium remotely located relative to the processor, and these remote storage media may be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0077] The processor can call the information and application stored in the storage medium through the transmission system to perform the following steps: Step 101, obtain heterogeneous data of Yangtze River Shipping from multiple data platforms, extract business processes in the heterogeneous data, and extract semantic triples of (business entity, attribute, connection relationship) from the business processes;

[0078] Specifically, the semantic triples of (business entity, attribute, connection relationship) extracted from the business process in step 101 include: extracting semantic events in the heterogeneous data, binding participating entities and their roles to each semantic event, and forming semantic triples of (business entity (role), attribute, connection relationship) corresponding to the semantic event.

[0079] Step 102: normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary;

[0080] Specifically, the step 102 further includes: extracting metadata of the heterogeneous data, concatenating relevant information of fields in the metadata, and generating a description text of the fields;

[0081] Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

[0082] Step 103, mapping the fields in the standard vocabulary in the form of the semantic triples, and finally generating multiple groups of semantic triples of the heterogeneous data, and forming a data middle platform for the Yangtze River shipping data.

[0083] Specifically, it also includes: setting a data query interface, and when a user queries through an entity, outputting the attributes and connection relationships corresponding to the query entity to the user.

[0084] Specifically, fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

[0085] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0086] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the system embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0088] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0089] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only storage medium (ROM, Read-Only Memory), random access storage medium (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and other media that can store program code.

[0091] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for constructing a Yangtze River shipping data center based on multi-source heterogeneous data, characterized in that: include: Step 101: Obtain heterogeneous data of Yangtze River Shipping from multiple data platforms, extract business processes from the heterogeneous data, and extract semantic triples of (business entity, attribute, connection relationship) from the business processes; Step 102: normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary; Step 103, mapping the fields in the standard vocabulary in the form of the semantic triples, and finally generating multiple groups of semantic triples of the heterogeneous data, and forming a data middle platform for the Yangtze River shipping data.

2. The method for constructing a Yangtze River shipping data middle platform based on multi-source heterogeneous data according to claim 1 is characterized in that: Also includes: Set up a data query interface. When a user queries an entity, the attributes and connection relationships corresponding to the query entity are output to the user.

3. The method for constructing a Yangtze River shipping data middle platform based on multi-source heterogeneous data according to claim 2 is characterized in that: Fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

4. The method for constructing a Yangtze River shipping data middle platform based on multi-source heterogeneous data according to claim 1 is characterized in that: The step 102 further includes: extracting metadata of the heterogeneous data, concatenating relevant information of fields in the metadata, and generating description text of the fields; Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

5. The method for constructing a Yangtze River shipping data center based on multi-source heterogeneous data according to claim 1 is characterized in that: The semantic triples of (business entity, attribute, connection relationship) extracted from the business process in step 101 include: extracting semantic events in the heterogeneous data, binding participating entities and their roles to each semantic event, and forming semantic triples of (business entity (role), attribute, connection relationship) corresponding to the semantic event.

6. A Yangtze River Shipping Data Middle Platform Construction System Based on Multi-Source Heterogeneous Data, Characterized by: include: A semantic triple extraction module is used to obtain heterogeneous data of Yangtze River Shipping from multiple data platforms, extract business processes from the heterogeneous data, and extract semantic triples (business entities, attributes, connection relationships) from the business processes; A standardization module is used to normalize the fields representing the same business in the heterogeneous data, and set a unified naming rule to uniformly name the normalized fields to form a standard vocabulary; Construct a data middle platform module to map the fields in the standard vocabulary in the form of the semantic triples, and finally generate multiple groups of semantic triples of the heterogeneous data to form a data middle platform for the Yangtze River shipping data.

7. A Yangtze River Shipping Data Middle Platform Construction System Based on Multi-Source Heterogeneous Data as claimed in claim 6, characterized in that: Also includes: Set up a data query interface. When a user queries an entity, the attributes and connection relationships corresponding to the query entity are output to the user.

8. A Yangtze River Shipping Data Middle Platform Construction System Based on Multi-Source Heterogeneous Data as claimed in claim 7, characterized in that: Fuzzy matching of natural language to entities or connection relationships is achieved through text vector retrieval.

9. The Yangtze River Shipping Data Middle Platform Construction System Based on Multi-Source Heterogeneous Data as claimed in claim 6, characterized in that: The standardization module further includes: extracting metadata of the heterogeneous data, splicing relevant information of fields in the metadata, and generating description text of the fields; Based on the field description text and the BERT model, a fixed-dimensional field vector is generated. The cosine similarity is calculated for each pair of field vectors. Each pair of fields whose cosine similarity exceeds the similarity threshold is classified as a field of the same business.

10. A Yangtze River Shipping Data Middle Platform Construction System Based on Multi-Source Heterogeneous Data as claimed in claim 6, characterized in that: The semantic triple extraction module extracts semantic triples (business entities, attributes, connection relationships) from the business process, including: extracting semantic events from the heterogeneous data, binding participating entities and their roles to each semantic event, and forming (business entities (roles), attributes, connection relationships) corresponding to the semantic events.

Images