A method for generating a trade data mirror database

Abstract

This invention relates to the field of trade data knowledge graph construction technology, specifically a method for generating a trade data mirror database. The method includes: configuring heterogeneous data source access points and periodically capturing a set of original trade records, obtaining intermediate data after multi-stage verification. A domain model is introduced to deconstruct and reassemble the intermediate data into data fragments with model labels. A virtual storage partition is allocated to each data fragment, and a data association network with trade entities as the core and trade events as the context is constructed within each partition. The association networks of all partitions are aggregated, and a panoramic trade knowledge graph is generated through cross-citation and link fusion techniques. The topology and attribute data of this knowledge graph are persisted to different storage engines to form the final mirror database. This invention realizes the transformation of trade data from discrete records to a panoramic associated knowledge system, improving the efficiency of data association queries and the ability for in-depth analysis.

Description

Technical Field

[0001] This invention relates to the field of trade data knowledge graph construction technology, and in particular to a method for generating a trade data mirror database. Background Technology

[0002] Currently, in fields such as cross-border trade and supply chain finance, integrating multi-source heterogeneous trade data is fundamental to business analysis. Existing technologies primarily achieve this by establishing data warehouses or data lakes. The core process involves extracting, cleaning, and formatting raw data to ultimately form a structured collection of records for centralized storage. This model essentially manages data as discrete entries; the inherent business semantic relationships between data points are not explicitly modeled and solidified at the storage layer. Business queries heavily rely on pre-designed table join logic for real-time correlation calculations, failing to directly present and utilize the implicit network structure within the data.

[0003] Existing solutions have limitations when dealing with complex trade chains that span multiple sources and long cycles. Correlation analysis between data is often limited to a single record or specific data source, lacking effective technical means to globally integrate and verify the scattered relationships between different links and entities. This makes it difficult to construct a coherent and complete panoramic view of trade, and cannot effectively support in-depth mining and analysis of complex patterns and risk chains across entities and processes.

[0004] A new method for generating data mirrors is needed. This method should not only clean and reconstruct the data, but also proactively establish a semantic network of relationships between entities and events within data fragments based on domain knowledge before the data is stored. It should also possess the ability to intelligently stitch together disparate local networks to form a unified trade knowledge graph with a global perspective, thereby changing the mode of data integration and use. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for generating a trade data mirror database.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for generating a trade data mirror database, comprising: Configure the access endpoints for heterogeneous trade data sources and schedule data capture tasks according to preset trade cycle patterns to pull the original trade record set from the access endpoints.

[0007] The original trade record set is subjected to multi-stage verification, including integrity verification, logical contradiction investigation and compliance screening, to generate a verified intermediate trade data set.

[0008] A domain-driven trade model is introduced, which deconstructs the intermediate trade data set into factual data and dimensional data, and reorganizes them according to the mapping rules defined by the model to generate trade data fragments with model labels.

[0009] A virtual storage partition is allocated to each trade data fragment with a model label, and a data association network with trade entities as the core and trade events as the context is constructed within the virtual storage partition.

[0010] By aggregating the data relationships across all virtual storage partitions and using cross-citation and link fusion technologies, a panoramic trade knowledge graph is generated.

[0011] The topology and attribute data of the panoramic trade knowledge graph are persisted to different storage engines to form the final trade data mirror database.

[0012] Preferably, configuring the access endpoint of the heterogeneous trade data source and scheduling data capture tasks according to a preset trade cycle pattern to pull the original trade record set from the access endpoint includes: Register an endpoint configuration file containing authentication, protocol adaptation, and rate limiting parameters for each trade data source to be connected.

[0013] Analyze the temporal distribution characteristics of historical trade activities and develop a data capture task plan that includes trigger time, execution frequency, and priority.

[0014] According to the data capture task schedule, the corresponding data capture task is activated at the specified time, and a connection channel with the trade data source is established according to the parameters of the endpoint configuration file.

[0015] The connection channel reads incremental or full amounts of raw data packets and temporarily caches them as an unprocessed set of raw trade records.

[0016] Preferably, the multi-stage verification performed on the original trade record set includes integrity checks, logical inconsistency checks, and compliance screening, generating a verified intermediate trade data set, including: Iterate through the original set of trade records, check each record for missing fields or invalid formats based on the preset list of necessary fields, and mark and repair incomplete records.

[0017] Based on the trade business rules database, logical contradictions are checked in records that have completed integrity verification, and conflicting transaction information records are identified and isolated.

[0018] The system invokes an external trade compliance strategy engine to perform compliance screening on records that pass the logical inconsistency check, filtering out records that violate preset compliance terms.

[0019] All records that pass the screening are integrated to form a unified, time-aligned, and logically consistent set of all verified intermediate trade data.

[0020] Preferably, the introduced domain-driven trade model deconstructs the intermediate trade data set into factual data and dimensional data, and reorganizes them according to the mapping rules defined in the model to generate trade data fragments with model labels, including: Load a predefined domain-driven trade model that explicitly defines the types of trade facts, the hierarchical structure of dimensions, and the relationships between facts and dimensions.

[0021] Based on the domain-driven trade model, each record in the intermediate trade dataset is decomposed into a factual data part describing trade metrics and a dimensional data part describing the trade context.

[0022] Based on the mapping rules in the model, the split factual data and the corresponding dimensional data are re-associated and combined, and labeled with the business type tags defined by the model.

[0023] Each set of associated data tagged with a business type is encapsulated into a structured data packet, which is the trade data fragment with the model tag.

[0024] Preferably, the step of allocating a virtual storage partition for each trade data fragment with model labels, and constructing a data association network within the virtual storage partition with trade entities as the core and trade events as the context, includes: Based on the business type label of the trade data fragment, it is routed to logically isolated virtual storage partitions, each of which maintains an independent metadata directory.

[0025] Within each virtual storage partition, key trade entity objects in the trade data fragments are identified, and each entity object is created as a network node.

[0026] Extract specific trade events recorded in the trade data fragments, treat each event as a directed edge connecting the relevant entity nodes, and attach the event's time, attribute, and status information to the directed edge.

[0027] Based on the continuously imported trade data fragments, the nodes and edges within the virtual storage partition are dynamically updated, allowing the data association network to evolve over time.

[0028] Preferably, the aggregation of data association networks across all virtual storage partitions, through cross-citation and link fusion technologies, generates a panoramic trade knowledge graph, including: Access the metadata directories of all virtual storage partitions in parallel to obtain the node list and edge list of the data association network within each partition.

[0029] Run the cross-reference program to compare the node lists of different partitions, identify overlapping nodes that point to the same real-world trading entity, and create cross-partition hyperlinks for the overlapping nodes.

[0030] The link fusion process is executed to analyze trade event edges across partitions and connect discrete event edges belonging to the same trade chain or business process into a continuous semantic path.

[0031] By integrating all linked nodes and merged paths, a unified panoramic trade knowledge graph that supports global queries is formed.

[0032] Preferably, the step of persisting the topology and attribute data of the panoramic trade knowledge graph to different storage engines to form the final trade data mirror database includes: The topological structure data describing the connection relationship between nodes and edges is extracted from the panoramic trade knowledge graph, converted into a graph serialization format, and imported into a dedicated graph structure storage engine for storage.

[0033] Detailed attribute data attached to nodes and edges are extracted from the panoramic trade knowledge graph, converted into row and column format, and imported into a high-performance relational or key-value storage engine for storage.

[0034] Establish a bidirectional index mapping relationship between the graph structure storage engine and the relational or key-value storage engine to ensure that attribute data can be located through topological relationships, and vice versa.

[0035] The access interface to the panoramic trade knowledge graph is encapsulated into a unified database service package, which is the trade data mirror database.

[0036] Preferably, after forming the trade data mirror database, the process further includes processing trade data query requests: It receives natural language or structured query requests initiated by users and uses semantic understanding components to decompose the intent and extract concepts from the query requests.

[0037] Based on the extracted concepts, graph pattern matching is performed in the graph structure storage engine of the trade data mirror database to locate the relevant topological substructures.

[0038] Based on the bidirectional index mapping relationship, a set of detailed attribute data corresponding to the topological substructure is retrieved from the relational or key-value storage engine.

[0039] The detailed attribute data set is assembled and formatted according to business relevance to generate the final query response result.

[0040] Preferably, the step of using a semantic understanding component to perform intent decomposition and concept extraction on the query request includes: Lexical analysis is performed on the natural language or structured query requests to identify trade-related keywords and operational intent terms.

[0041] By combining the trade domain ontology, the identified keywords are mapped to standard trade concept terms, and the potential relationships between trade concept terms are determined.

[0042] Based on the intent words, infer the user's potential query goal: whether to find a specific entity, analyze event chains, or aggregate statistical information.

[0043] Output a structured query expression that includes standardized conceptual terms, relationships between concepts, and the type of query target.

[0044] Preferably, the method further includes a process for maintaining the consistency between the trade data mirror database and the source data: Continuously monitor change notifications from the heterogeneous trade data sources, or periodically compare summary information of the source data with that of the data in the mirror database.

[0045] When a substantial difference is detected in the data, the affected data range is determined, and the corresponding trade data fragment in the trade data mirror database is marked as pending synchronization.

[0046] Only for trade data fragments that are in a state of pending synchronization, the entire processing chain from data verification to updating the panoramic trade knowledge graph is retried.

[0047] After the update is completed, the new version of the Panoramic Trade Knowledge Graph will be switched to the online service, and a copy of the old version's data will be archived.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are as follows: Within the virtual storage partition, data fragments are transformed into a network structure composed of entity nodes and event relationship edges based on the trade model, explicitly modeling and solidifying the implicit business semantics at the data storage level. When performing data queries, the system can directly traverse and utilize these pre-built networked relationships, avoiding the computational overhead of complex joins and real-time relationship derivation of multiple data tables in traditional methods, thereby improving the response speed and execution efficiency for querying and retrieving complex business relationships.

[0049] By using cross-citation technology to identify and link data nodes representing the same real-world object in different partitions, global entity resolution and unification are achieved. Building upon this, link fusion technology is applied to automatically stitch together and integrate related fragments distributed across multiple partitions that belong to the same business process but are separate from each other, constructing a coherent path that transcends original data boundaries and business processes. This connects previously isolated data fragments into a graph reflecting the entire trade landscape, supporting the overall tracing and review of long-term, multi-participant business chains.

[0050] The panoramic knowledge graph generated based on the above process has its topology and entity attributes persisted to different optimized storage engines. This storage method allows the network relationships and attribute details of the data to be accessed and computed independently and efficiently. Relational queries directly manipulate the graph topology, while detailed query queries access the attribute storage, achieving targeted optimization of query load and improving the overall performance and scalability of the mirror database when facing mixed loads. Attached Figure Description

[0051] Figure 1 This is a flowchart of the trade data mirror database generation method described in this invention; Figure 2 A flowchart for configuring the data source and capturing data; Figure 3 A flowchart for domain model-driven data deconstruction and reconstruction; Figure 4 Grouped bar chart for analyzing the clarity of query intent in trade data; Figure 5 Grouped bar chart for performance analysis of heterogeneous data source access to trade data mirror database. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0053] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0054] See Figure 1 The system configures access endpoints for heterogeneous trade data sources and schedules data capture tasks according to preset trade cycle patterns, pulling raw trade record sets from these endpoints. Multi-stage verification is performed on the raw trade record sets, including integrity checks, logical inconsistency checks, and compliance screening, generating a verified intermediate trade data set. A domain-driven trade model is introduced, deconstructing the intermediate trade data set into factual data and dimensional data, and reorganizing them according to the model-defined mapping rules to generate trade data fragments with model labels. A virtual storage partition is allocated to each trade data fragment with a model label, and a data association network with trade entities at its core and trade events as its context is constructed within the virtual storage partition. The data association networks in all virtual storage partitions are aggregated, and a panoramic trade knowledge graph is generated through cross-citation and link fusion technologies. The topology and attribute data of the panoramic trade knowledge graph are persisted to different storage engines, forming the final trade data mirror database.

[0055] In one embodiment of the present invention, see [reference] Figure 2 The process of configuring access endpoints for heterogeneous trade data sources and scheduling data capture tasks according to preset trade cycle patterns is achieved by registering an endpoint configuration file containing authentication, protocol adaptation, and rate limiting parameters for each trade data source to be accessed. For example, for a trade platform that provides data through an application programming interface (API), the endpoint configuration file will record its API key, timeout period, and maximum number of requests per minute. The temporal distribution characteristics of historical trade activities are analyzed to formulate a data capture task schedule including trigger time, execution frequency, and priority. For example, based on the analysis of historical import customs declaration submission times, the task schedule can be set to execute incremental data capture twice daily at 2:00 AM and 4:00 PM Beijing time, with higher execution priorities. According to the data capture task schedule, the corresponding data capture task is activated at the specified time, establishing a connection channel with the trade data source according to the parameters in the endpoint configuration file. The activated task will use the authentication credentials and protocol adapter recorded in the configuration file to initiate a secure connection to the target trade data source. Incremental or full raw data packets are read through the connection channel and temporarily cached as an unprocessed collection of raw trade records. The raw data packets are typically transmitted in JavaScript object representation or Extensible Markup Language format and are completely stored temporarily in a specified directory of the distributed file system.

[0056] In some embodiments, the process of performing multi-stage verification on the original trade record set begins by traversing the original trade record set, checking each record for missing fields or illegal formats according to a preset list of necessary fields, marking and repairing incomplete records. The list of necessary fields mandates that each trade record must have a transaction identifier, transaction date, transacting party name, and amount fields. For records missing key fields, the system will interpolate based on previous and subsequent records or place them in a repair queue. Based on the trade business rule base, logical inconsistencies are checked on the records that have completed integrity checks, identifying and isolating conflicting transaction information records. The trade business rule base defines constraints such as "the total amount of the same batch of goods should be equal to the sum of the amounts of all its sub-items" or "the shipment date cannot be later than the arrival date." Records that violate these constraints will be marked as logical inconsistencies. An external trade compliance strategy engine is invoked to perform compliance screening on the records that have passed the logical inconsistency check, filtering out records that violate preset compliance clauses. The external trade compliance strategy engine will access the latest sanctions list and trade control regulations, compare the transacting party information with the list, and automatically block transaction records involving sanctioned entities. All records that pass the screening are integrated to form a verified set of intermediate trade data with unified coding, time alignment, and logical consistency. The integration process converts the date fields of all records to UTC timestamps and performs standardized coding on the names of all trading parties, ultimately outputting a well-structured data set.

[0057] In some embodiments, the logical contradiction-checking step of multi-stage verification can introduce a consistency verification function to quantitatively assess the degree of contradiction in records. This function mathematically evaluates the consistency of multiple related fields in a record, and its formal expression is as follows:

[0058] Where: symbol Representative Record Contradictory fractions, sign Representative applied to records The total number of business rules. (Symbol) Representing the The preset weighting factor of each business rule, symbol It is an indicator function, when the field The actual value and according to the rules and records Expected value calculated from other fields If the match is not found, the function outputs 1; otherwise, it outputs 0. Each business rule defines the target field to be checked. And a function to calculate the expected value of this field. This function is based on records The calculation is performed on other relevant fields in the original trade record set. Each record is input into this function for calculation as the original trade record set is iterated over. When a record has a contradiction score... When a preset threshold is exceeded, the system determines that the record contains a logical contradiction and automatically isolates it, initiating a manual review process. This quantitative method transforms the logical contradiction detection process from a qualitative judgment into a measurable and adjustable automated operation.

[0059] It is understandable that the authentication parameters in the endpoint configuration file may include digital certificates, tokens, or username-password combinations, while the protocol adaptation parameters define the communication protocol and data exchange format required to interact with a specific data source. It is also understandable that the external trade compliance strategy engine can be invoked via synchronous queries or asynchronous message notifications, depending on the real-time requirements of compliance screening and the system architecture design.

[0060] In one embodiment of the present invention, see [reference] Figure 3 The introduction of a domain-driven trade model involves loading a predefined domain-driven trade model, which exists in the form of a configuration file or a domain-specific language. This model explicitly defines the types of trade facts such as "purchase orders," "logistics waybills," and "customs declarations," as well as the hierarchical structure of dimensions such as "enterprise entities," "commodities," "geographical locations," and "time periods." It also specifies the relationships between facts and dimensions; for example, the fact of "purchase orders" must be associated with both the "buyer company" and "seller company" dimensions. Based on the domain-driven trade model, each record in the intermediate trade dataset is decomposed into factual data describing trade metrics and dimensional data describing the trade context. For a raw record containing order number, amount, transaction time, buyer name, and commodity code, the decomposition operation extracts "order amount" as factual data, while "buyer name," "commodity code," and "transaction time" are categorized as their respective dimensional data. Based on the mapping rules in the domain-driven trade model, the extracted factual data and corresponding dimensional data are re-associated and combined, and tagged with business type labels defined by the domain-driven trade model. The mapping rules instruct that the "order amount" fact be associated with the "buyer company" dimension obtained by querying the standardized enterprise database through "buyer name," and with the "product" dimension obtained by querying the product master data through "product code." Finally, this set of data is tagged with the business type "purchase order." Each set of associated data with business type labels is encapsulated into a structured data packet. The structured data packet adopts a unified self-describing format, such as containing factual values, associated dimension key-value pairs, and metadata headers. The structured data packet is the trade data fragment with model labels.

[0061] In some embodiments, the process of allocating virtual storage partitions and constructing a data association network for trade data fragments with model labels routes them to logically isolated virtual storage partitions based on the business type labels of the trade data fragments. Each virtual storage partition maintains an independent metadata directory. For example, all trade data fragments labeled "Logistics Waybill" are routed to virtual storage partition A, while trade data fragments labeled "Customs Declaration" are routed to virtual storage partition B. Virtual storage partitions A and B each have their own metadata directories recording node and edge patterns. Within each virtual storage partition, key trade entity objects in the trade data fragments are identified, and each entity object is created as a network node. From a "Logistics Waybill" trade data fragment, multiple entity objects such as "Carrier Company ABC", "Container Number XYZ123", and "Port of Departure Shanghai" can be identified, and network nodes with unique identifiers are created for each of these entity objects within virtual storage partition A. Specific trade events recorded in trade data fragments are extracted, and each event is treated as a directed edge connecting related entity nodes. The time, attribute, and status information of the event are appended to the directed edges. Continuing the example above, the "Container Loading" event is extracted from the same "Logistics Waybill" trade data fragment. This event creates a directed edge in the data association network of virtual storage partition A, pointing from the "Container Number XYZ123" node to the "Carrier Company ABC" node. The time attribute appended to the directed edge is the specific timestamp of the loading operation, and the status attribute is "Loaded". Based on continuously imported trade data fragments, the nodes and edges within the virtual storage partition are dynamically updated, allowing the data association network to evolve over time. When a new trade data fragment indicates that container number XYZ123 has completed customs clearance, the system will find the corresponding "Container Number XYZ123" node in virtual storage partition A and create a new "Customs Clearance Completed" event edge pointing to the "Customs Port" node.

[0062] In some embodiments, when data is reorganized based on a domain-driven trade model, the strength of the association between factual data and dimensional data can be quantitatively assessed. This assessment is achieved through an association strength function, which calculates a semantic relevance score between a given factual data instance and its associated dimensional data instance. The mathematical expression of the association strength function is as follows:

[0063] Where: symbol Representative factual data examples With dimensional data instances The correlation strength score between them. (Symbol) This represents the total number of feature pairs used to assess similarity. (Symbol) Representing the Each feature pair has a preset weight coefficient, and the sum of all weight coefficients is 1. (Symbol) It is an example from factual data. Extract the first A function that compares characteristics. Symbol It is a dimensional data instance Extract the first A function that compares characteristics. Symbol It is a function that measures the similarity between two feature values, such as cosine similarity or Jaccard similarity. During the execution of mapping rules, the system calculates the association strength score between each fact data instance and all candidate dimension data instances. The dimension data instance with the highest association strength score exceeding a preset threshold will be selected as the final association object. This quantitative evaluation mechanism makes the reorganization process of factual data and dimensional data more accurate and reduces association errors caused by fuzzy matching.

[0064] Optionally, the loading process of the domain-driven trade model supports hot updates. When the business definition changes, the new domain-driven trade model configuration file can be loaded and taken effect without downtime. Optionally, in addition to business type tags, the routing strategy for virtual storage partitions can also combine the geographic region or time range attributes of the data to perform multi-level partitioning, thereby optimizing the query and management efficiency of the data association network.

[0065] It is understandable that the encapsulation format of structured data packets needs to include version information to support compatibility handling of different versions of trade data fragments as the domain-driven trade model evolves. It is also understandable that nodes and edges in a data association network, in addition to storing identifiers and attributes, can also store reference pointers to the original trade data fragments upon which they were created, thus enabling data traceability.

[0066] In one embodiment of the present invention, the execution of the cross-citation procedure relies on an entity resolution algorithm. This algorithm identifies overlapping nodes by comparing the node attribute lists in the metadata directory of the virtual storage partition. For example, a node named "ABC Import Company" in the "Purchase Contract" partition and a node named "ABCCo.,Ltd." in the "Logistics Waybill" partition may have different names, but the algorithm can determine that they point to the same real-world trading entity by comparing core identifying attributes such as the unified social credit code and registered address. The algorithm assigns a globally unique identifier to the identified overlapping node and creates a bidirectional hyperlink edge in the graph structure storage engine. The link type can be defined as "identifying the same entity," thereby achieving a unified view of entities across virtual storage partitions. For example, after the "ABC Import Company" node and the "ABCCo.,Ltd." node are associated through a hyperlink, they will be regarded as the same entity in subsequent queries. In some embodiments, the link fusion program analyzes the semantic and timestamp attributes of trade event edges. The program searches for event edges with causal or temporal relationships. For example, in the "Purchase Contract" partition, there exists a "Contract Signing" event edge connecting the "ABC Import Company" node and the "Overseas Supplier" node. Simultaneously, in the "Payment Record" partition, there exists a "Deposit Payment" event edge connecting the "ABC Import Company" node and the same "Overseas Supplier" node, with the "Deposit Payment" event's timestamp being later than the "Contract Signing" event. The link fusion program can connect these two discrete event edges through their shared entity nodes, forming a continuous semantic path from "Contract Signing" to "Deposit Payment." This path represents the business logic segment of "Deposit Payment after Contract Signing." In specific implementations, to quantitatively evaluate the rationality of fusion between event edges, a path coherence scoring function can be introduced. This function comprehensively considers the pre- and post-event relationships of event types, the rationality of time intervals, and the matching degree of related entities. An example of the coherence scoring function is shown below:

[0067] Where: symbol Represents consistency score; symbol Representative event type With event type The semantic association strength weight value between them; symbols Represents the absolute difference between the timestamps of two events; symbol It is the time decay coefficient used for normalization; symbol Representative event With the event The overlap measure of the entity sets involved; symbol , , These are the weighting coefficients for each item. The link fusion process calculates the coherence score of candidate event edge pairs and sets a threshold, merging event edge pairs with scores higher than the threshold into continuous semantic paths. In practice, nodes and edges processed by cross-citation and link fusion are integrated into the panoramic trade knowledge graph. The integration process updates the global index of the graph, enabling networks such as "purchase contracts" and "logistics waybills," which were originally limited to a single virtual storage partition, to be connected in the panoramic trade knowledge graph through hyperlinks and fusion paths into a globally traversable overall network structure.

[0068] The process of separating topological structure data from the panoramic trade knowledge graph involves traversing all nodes and edges in the graph, extracting globally unique identifiers of nodes, starting and target node identifiers of edges, and edge type information, converting this information into a graph serialization format, such as a JSON file or binary format for an attribute graph model, and then importing it in batches into dedicated graph structure storage engines such as Neo4j or JanusGraph. The process of separating attribute data involves extracting all attribute key-value pairs from nodes and edges, excluding connection relationships, such as detailed company registration information, specific monetary terms of contracts, and event status details. This attribute data is then organized into row and column formats according to the correspondence between "entity-attribute" or "event-attribute" and imported into relational or key-value storage engines such as PostgreSQL or Cassandra. It is understandable that establishing a bidirectional index mapping relationship requires storing the primary key or location address of each node and edge object in the attribute storage engine within the graph structure storage engine. Simultaneously, each record in the attribute storage engine stores a globally unique identifier for the corresponding node or edge within the graph structure storage engine. This bidirectional index mapping relationship allows for immediate retrieval of all detailed attributes of a node in the attribute storage engine after querying the relevant node through graph traversal. Conversely, after filtering records by attribute conditions, the corresponding nodes or edge objects in the graph can be immediately located. In some embodiments, the access interface is encapsulated as a unified database service. This can be achieved by defining a set of standard graph query languages ​​(such as Gremlin or Cypher) extension interfaces and attribute query interfaces. The service layer receives query requests from the upper-layer application, decomposes them into topology queries on the graph structure storage engine and attribute queries on relational or key-value storage engines, merges the two results using the bidirectional index mapping relationship, and assembles the final result into a unified format to return to the application layer. This service layer, which encapsulates the underlying dual-storage engine collaboration logic, constitutes the external service entity of the trade data mirror database.

[0069] In one embodiment of the present invention, the process of processing trade data query requests receives natural language or structured query requests initiated by users. Users submit queries through a unified database service encapsulation. The query can be a natural language statement such as "find all companies that exported electronic products from Shanghai Port with a value exceeding one million US dollars last month," or it can be a predefined structured query object. A semantic understanding component is used to decompose the query request into intents and extract concepts. The semantic understanding component parses the input statement and identifies key elements. Based on the extracted concepts, graph pattern matching is performed in the graph structure storage engine of the trade data mirror database to locate relevant topological substructures. For example, the concept of "company" is mapped to the "enterprise entity" node type, "export" is mapped to the "departure" event edge type, and "Shanghai Port" is mapped to a specific geographical location node. The graph structure storage engine searches for subgraph structures that satisfy the pattern "enterprise entity - [departure] -> geographical location" and whose geographical location node attribute is "Shanghai Port" based on these pattern conditions. Based on the bidirectional index mapping between the graph-structured storage engine and the relational or key-value storage engine, the system retrieves the detailed attribute data set corresponding to the topological substructure from the relational or key-value storage engine. Through the bidirectional index mapping, it obtains the globally unique identifier of each node from the "Enterprise Entity" node list returned by the graph-structured storage engine, and uses these identifiers as keys to batch query the corresponding detailed attributes such as company name and registered location from the relational database. The detailed attribute data set is then assembled and formatted according to business relevance to generate the final query response. The system summarizes and sorts the retrieved attribute data, including company name, corresponding export commodity list, and total amount, and encapsulates and returns it according to the format required by the user interface or application programming interface.

[0070] In some embodiments, the semantic understanding component is used to perform intent decomposition and concept extraction on query requests. This includes lexical analysis of natural language or structured query requests to identify trade domain keywords and operational intent words. The lexical analysis segments the query statement and labels the part of speech of each word, identifying noun trade domain keywords such as "Shanghai Port," "export," "electronic products," "company," "amount," and "million US dollars," as well as operational intent words such as "find." Combined with a trade domain ontology, the identified keywords are mapped to standard trade concept terms, and the potential relationships between these terms are determined. The trade domain ontology defines "Shanghai Port" as an instance of "port," which is a subclass of "geographical location"; "electronic products" as a category under the broad category of "commodities"; "company" as the concept of "enterprise entity"; and the "export" business action as a "logistics event" with "trade direction" as "export." Based on the operational intent words, the potential target of the user's query is inferred—whether it is to find specific entities, analyze event chains, or statistically aggregate information. From the operational intent word "find," it can be inferred that the user's potential target is to find a set of entity objects that meet specific conditions, i.e., a list of "enterprise entities." Output a structured query expression. The structured query expression contains standardized conceptual terms, relationships between concepts, and the type of query target. The structured query expression may be represented by a logical expression tree or a specific query description language. Its core elements include the target type "enterprise entity", the filter condition "participated in 'logistic events' originating from 'Shanghai Port'", the "goods" associated with the event are classified as "electronic products", and the cumulative amount is greater than "US$1,000,000".

[0071] In practical implementation, the semantic understanding component's processing quality for parsing the original query can be evaluated using a query intent explicitness function. This function quantitatively analyzes the completeness of the transformation from the original query to a structured query expression. The mathematical expression of the query intent explicitness function is as follows:

[0072] Where: symbol Represents the original query request The query intent clarity score obtained after parsing; symbol Represents the original query The trade-related keywords identified can be successfully mapped to the number of standard trade concept terms in the trade-related ontology; symbols Represents the original query The total number of keywords across all trade sectors identified; symbols Represents the intention of operation. The resolved confidence factor is a value between 0 and 1, calculated by the semantic understanding component based on the clarity of intent words and contextual support. A higher query intent clarity score indicates a higher query intent clarity. This demonstrates that the semantic understanding component can clearly and completely transform user queries into machine-executable structured query expressions.

[0073] To illustrate the final format of the query response, refer to Table 1, which describes the structure of a possible assembled query response result.

[0074] Table 1: Results of the query "Find all companies that exported electronic products from Shanghai Port with a value exceeding one million US dollars last month".

[0075] The semantic understanding component supports interactive clarification. When the query intent clarity score falls below a certain threshold, the system can automatically generate clarification questions to confirm the user's query intent. Optionally, structured query expressions can be cached. When semantically similar query requests are received, cached expressions can be reused directly, improving query processing efficiency.

[0076] Trade ontology databases require regular maintenance and updates to cover new trade terms, commodity classifications, and business rules. It's understandable that when retrieving detailed attribute datasets from relational or key-value storage engines, batch querying and join optimization techniques can be used to reduce input / output operations and improve retrieval speed.

[0077] See Figure 4 This is a grouped bar chart analyzing the clarity of trade data query intent. The core of the structured query chart shows three key indicators of semantic parsing for different query types. Keyword matching rate and intent confidence are both close to 1, indicating the highest clarity, reflecting the machine-friendly nature of structured queries. Natural language-complex / speech-to-text keyword matching rate or intent confidence is lower, leading to a significant decrease in clarity, reflecting the difficulty of parsing unstructured input. Multi-condition combination queries have higher intent confidence, but limited keyword matching rate, resulting in moderate clarity. This chart is used to evaluate the semantic parsing capabilities of the trade data query system, helping technical personnel identify parsing bottlenecks for different query types, and thus optimize the accuracy and efficiency of query responses.

[0078] In one embodiment of the present invention, the process of maintaining the consistency between the trade data mirror database and the source data continuously monitors change notifications from heterogeneous trade data sources, or periodically compares the summary information of the source data with that of the data in the trade data mirror database. For source systems that support database triggers, the system subscribes to the add, delete, and modify events of their data tables as change notifications. For sources that do not provide active notifications, the system calculates the row hash or checksum of key fields in the source's key data tables at regular intervals and compares it with the previous version summary information recorded in the trade data mirror database. When a substantial difference in data is detected, the affected data range is determined, and the corresponding trade data fragments in the trade data mirror database are marked as pending synchronization. For example, when a batch of new customs declaration records is received from the customs data source, by comparing the record identifiers, the system determines that the affected data fragments are a specific set of trade data fragments of the "customs declaration" business type, and marks the status of these fragments in the metadata as "pending synchronization". For trade data fragments awaiting synchronization, the entire processing chain from data verification to updating the panoramic trade knowledge graph is retried. Based on the original data identifiers associated with the trade data fragments awaiting synchronization, the system retrieves relevant data from the corresponding heterogeneous trade data sources and re-executes integrity checks, logical contradiction checks, and compliance screenings to generate a new, verified intermediate trade data set. This set is then reorganized according to a domain-driven trade model, updating the data association network within the virtual storage partition. Finally, through cross-citation and link fusion, the panoramic trade knowledge graph is updated. After the update is complete, the new panoramic trade knowledge graph version is hot-swapped to the online service, and a copy of the old version is archived. The hot-swapping operation is achieved by redirecting access routes from the current online graph structure storage engine and attribute storage engine to the newly constructed storage instance. The old version's data copy is compressed and migrated to the archive storage area, retained for a preset period for auditing or rollback.

[0079] In some embodiments, the determination of substantial differences in the detection data can be based on a quantified synchronization decision function. This function comprehensively evaluates the scope of the change, the importance of the data, and the synchronization lag time, and its mathematical expression is as follows:

[0080] Where: symbol Represents the detected data changes Calculated synchronous decision score; symbol Represents the change coverage rate, which is the ratio of the number of trade data fragments affected by the changed data to the total number of corresponding trade data fragments in the trade data mirror database; symbol The importance factor represents a change, which is a weighted value assigned to the type of entity or transaction involved in the changed data based on predefined rules. For example, changes involving core enterprises or high-value transactions have a higher importance factor. (Symbol) This represents the time lag between when a change occurs and when it is detected. (Symbol) These are used for weighted change coverage respectively. Changes in importance factors and lag time The coefficient. When the simultaneous decision score When the preset decision threshold is exceeded, the system determines that the data difference is a substantial difference and initiates the subsequent synchronization process; otherwise, the system may only record the change without immediately triggering synchronization.

[0081] In some embodiments, the process of re-triggering the entire processing chain is incremental and selective. The system only acquires incremental data related to the trade data fragments to be synchronized, rather than the full data, and only re-executes data verification, model mapping, network construction, and graph fusion operations within a local scope related to this incremental data. Methods for determining the affected data range include analyzing the primary key list in the change notification or locating changed business record identifiers by comparing old and new summary information, and mapping these identifiers back to the corresponding set of unique identifiers for the trade data fragments within the trade data mirror database. The operation of marking a trade data fragment as pending synchronization is atomic, and the timestamp and triggering reason for the pending synchronization state are recorded in the metadata to prevent inconsistent queries on that fragment during synchronization. The process of switching to a new version of the panoramic trade knowledge graph is typically completed in a transactional operation to ensure the consistency and continuity of the query service before and after the switch. The strategy for archiving old version data copies can be managed based on time or storage space, for example, retaining only snapshots of the most recent seven versions of the panoramic trade knowledge graph.

[0082] Optionally, the continuous monitoring process can adopt a push-pull combined mode. For heterogeneous trade data sources that support message queues, a "push" mode that listens for change messages is used, while for other data sources, a "pull" mode that periodically queries is used. Optionally, the frequency of periodically comparing summary information can be dynamically adjusted according to the change frequency of the data source. For data sources that are updated frequently, the comparison period is set to be shorter, and for data sources that are updated infrequently, the comparison period can be extended accordingly.

[0083] It is understandable that query requests for trade data fragments marked as pending synchronization may be temporarily redirected to the older version of the data or return a "data synchronization in progress" message before synchronization is complete. It is also understandable that archived copies of the older data version can be used for historical data queries, change tracking analysis, or data recovery in case of synchronization problems.

[0084] See Figure 5 This is a grouped bar chart analyzing the performance of heterogeneous data source access to a trade data mirror database. It primarily displays the performance of different data sources across three dimensions: synchronization, quality, and storage. The synchronization speed of all data sources is close to 0, presumably because this metric is significantly lower than the quality / efficiency metric. The banking data source scores the highest (close to 95%), reflecting the standardization of financial data; the logistics data source scores second (approximately 88%); the enterprise data source has the highest efficiency (approximately 85%), while the banking data source has relatively lower efficiency (approximately 75%), reflecting the differences in the adaptability of different data source structures to storage compression. This chart is used to evaluate the heterogeneous data source access capabilities of the trade data mirror database, helping technical personnel optimize access strategies for different data sources and ensuring the overall performance and data quality of the mirror database.

[0085] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for generating a trade data mirror database, characterized in that, include: Configure the access endpoints for heterogeneous trade data sources and schedule data capture tasks according to preset trade cycle patterns to pull the original set of trade records from the access endpoints; The original trade record set is subjected to multi-stage verification, including integrity verification, logical contradiction investigation and compliance screening, to generate a verified intermediate trade data set; A domain-driven trade model is introduced, which deconstructs the intermediate trade data set into factual data and dimensional data, and reorganizes them according to the mapping rules defined by the model to generate trade data fragments with model labels. A virtual storage partition is allocated to each trade data fragment with a model label, and a data association network with trade entities as the core and trade events as the context is constructed within the virtual storage partition; By aggregating the data relationship networks in all virtual storage partitions, and using cross-citation and link fusion technologies, a panoramic trade knowledge graph is generated. The topology and attribute data of the panoramic trade knowledge graph are persisted to different storage engines to form the final trade data mirror database.

2. The method for generating a trade data mirror database according to claim 1, characterized in that, The configuration includes access endpoints for heterogeneous trade data sources, and scheduling data capture tasks according to preset trade cycle patterns to pull raw trade record sets from the access endpoints, including: Register an endpoint configuration file containing authentication, protocol adaptation, and rate limiting parameters for each trade data source to be connected; Analyze the temporal distribution characteristics of historical trade activities and develop a data capture task plan that includes trigger times, execution frequencies, and priorities. According to the data capture task schedule, the corresponding data capture task is activated at the specified time, and a connection channel with the trade data source is established according to the parameters of the endpoint configuration file. The connection channel reads incremental or full amounts of raw data packets and temporarily caches them as an unprocessed set of raw trade records.

3. The method for generating a trade data mirror database according to claim 2, characterized in that, The process of performing multi-stage verification on the original trade record set includes integrity checks, logical inconsistency checks, and compliance screening, generating a verified intermediate trade data set, including: Iterate through the original trade record set, check each record for missing fields or illegal formats according to the preset list of necessary fields, and mark and repair incomplete records; Based on the trade business rules database, logical contradictions are checked in the records that have completed integrity verification, and conflicting transaction information records are identified and isolated. Call an external trade compliance strategy engine to perform compliance screening on records that pass the logical contradiction check and filter out records that violate preset compliance terms; All records that pass the screening are integrated to form a unified, time-aligned, and logically consistent set of all verified intermediate trade data.

4. The method for generating a trade data mirror database according to claim 3, characterized in that, The introduced domain-driven trade model deconstructs the intermediate trade data set into factual data and dimensional data, and reorganizes them according to the mapping rules defined in the model to generate trade data fragments with model labels, including: Load a predefined domain-driven trade model, which explicitly defines the types of trade facts, the hierarchical structure of dimensions, and the relationships between facts and dimensions; Based on the domain-driven trade model, each record in the intermediate trade dataset is decomposed into a factual data part describing trade metrics and a dimensional data part describing the trade context. Based on the mapping rules in the model, the split factual data and the corresponding dimensional data are re-associated and combined, and labeled with the business type tags defined by the model. Each set of associated data tagged with a business type is encapsulated into a structured data packet, which is the trade data fragment with the model tag.

5. The method for generating a trade data mirror database according to claim 4, characterized in that, The process involves allocating a virtual storage partition for each trade data segment with a model label, and constructing a data association network within the virtual storage partition, centered on trade entities and linked by trade events. Based on the business type label of the trade data fragment, it is routed to a logically isolated virtual storage partition, with each partition maintaining an independent metadata directory; Within each virtual storage partition, key trade entity objects in the trade data fragments are identified, and each entity object is created as a network node; Extract specific trade events recorded in the trade data fragments, treat each event as a directed edge connecting the relevant entity nodes, and attach the event's time, attribute, and status information to the directed edge; Based on the continuously imported trade data fragments, the nodes and edges within the virtual storage partition are dynamically updated, allowing the data association network to evolve over time.

6. The method for generating a trade data mirror database according to claim 5, characterized in that, The data association network across all virtual storage partitions is aggregated and, through cross-citation and link fusion technologies, a panoramic trade knowledge graph is generated, including: Parallel access to the metadata directories of all virtual storage partitions to obtain the node list and edge list of the data association network within each partition; Run the cross-reference program, compare the node lists of different partitions, identify overlapping nodes that point to the same real-world trade entity, and create cross-partition hyperlinks for overlapping nodes. The link fusion process is executed to analyze trade event edges across partitions and connect discrete event edges belonging to the same trade chain or business process into a continuous semantic path. By integrating all linked nodes and merged paths, a unified panoramic trade knowledge graph that supports global queries is formed.

7. The method for generating a trade data mirror database according to claim 6, characterized in that, The step of persisting the topology and attribute data of the panoramic trade knowledge graph to different storage engines to form the final trade data mirror database includes: The topological structure data describing the connection relationship between nodes and edges is extracted from the panoramic trade knowledge graph, converted into a graph serialization format, and imported into a dedicated graph structure storage engine for storage. Detailed attribute data attached to nodes and edges are extracted from the panoramic trade knowledge graph, converted into row and column format, and imported into a high-performance relational or key-value storage engine for storage. Establish a bidirectional index mapping relationship between the graph structure storage engine and the relational or key-value storage engine to ensure that attribute data can be located through topological relationships, and vice versa; The access interface to the panoramic trade knowledge graph is encapsulated into a unified database service package, which is the trade data mirror database.

8. The method for generating a trade data mirror database according to claim 7, characterized in that, After forming the trade data mirror database, the process also includes processing trade data query requests: Receive natural language or structured query requests initiated by users, and use semantic understanding components to decompose the intent and extract concepts from the query requests; Based on the extracted concepts, graph pattern matching is performed in the graph structure storage engine of the trade data mirror database to locate the relevant topological substructures; Based on the bidirectional index mapping relationship, a set of detailed attribute data corresponding to the topological substructure is retrieved from the relational or key-value storage engine; The detailed attribute data set is assembled and formatted according to business relevance to generate the final query response result.

9. The method for generating a trade data mirror database according to claim 8, characterized in that, The process of using a semantic understanding component to decompose the intent and extract concepts from the query request includes: Lexical analysis is performed on the natural language or structured query requests to identify trade-related keywords and operational intent words. By combining the trade domain ontology, the identified keywords are mapped to standard trade concept terms, and the potential relationships between trade concept terms are determined. Based on the keywords of the user's intent, infer the user's potential query goal, whether it is to find a specific entity, analyze event chains, or statistically aggregate information; Output a structured query expression that includes standardized conceptual terms, relationships between concepts, and the type of query target.

10. The method for generating a trade data mirror database according to claim 9, characterized in that, The method also includes a process for maintaining the consistency between the trade data mirror database and the source data: Continuously monitor change notifications from the heterogeneous trade data sources, or periodically compare summary information of the source data with that of the data in the mirror database; When a substantial difference is detected in the data, the affected data range is determined, and the corresponding trade data fragment in the trade data mirror database is marked as pending synchronization. Only for trade data fragments that are in a state of pending synchronization, the entire processing chain from data verification to updating the panoramic trade knowledge graph is retried; After the update is completed, the new version of the Panoramic Trade Knowledge Graph will be switched to the online service, and a copy of the old version's data will be archived.

Images