Multi-source data synchronization method applied to multi-gateway nodes

By identifying the interface type and data source of multiple gateway nodes, establishing a data synchronization import relationship network, and performing path search and re-checking processing, the repeated collection problem in data synchronization between multiple gateway nodes is solved, and the accuracy of management goals is improved. efficiency.

CN118733680BActive Publication Date: 2024-11-19NANJING HUIPAI INTELLIGENT LOGISTICS SERVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411235293.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-19
Estimated Expiration
2044-09-04

AI Technical Summary

Technical Problem

There is a problem of repeated acquisition of the same data items in data synchronization between multiple gateway nodes, resulting in errors in management target results and inefficient management.

Method used

By obtaining the interface types of multiple gateway nodes and their connection data sources, monitoring the target identification data item information, establishing a data synchronization import relationship network, conducting path searches and checking path data, filtering out duplicate data for synchronous import.

Benefits of technology

Avoid repeated import of the same data, save storage space, improve data processing efficiency, and ensure data accuracy and consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118733680B_ABST
    Figure CN118733680B_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of multi-source data synchronization, and provides a multi-source data synchronization method applied to multi-gateway nodes. The method includes: obtaining the interface type and data source of the multi-gateway nodes; identifying the monitoring target and determining the data item information; obtaining the management target and establishing a synchronization import relationship; establishing a node synchronization relationship based on the acquisition relationship between the interface and the data item information, and connecting to form a data synchronization relationship network; performing a path search with the management target as the starting point, and extracting the path data; checking the path data for duplicates, and filtering out duplicate data for synchronous import. The present application solves the technical problem that when synchronizing multi-source data between multiple gateway nodes, the repeated collection of the same data items leads to incorrect management target results and low management efficiency. It achieves the effect of avoiding repeated collection of the same data items through path search and data duplication checking mechanisms, and improving the accuracy and efficiency of management targets such as inventory management, transportation monitoring, and production control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, specifically to the field of multi-source data synchronization technology, and in particular to a multi-source data synchronization method applied to multiple gateway nodes. Background Art

[0002] With the rapid development of Internet of Things technology, the demand for multi-gateway node data collection in logistics management is increasing. In areas such as inventory management, transportation monitoring and production control, the accuracy and timeliness of multi-source data are crucial to improving management efficiency. However, with the continuous expansion of production scale and the complexity of management systems, data synchronization between multi-gateway nodes is also facing increasing challenges. Traditional multi-gateway node data synchronization methods often have the problem of repeated collection of the same data items, resulting in incorrect management target results and low management efficiency. These problems seriously affect the accuracy and efficiency of logistics management such as inventory management, transportation monitoring and production control. Summary of the invention

[0003] The present application provides a multi-source data synchronization method applied to multiple gateway nodes, aiming to solve the technical problems of management target result errors and low management efficiency due to repeated collection of the same data items during multi-source data synchronization between multiple gateway nodes.

[0004] In view of the above problems, the present application provides a multi-source data synchronization method applied to multiple gateway nodes.

[0005] The present application provides a multi-source data synchronization method applied to multi-gateway nodes, the method comprising: obtaining the interface type of the multi-gateway node and its connection data source; performing monitoring target identification based on the connection data source and determining data item information, the data item information comprising data category, data target, and data application; obtaining the management target of the data management platform and establishing a data synchronization import relationship among the management target, data application, data target, and data category; establishing a node synchronization import relationship based on the acquisition relationship between the interface and the data item information, and forming a data synchronization import relationship network by performing relationship transfer connection based on the data synchronization import relationship and the node synchronization import relationship; performing path search based on the data synchronization import relationship network with the management target as the starting point and extracting path data; performing duplicate checking on the path data with the data target and data category as the identification target, screening out the duplicated data and performing data synchronization import.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0007] The multi-source data synchronization method applied to multiple gateway nodes first identifies the interface types on multiple network nodes and the data sources connected to these interfaces. Subsequently, monitoring is performed based on the connected data sources to identify the data items that need attention. These data items include data categories, data targets, and data application scenarios. After that, the management objectives of the data management platform are obtained, and then based on the management objectives, data applications, data targets, and data categories, a data synchronization import relationship is established to ensure that the data can be imported into the management platform according to the established logic and purpose. Then, the synchronization import relationship between nodes is constructed by using the acquisition relationship between the interface and the data item information, and then the data synchronization import relationship is combined to form a data synchronization import relationship network. In this data synchronization import relationship network, a path search is performed with the management objective as the starting point to extract the path data. These path data are the basis for subsequent data processing and synchronization. Finally, the extracted path data is checked for duplicates, and the non-duplicate data selected is synchronously imported into the management platform. This can avoid repeated import of the same data, thereby saving storage space and improving the efficiency of data processing.

[0008] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of a flow chart of a multi-source data synchronization method applied to multiple gateway nodes in one embodiment;

[0011] Figure 2 The present invention is a flowchart of establishing a node synchronization import relationship in a multi-source data synchronization method applied to multiple gateway nodes in an embodiment. DETAILED DESCRIPTION

[0012] The embodiment of the present application provides a multi-source data synchronization method applied to multiple gateway nodes to solve the technical problem that when synchronizing multi-source data between multiple gateway nodes, there is repeated collection of the same data items, which leads to incorrect management target results and low management efficiency.

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0014] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0015] Examples, such as Figure 1 As shown, the present application provides a multi-source data synchronization method applied to multiple gateway nodes, the method comprising:

[0016] Gets the interface type of a multi-gateway node and the data source to which it is connected.

[0017] In the embodiment of the present application, the multi-gateway node serves as a hub for connecting different networks and data sources, and the system terminal obtains the interface type of the multi-gateway node and the connected data source. The interface types include RS232 interface, RS485 interface, Ethernet interface, WiFi interface, and 5G interface, etc. Each interface is designed for a specific communication protocol and data transmission rate, and is used to establish a connection with a data source (sensor, device, etc.).

[0018] Further, the present application provides that the method of obtaining the interface types of multiple gateway nodes and the data sources connected thereto includes:

[0019] Perform data format analysis on multi-source data obtained from multiple data sources to obtain format features of the multi-source data; decompose and align the format features of the multi-source data to determine the format alignment relationship of the multi-source data; determine the format type according to the format alignment relationship of the multi-source data, perform data standardization conversion on each format type, and set a missing compensation value according to the alignment missing relationship to perform standardized preprocessing of the multi-source data.

[0020] Preferably, after the system terminal obtains the connected data source, multi-source data will be extracted from these data sources. The system terminal performs format analysis on the multi-source data obtained from different data sources to obtain the format characteristics of the multi-source data. This step is intended to identify the unique format characteristics of each data source, including date format, numerical representation, text encoding, etc., to provide a basis for subsequent data processing. Subsequently, according to the format characteristics obtained by the analysis, the system terminal disassembles the format of the multi-source data and aligns the formats with similar or convertible characteristics. This process determines the format alignment relationship between the multi-source data, laying the foundation for subsequent format unification and data integration. Afterwards, according to the determined format alignment relationship, the system terminal performs standardized conversion on the data of each format type. This includes unifying the date format into an international standard format, converting the numerical type into a unified precision and range, unifying the text encoding into a compatible encoding method, etc. For example, for date formats such as 01 / 04 / 2023 and January 4, 2023, which are not consistent with the international standard format "YYYY-MM-DD", the system terminal will decompose the current date format into three independent parts: year, month, and day, and splice them according to the international standard format to complete the unification of the date format. Through this step, the consistency of the multi-source data in format is ensured, which is convenient for subsequent data analysis and processing. Then, the system terminal checks whether there are missing values ​​in the data based on the alignment missing relationship, and identifies the incompleteness of the multi-source data. Then, the mean is calculated using the 5 adjacent data near the missing value, and the calculation result is set as the missing compensation value of the current missing position to fill the current missing position. Through the above data standardization conversion and missing compensation value setting, the system terminal completes the standardized preprocessing of multi-source data, ensures the consistency, completeness and accuracy of the multi-source data in format, and provides a reliable data basis for subsequent data analysis.

[0021] Based on the connected data source, monitoring target identification is performed to determine data item information, where the data item information includes data category, data target, and data application.

[0022] In one embodiment, after obtaining the connection data source, the system terminal identifies the monitoring targets of these connection data sources to clarify the specific source and purpose of the data. By identifying these connection data sources, the system terminal can determine the data item information of the multi-source data in these connection data sources, and this data item information includes data category, data target, and data application. Data category refers to the specific parameter index collected from the data source. These parameter indicators are the basic components of multi-source data and reflect the state or performance of a certain aspect of the data target. For example, in equipment monitoring, data categories include temperature, pressure, current, etc. The clarification of data categories helps the system terminal understand and process the essential content of multi-source data. Data target refers to the specific collection object targeted by the data, including sensors, equipment, etc. The data target is the starting point and end point of data collection, which determines the scope and focus of data collection. For example, in inventory management, the data target may be every shelf or cargo in the warehouse; in transportation monitoring, the data target is every transportation vehicle. Clarifying the data target helps the system terminal accurately locate the source of data collection and ensure the accuracy and integrity of the data. The data application corresponds to the management target in the platform, that is, the management or business goal to be achieved by collecting these data. Data application is the ultimate goal of data processing, which determines how the data will be used, analyzed and displayed. For example, in inventory management, data application may be to monitor inventory in real time, predict inventory demand, optimize inventory layout, etc.; in transportation monitoring, data application may be to track vehicle location, monitor transportation status, analyze transportation efficiency, etc. Clarifying data application helps system terminals build a reasonable data processing process and achieve maximum data synchronization.

[0023] Obtain the management objectives of the data management platform, and establish a data synchronization import relationship among the management objectives, data applications, data objectives, and data categories.

[0024] In one embodiment, in the data management platform, the system terminal first obtains the management objectives of the data management platform, which is the cornerstone of the entire data processing process. The management objectives define the functions and business requirements that the platform aims to achieve, such as optimizing inventory management, improving transportation efficiency, monitoring equipment operating status, etc. In order to achieve these management objectives, the system terminal establishes a correspondence between management objectives and data applications based on management objectives and data applications. A correspondence between data applications and data objectives is established based on data applications and data objectives. A correspondence between data objectives and data categories is established based on data objectives and data categories. Subsequently, the system terminal performs relationship transfer connection on the three corresponding relationships obtained to obtain a data synchronization import relationship. Through the established data synchronization import relationship, the data management platform can ensure that the multi-source data collected from the connected data sources can accurately meet the needs of the management objectives and avoid duplicate data.

[0025] Furthermore, the present application provides the acquisition of the management objectives of the data management platform, and establishes the data synchronization import relationship between the management objectives, data applications, data objectives, and data categories, including:

[0026] Perform management requirement analysis on the management target, conduct correlation analysis between management requirement and data application, and establish a correspondence between management target and data application; perform management target object analysis based on the data application, and establish a correspondence between data application and data target; perform parameter indicator analysis on the data target, and establish a correspondence between the data target and the data category using parameter indicators.

[0027] Preferably, after obtaining the management target of the data management platform, the system terminal performs management demand analysis on the management target, that is, extracts the task requirements of this management from the management target. Taking inventory management as an example, the management target may be to ensure the accuracy of inventory, optimize inventory levels, reduce inventory costs, etc. By performing demand analysis on management targets, the system terminal can identify the key information, resources and capabilities required to achieve management targets, including real-time grasp of inventory levels, accurate prediction of demand trends, optimization of inventory layout to reduce inventory costs, etc. In order to achieve inventory management targets, the system terminal defines a series of data applications. These data applications directly correspond to the specific requirements of management targets. For example, the management application of inventory product categories and inventory consumption information can help understand the composition and consumption of inventory; while the management application of inventory collection data and demand quantity can help predict inventory demand, adjust procurement plans, etc. After determining the data application, the system terminal performs a correlation analysis to determine which data applications can most directly and effectively meet the management requirements. In this step, the system terminal evaluates the functions, performance, cost and other factors of each data application in combination with historical data. Through comparative analysis, data applications closely related to management targets are screened out. Then, based on the results of the correlation analysis, the corresponding relationship between management targets and data applications is established.

[0028] After obtaining the correspondence between management objectives and data applications, the system terminal determines the data objectives that need to be collected based on the needs of data applications. Data objectives are the source of actual collected data, and they directly reflect the specific objects and scenarios of inventory management. For example, in order to grasp the dynamic changes of inventory products, it is necessary to collect transportation monitoring data to understand the transportation status and arrival time of products; in order to monitor inventory consumption, it is necessary to collect the usage status data of consumables to analyze the consumption speed and remaining amount of consumables. The system terminal establishes the correspondence between data applications and data objectives by matching these data that need to be collected with the data applications.

[0029] After obtaining the correspondence between the data application and the data target, the system terminal parses these data targets, that is, extracts the required parameter indicators from the data targets as data types. For example, in transportation monitoring data, parameter categories include transportation location, transportation time, etc.; in warehoused product category data, parameter categories may include product name, specification model, quantity, etc. These data indicators provide rich materials and basis for subsequent data processing and analysis. The system terminal establishes a correspondence between the data target and the data category by corresponding each data target with the extracted data type.

[0030] The corresponding relationship between the management target and the data application, the corresponding relationship between the data application and the data target, and the corresponding relationship between the data target and the data category are connected by relationship transmission to obtain the data synchronization import relationship.

[0031] Preferably, after obtaining the correspondence between the management target and the data application, the correspondence between the data application and the data target, and the correspondence between the data target and the data category, the system terminal uses the management target in the correspondence between the management target and the data application as the starting point, and the data category in the correspondence between the data target and the data category as the end point, starting from the data application corresponding to the management target, matching the data application in the correspondence between the data application and the data target, and directly connecting the data target corresponding to the matched data application with the corresponding data application in the correspondence between the management target and the data application to obtain the primary data synchronization import relationship. Subsequently, for each data target, its corresponding data category is determined from the correspondence between the data target and the data category, and these data categories are connected to the primary data synchronization import relationship using the same method as described above to obtain the data synchronization import relationship.

[0032] Furthermore, the present application provides for filtering the imported data of the interface, including:

[0033] Obtain cross-management data of multiple management targets; perform demand difference and demand same-direction analysis on the cross-management data to determine constraints introduced by data synchronization.

[0034] Optionally, when multiple management targets are obtained from the data management platform, there are often intersections between these management targets, that is, some data are necessary to achieve multiple targets. In order to use these shared resources more effectively, the system terminal extracts these data to obtain cross-management data. Subsequently, the obtained cross-management data is analyzed. This analysis process includes two parts, namely, demand difference analysis and demand same-direction analysis. In demand difference analysis, the system terminal identifies the differences in data requirements between different management targets. Since each management target has its own unique perspective and focus, their data requirements will also be different. Through difference analysis, it can be clarified which data is specific to a certain target, which is common to multiple targets, and how these differences affect data collection, processing and utilization. In demand same-direction analysis, the system terminal analyzes the common points or similarities of data requirements of different management targets. By comparing the data requirements of different management targets, the intersections between them are found. In the process of finding common points, the system terminal determines the key data that are critical to multiple management targets. These key data will become the focus of attention when data synchronization is introduced. For the difference points identified in the demand difference analysis, the system terminal formulates corresponding difference data synchronization constraints. For example, for management objectives that require real-time monitoring of inventory changes, real-time constraints on data synchronization are set to ensure that inventory data is formatted, missing, and other anomalies. For key data identified in the same-direction demand analysis, the system terminal formulates unified same-direction data synchronization constraints. For example, for user information data required by all management objectives, accuracy and consistency constraints on data synchronization are set to ensure that customer data remains consistent across different systems. Afterwards, the system terminal summarizes the obtained difference data synchronization constraints and same-direction data synchronization constraints to form data synchronization constraints. By setting these constraints, the orderly progress of the data synchronization process can be ensured, and the data needs of various management objectives can be met to the greatest extent possible.

[0035] According to the constraint conditions introduced by data synchronization, synchronization strategies are set for demand differences and demand common directions respectively; according to the synchronization strategies, collection configuration instructions and port filtering instructions are set, the collection configuration instructions are used to control the collection settings of the data source, and the port filtering instructions are used to filter the imported data of the interface.

[0036] Optionally, in the process of data synchronization, in order to ensure the accuracy and efficiency of data, the system terminal generates a strategy based on the obtained data synchronization introduction constraint conditions. Based on the difference data synchronization constraint conditions in the data synchronization introduction constraint conditions, determine the difference synchronization strategy that can identify and process these differences, including data conversion, data completion, data verification, etc. The goal is to enable the data to be automatically adjusted during the synchronization process to reduce problems caused by data differences. Based on the same-direction data synchronization constraint conditions in the data synchronization introduction constraint conditions, the system terminal sets a same-direction synchronization strategy that ensures the timeliness and accuracy of data during synchronization under specific conditions, including setting triggers, timed tasks, incremental synchronization, etc. The goal is to reduce unnecessary data transmission, improve synchronization efficiency, and ensure data consistency. Subsequently, the system terminal integrates the obtained difference synchronization strategy and the same-direction synchronization strategy to form a synchronization strategy. Then, according to this synchronization strategy, set the acquisition configuration instruction and the port screening instruction. Among them, the acquisition configuration instruction is used to control the specific settings for obtaining data from the connected data source, including the type of data source, connection information, acquisition frequency, and data range. Through these settings, the system terminal can accurately capture the required data from the connected data source. The port filter command is used to define the conditions for filtering the imported data before synchronizing multi-source data, including filtering based on data fields, filtering based on data values, and checking based on data formats. These filtering conditions help improve the accuracy and reliability of data.

[0037] Based on the acquisition relationship between the interface and the data item information, a node synchronization import relationship is established, and a data synchronization import relationship network is formed by performing relationship transmission connection according to the data synchronization import relationship and the node synchronization import relationship.

[0038] In one embodiment, the system terminal determines the node-data item relationship based on the acquisition relationship between the interface and the data item information, and sets a distributed lock for each data item information in combination with the distributed lock mechanism to obtain the node synchronization import relationship. Subsequently, the system terminal uses the data item information as the transmission node, and based on these transmission nodes, imports the interface-data source acquisition relationship into the data synchronization import relationship to form a data synchronization import relationship network. This data synchronization import relationship network not only includes the nodes corresponding to each data item, but also includes the complex relationship between the nodes, making the flow of data between different data sources more efficient and orderly. Through the relationship network, the source, destination and dependency relationship of multi-source data can be clearly understood, providing strong support for the further processing and analysis of multi-source data.

[0039] Further, if Figure 2 As shown, the present application provides a method for establishing a node synchronization import relationship based on the acquisition relationship between the interface and the data item information, including:

[0040] A timestamp is extracted for the multi-source data collected through the interface, where the timestamp is a time identifier attached to each data source when it is generated, and the node-data item relationship is determined based on the collection correspondence between the interface and the data item information; a distributed lock mechanism is established, and a distributed lock is set for each data item information based on the distributed lock mechanism, and the distributed lock is used to set the data import requirements of each node according to the timestamp, wherein the data update and import can only be performed once for the same data item at the same time; the distributed lock mechanism is fitted into the node-data item relationship to establish the node synchronous import relationship.

[0041] Preferably, when processing multi-source data collected through a specific interface, in order to ensure the consistency and accuracy of the multi-source data, especially in a complex environment of multi-node synchronization, the system terminal extracts timestamps from each connected data source, which are automatically attached to each data item information when it is generated, as a clear identification of the data generation time. Subsequently, according to the acquisition correspondence between the interface and the data item information, the system terminal determines the association between different nodes and data items, that is, the node-data item relationship. This step is based on the correspondence between the interface and the node, and is the basis for subsequent data synchronization and processing, ensuring the traceability and time sequence of the data. Afterwards, in order to cope with the concurrent update problem that may occur in a multi-node environment, the system terminal introduces a distributed coordination service to establish a distributed lock mechanism. The system terminal uses the established distributed lock mechanism to assign a distributed lock to each data item according to the timestamp, and this distributed lock can set the data import requirements of each node according to the timestamp configured by the distributed lock mechanism. This distributed lock mechanism can ensure that at the same time, for the same data item, only one node can perform data update import. In this way, even in the presence of network delays, failures or asynchronous clocks between nodes, data can be effectively avoided from being repeated or incorrectly overwritten, thereby maintaining the consistency and integrity of the data. Then, the system terminal combines the distributed lock mechanism with the determined node-data item relationship to establish a node synchronization import relationship. This step is the key to achieving efficient and orderly synchronization of data between multiple nodes. Through the control of distributed locks, each node can work together to maintain the accuracy and consistency of data under the premise of following certain rules.

[0042] Furthermore, the present application provides a method for establishing a data synchronization import relationship network by performing relationship transfer connection based on the data synchronization import relationship and the node synchronization import relationship, including:

[0043] Based on the node-data item relationship, the data item information is used as a transmission node; according to the connection relationship of the transmission node in the node synchronization import relationship, the interface-data source collection relationship is added to the data synchronization import relationship to construct the data synchronization import relationship network.

[0044] Optionally, the system terminal uses the data item information as a transmission node according to the determined node-data item relationship. These transmission nodes are not only the carriers of data, but also play the role of transmitting information during the synchronization process, and these transmission nodes correspond one-to-one to the nodes in the node-data item relationship. Subsequently, the system terminal uses the connection relationship of these transmission nodes in the node synchronization import relationship, that is, the connection relationship of the nodes corresponding to these transmission nodes in the node synchronization import relationship, including dependency, parent-child, association, etc., to guide the specific path and direction of multi-source data synchronization. This connection relationship ensures that multi-source data can flow between different nodes according to the established logical order and rules. In order to enhance the integrity and accuracy of data synchronization, the system terminal adds the interface-data source acquisition relationship to the data synchronization import relationship according to the corresponding relationship between the interface and the node, and constructs a data synchronization import relationship network. Through this data synchronization import relationship network, the system terminal can not only understand the information of the data item itself, but also understand how these multi-source data are collected and introduced from the connected data source through a specific interface. The addition of this acquisition relationship makes the entire data synchronization process more transparent and traceable.

[0045] According to the data, the relationship network is synchronously imported, a path search is performed with the management target as the starting point, and the path data is extracted.

[0046] In one embodiment, after the data synchronization import relationship network is constructed, the system terminal defines the management target as the starting point in the data synchronization import relationship network, initializes the search queue, and adds the starting point to it. Subsequently, the node is taken out from the search queue, and all neighbor nodes of the node are traversed according to the data synchronization import relationship network. For each neighbor node, check whether the data item of the neighbor node is the data item required by the management target. If it is the required data item, the system terminal records the current path, that is, the path from the current node to the neighbor node. After that, the neighbor nodes that meet the conditions are added to the search queue and the search process continues. When the search process ends, that is, all target nodes are found, the system terminal extracts and records all path data that meet the conditions.

[0047] The path data is checked for duplicates using data targets and data categories as identification targets, and duplicate data is screened out for data synchronous import.

[0048] In one embodiment, after extracting the path data in the data synchronization import relationship network, in order to ensure the consistency of multi-source data and avoid duplicate import, the system terminal performs duplicate checking on these path data. This process is based on data targets, data categories and hash functions. By introducing hash functions for duplicate checking, the system terminal can effectively avoid introducing duplicate data during the data synchronization import process and ensure the accuracy and consistency of the data.

[0049] Furthermore, the present application provides a method for checking the path data for duplication by using data targets and data categories as identification targets, including:

[0050] The path data is extracted with the data target as the primary feature to obtain a first data decision set; the first data decision set is extracted with the data category as the secondary feature to obtain a second data decision set; multi-source data is acquired according to the second data decision set, and hash values ​​are calculated for the multi-source data using a hash function; and duplication checking is performed on the multi-source data according to the hash values.

[0051] Optionally, the system terminal uses the data target as a primary feature to analyze and extract the path data. In this process, the system terminal extracts relevant information from the path data according to the data target pointed to by each path to form a first data decision set. This set contains data items classified based on the data target, which provides a basis for subsequent data processing. Subsequently, based on the first data decision set, the system terminal uses the data category as a secondary feature to classify the data in the first data decision set, and organizes the data that meets the secondary feature into a second data decision set. This second data decision set describes the characteristics of the data and provides guidance for subsequent data processing and synchronization. After that, the system terminal obtains multi-source data from the path data stored in the second data decision set, and calculates the hash value of each piece of data using a hash function. The hash function can compress input data of any length into a hash value of a fixed length, and for different input data, its output hash value has a high uniqueness. In this way, the system terminal can determine whether two pieces of data are the same by comparing the hash values. Then, the multi-source data is checked for duplicates according to the calculated hash value, that is, the hash value of each piece of data is compared with the stored hash value. If the same hash value is found, it means that the two copies of data are highly similar and need to be deduplicated; if the hash values ​​are different, it means that the two copies of data are different and can be retained for subsequent data synchronization import. Through such duplication checking, it can be ensured that duplicate data will not be introduced during the data synchronization import process, thereby improving the accuracy and consistency of the data.

[0052] Further, the present application provides that after the duplicate-checking data is screened out and data is synchronously imported, the following steps are included:

[0053] A multi-source synchronization data set is extracted based on the management target; and according to the corresponding relationship between the multi-source synchronization data set and the management target, the data set is synchronized to the management target module for management calculation.

[0054] Preferably, after completing the duplicate checking of multi-source data, the system terminal extracts the multi-source data after the duplicate checking to obtain a multi-source synchronization data set. Subsequently, according to the correspondence between the multi-source data in the multi-source synchronization data set and the management target, the multi-source synchronization data set and the management target are synchronized to the management target module. After receiving the multi-source synchronization data set and the management target, the management target module will perform management operations on the multi-source synchronization data set and the management target, that is, extract the management target and the corresponding multi-source synchronization data in the multi-source synchronization data set, and use the management target as the key name and the corresponding multi-source synchronization data as the key value to store in the storage space of the management target module.

[0055] In summary, the embodiments of the present application have at least the following technical effects:

[0056] The embodiment of the present application first identifies the interface type of the multi-gateway node and the data source connected to it, and clarifies the starting point of data synchronization. Subsequently, the monitoring target is identified based on the connected data source, and the key data item information, including data category, data target and data application, is determined to lay the foundation for the establishment of data synchronization relationship. Afterwards, according to the management objectives of the management platform, the management requirements are parsed, and the correlation analysis is performed with the data application, data target and data category to build a data synchronization import relationship network. At the same time, the acquisition relationship between the interface and the data item information is combined with the distributed lock mechanism to ensure the consistency and accuracy of the data during the synchronization process, and avoid data conflicts and duplications. Before data synchronization, the multi-source data is processed, including format analysis, disassembly alignment, standardization conversion and missing compensation steps to improve the quality and availability of the data. For the cross-management data of multiple management targets, the synchronization strategy is set through the demand difference and demand same direction analysis, and the acquisition configuration and port screening instructions are optimized to improve the efficiency and accuracy of data synchronization. Then, the multi-source synchronization data set extracted based on the management target will be synchronized to the management target module for management operation. These technical effects jointly solve the technical problem of repeated collection of the same data items, which leads to incorrect management target results and low management efficiency when multi-source data is synchronized between multiple gateway nodes. It achieves the effect of avoiding repeated collection of the same data items through path search and data duplication detection mechanism, thereby improving the accuracy and efficiency of management targets such as inventory management, transportation monitoring, and production control.

[0057] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0059] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A multi-source data synchronization method applied to multiple gateway nodes, characterized in that: The multi-source data synchronization method applied to multiple gateway nodes includes: Get the interface type of multiple gateway nodes and their connected data sources; Perform monitoring target identification based on the connected data source and determine data item information, wherein the data item information includes data category, data target, and data application; Obtain the management objectives of the data management platform, and establish a data synchronization import relationship among the management objectives, data applications, data objectives, and data categories; Based on the acquisition relationship between the interface and the data item information, a node synchronization import relationship is established, and a data synchronization import relationship network is formed by performing relationship transfer connection based on the data synchronization import relationship and the node synchronization import relationship; Synchronously import the relationship network according to the data, perform path search with the management target as the starting point, and extract path data; The path data is checked for duplicates using data targets and data categories as identification targets, and duplicated data is screened out for data synchronization import; The step of obtaining the management target of the data management platform and establishing the data synchronization import relationship among the management target, data application, data target, and data category includes: Perform management demand analysis on the management objectives, conduct correlation analysis between management demands and data applications, and establish a corresponding relationship between management objectives and data applications; Performing data target analysis according to the data requirements of the data application and establishing a corresponding relationship between the data application and the data target; Performing parameter index analysis on the data target, and using the parameter index to establish a corresponding relationship between the data target and the data category; The corresponding relationship between the management target and the data application, the corresponding relationship between the data application and the data target, and the corresponding relationship between the data target and the data category are connected by relationship transmission to obtain the data synchronization import relationship.

2. The multi-source data synchronization method applied to multiple gateway nodes according to claim 1, characterized in that: Based on the acquisition relationship between the interface and the data item information, a node synchronization import relationship is established, including: Extracting a timestamp from the multi-source data collected through the interface, the timestamp being a time identifier attached to each data source when it is generated, and determining a node-data item relationship based on a corresponding relationship between the interface and the collection of the data item information; Establish a distributed lock mechanism, and set a distributed lock for each data item information based on the distributed lock mechanism. The distributed lock is used to set the data import requirements of each node according to the timestamp, where only one data update import can be performed on the same data item at the same time; The distributed lock mechanism is fitted into the node-data item relationship to establish the node synchronization import relationship.

3. The multi-source data synchronization method applied to multiple gateway nodes as claimed in claim 2, characterized in that: The data synchronization import relationship and the node synchronization import relationship are connected through relationship transmission to form a data synchronization import relationship network, including: Based on the node-data item relationship, the data item information is used as a transfer node; According to the connection relationship of the transfer node in the node synchronous import relationship, the interface-data source collection relationship is added to the data synchronous import relationship to construct the data synchronous import relationship network.

4. The multi-source data synchronization method applied to multiple gateway nodes as claimed in claim 2, characterized in that: The path data is checked for duplicates with data targets and data categories as identification targets, including: Extracting data from the path data using the data target as a primary feature to obtain a first data decision set; Extracting data from the first data decision set using the data category as a secondary feature to obtain a second data decision set; Acquire multi-source data according to the second data decision set, and calculate hash values ​​of the multi-source data using a hash function; The multi-source data is checked for duplication according to the hash value.

5. The multi-source data synchronization method applied to multiple gateway nodes according to claim 1, characterized in that: The step of obtaining the interface types of the multiple gateway nodes and the data sources connected thereto further includes: Perform data format analysis on multi-source data obtained from multiple data sources to obtain format features of the multi-source data; Decomposing and aligning the format features of the multi-source data to determine the format alignment relationship of the multi-source data; The format type is determined according to the format alignment relationship of the multi-source data, data standardization conversion is performed on each format type respectively, and a missing compensation value is set according to the alignment missing relationship to perform standard preprocessing of the multi-source data.

6. The multi-source data synchronization method applied to multiple gateway nodes according to claim 1, characterized in that: The multi-source data synchronization method applied to multiple gateway nodes also includes: Obtain cross-management data for multiple management objectives; Performing demand difference and demand sameness analysis on the cross-management data to determine the constraints for data synchronization introduction; According to the constraint conditions introduced by data synchronization, synchronization strategies are set for demand differences and demand sameness respectively; According to the synchronization strategy, a collection configuration instruction and a port screening instruction are set, wherein the collection configuration instruction is used to control the collection settings of the data source, and the port screening instruction is used to screen the imported data of the interface.

7. The multi-source data synchronization method applied to multiple gateway nodes according to claim 1, characterized in that: After the duplicate checking data is screened out and the data is imported synchronously, the following is also included: Extracting a multi-source synchronization data set based on the management target; According to the correspondence between the multi-source synchronization data set and the management target, the data set is synchronized to the management target module for management calculation.