Data synchronization method and computing device

Abstract

The invention relates to the field of computing, in particular to a data synchronization method and computing equipment, applied to the computing equipment, the computing equipment comprises at least one drive program, the method comprises: receiving a synchronization request, the synchronization request comprising a data source type of a to-be-synchronized data source and a to-be-synchronized field of the to-be-synchronized data source; determining a target driving program matched with the to-be-synchronized data source from the at least one driving program according to the data source type of the to-be-synchronized data source; based on the target drive program, acquiring data of the to-be-synchronized field, and based on a preset data processing rule, processing the data of the to-be-synchronized field to obtain target data; and synchronizing the target data to target equipment.

Description

Technical Field

[0001] This application relates to the field of computing, and in particular to a data synchronization method and computing device. Background Technology

[0002] Currently, some application management software / platforms / systems can provide high-performance data query services, supporting users to query and analyze large amounts of business data without requiring modeling, aggregation, or other processing. This greatly improves the efficiency of users in querying and managing business data. The business data required by application management software / platforms / systems generally originates from source systems, and the data stored in these source systems can also be called data sources.

[0003] When application management software / platforms / systems synchronize business data from data sources, they generally rely on ETL (Extract-Transform-Load) tools. That is, ETL tools can extract business data from the data source and synchronize the extracted business data to the application management software / platform / system, allowing the application management software / platform / system to perform data processing, data visualization, graphing, and other processing.

[0004] However, this method of data synchronization using traditional ETL tools can only synchronize one fixed data source during the data source synchronization process, resulting in limited application scenarios and low efficiency. Summary of the Invention

[0005] This application provides a data synchronization method and computing device, which realizes dynamic access to data sources and data query through dynamic access to data sources and automatic data synchronization, expands data synchronization scenarios, and improves data synchronization efficiency.

[0006] According to a first aspect of the embodiments of this application, a data synchronization method is provided, applied to a computing device, the computing device including at least one driver, the method comprising: Receive synchronization requests, which include the data source type and the fields to be synchronized in the data source to be synchronized.

[0007] Based on the data source type of the data source to be synchronized, determine the target driver that matches the data source to be synchronized from at least one driver.

[0008] Based on the target driver, obtain the data of the fields to be synchronized; Based on preset data processing rules, the data in the fields to be synchronized is processed to obtain the target data. The target data is then synchronized to the target terminal.

[0009] In this embodiment, at least one driver is first configured in the computing device. This driver can be used to access and query data from a data source. A synchronization request is received, which may include the data source type and the fields to be synchronized. The data source type is related to the driver of the data source. Therefore, a target driver matching the data source to be synchronized can be determined from at least one driver. Based on the target driver, the data source to be synchronized is connected, and the data of the fields to be synchronized is read. This achieves automated matching and rapid connection for data source access, thereby improving the standardization, maintainability, and scalability of data source access, reducing the difficulty of data source access, and improving access efficiency. Furthermore, by setting preset data processing rules, the data of the fields to be synchronized is processed in a regularized manner, making the target data more standardized and normalized, providing higher quality data to the target device, and improving data validity. In addition, regardless of whether there is one or more data sources to be synchronized, or whether it is a single type of data source or multiple types of data sources, data querying and synchronization can be completed through the automatic adaptation of the target driver for each data source, solving the limitations of traditional data synchronization methods and making data synchronization highly scalable and efficient.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, data of the field to be synchronized is obtained based on the target driver, including: Determine the configuration parameters corresponding to the data source type of the data source to be synchronized; Based on the configuration parameters, the target driver is run to obtain the data of the fields to be synchronized.

[0011] In this embodiment, configuration parameters corresponding to the data source type to be synchronized are determined, and these parameters are input into the target driver program, which then runs. This automates the driver program's operation, avoiding manual parameter configuration and ensuring efficient operation based on the adapted parameters, thus improving the stability and success rate of the data source connection. Associating the data source type with configuration parameters enables automatic parameter configuration, facilitating unified management and maintenance of connection rules for different data sources, and further enhancing the standardization and scalability of the multi-data source access process.

[0012] In conjunction with the first aspect, in certain implementations of the first aspect, configuration parameters corresponding to the data source type of the data source to be synchronized are determined, including: Based on a pre-set mapping table, query the configuration parameters corresponding to the data source type of the data source to be synchronized. The mapping table includes at least one set of mapping relationships, which include the data source type and the configuration parameters corresponding to the data source type.

[0013] In this embodiment, by pre-constructing a mapping table containing the mapping relationship between data source types and corresponding configuration parameters, configuration parameters matching the data source type can be directly queried when accessing the data source. This achieves fast and accurate matching and automated acquisition of configuration parameters, improving the efficiency and accuracy of parameter acquisition. The mapping table unifies and makes transparent the configuration rules for different data source types. When adding new data source types or adjusting configuration parameters, only the mapping table needs to be maintained, without modifying the core business logic. This significantly enhances the flexibility and scalability of configuration management, providing crucial support for efficient and stable data source connections.

[0014] In conjunction with the first aspect, some implementations of the first aspect also include: Receive a registration request, which includes driver information for unregistered data source types; In response to the registration request, the driver corresponding to the driver information is registered and added to the driver repository.

[0015] In this embodiment, a registration request can be received, which may contain driver information corresponding to an unregistered data source type. The driver is then registered to the driver repository in response to the request. This enables dynamic expansion of the driver and efficient management of the driver repository, allowing for rapid adaptation to the access requirements of new data types and significantly improving the scalability and adaptability of the entire data source access system.

[0016] In conjunction with the first aspect, some implementations of the first aspect also include: Read the current metadata of the data source to be synchronized; The current metadata of the data source to be synchronized is compared with the historical metadata to obtain the comparison result, which indicates whether the metadata has changed or not. If the comparison result indicates that the metadata has changed, delete the historical metadata and save the current metadata.

[0017] In this embodiment, the current metadata of the connected data source can be read and compared with historical data to determine whether the metadata of the connected data source has changed. When a metadata change is confirmed, the metadata is updated, achieving real-time awareness and dynamic updating of the metadata of the connected data source. This ensures that the stored metadata always remains consistent with the actual state of the data source, providing an accurate data foundation for data source-related operations.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, data processing rules include: data governance rules; Based on preset data processing rules, the data in the fields to be synchronized is processed to obtain the target data, including: Based on preset data governance rules, data governance is performed on the data of the fields to be synchronized to obtain the target data.

[0019] Specifically, based on preset data governance rules, the data of the fields to be synchronized is filtered and validated to obtain the target data.

[0020] In this embodiment of the application, it is specified that data processing rules may include data governance rules. By using data governance rules to govern the data stream and obtain target data that conforms to the data governance rules, dirty data that does not conform to the data governance rules can be prevented from flowing into subsequent processes, thereby improving the overall data synchronization efficiency and quality and providing reliable data support for subsequent data.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, based on preset data governance rules, data governance is performed on the data of the field to be synchronized to obtain the target data, including: Filter out erroneous data and data that does not conform to preset validation rules from the data in the fields to be synchronized, and obtain the filtered data; Data that does not conform to the preset standard format in the filtered data is converted into data that conforms to the standard format to obtain the target data.

[0022] In this embodiment, data governance rules are used to filter out erroneous data and data that do not conform to preset verification rules in the data of the field to be synchronized. Data that does not conform to the preset standard format in the filtered data is then converted to obtain target data that conforms to the standard format. By filtering invalid data and verifying data, data quality is guaranteed from the source, redundant and abnormal data are eliminated, and more reliable and secure data is provided for subsequent application management systems.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the data processing rules include: data governance rule semantic mapping rules, which, based on preset data processing rules, integrate the data streams of the fields to be synchronized to obtain the integrated target data stream, including: Based on preset semantic mapping rules, semantic mapping processing is performed on the data of the fields to be synchronized to obtain the target data.

[0024] In this embodiment, semantic mapping rules are used to perform semantic analysis on the data of the fields to be synchronized to obtain the target data. By using semantic mapping, the semantic rules of the data are unified, which ensures the accuracy and compliance of the target data stream, and also achieves semantic standardization and unification. This provides high-quality data for the application management system and improves the usability and value of the data.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, based on preset semantic mapping rules, semantic mapping processing is performed on the data of the field to be synchronized to obtain the target data, including: The different names of the fields to be synchronized in different data sources are uniformly mapped to standard semantic fields, and the values ​​of the standard semantic fields are obtained. The target data includes the values ​​of the standard semantic fields. And / or, map different values ​​of the same field to be synchronized in different data sources to standard values ​​to obtain the standard values ​​of the field to be synchronized, and the target data includes the standard values ​​of the field to be synchronized; And / or, if the field to be synchronized is updated in the data source to be synchronized, the value of the field to be synchronized is updated locally to obtain the updated value of the field to be synchronized. The target data includes the updated value of the field to be synchronized. And / or, if the field to be synchronized involves multiple calculations of the same data object, then based on the data related to the data object in the data of the field to be synchronized, perform a calculation of the target value of the data object once to obtain the calculation result of the data object. The target data includes the calculation result of the data object.

[0026] In this embodiment, the semantics of fields from different data sources are unified, eliminating naming differences. Mapping the same field value to a standard value across different data sources resolves value conflicts. When a field to be synchronized is updated in the data source, its value is updated locally, enabling traceable updates and ensuring end-to-end data consistency during data changes. Furthermore, when the same data object involves multiple calculations, only one calculation is performed, reducing computational burden. Through these semantic mapping rules, deep alignment and business consistency of data semantics can be achieved, ensuring data quality and availability while maintaining unified data meaning. This provides application management systems with a high-quality, highly consistent, and highly business-adaptable target data stream, significantly reducing data integration complexity and improving data application efficiency.

[0027] According to a second aspect of the embodiments of this application, a data synchronization apparatus is provided, the apparatus comprising: The request receiving unit is used to receive synchronization requests, which include the data source type of the data source to be synchronized and the fields of the data source to be synchronized.

[0028] The driver matching unit is used to determine, from at least one driver, a target driver that matches the data source to be synchronized, based on the data source type of the data source to be synchronized.

[0029] The data acquisition unit is used to acquire data of the fields to be synchronized based on the target driver.

[0030] The data processing unit is used to process the data of the fields to be synchronized based on preset data processing rules to obtain the target data; The data synchronization unit is used to synchronize target data to the target device.

[0031] According to a third aspect of the embodiments of this application, a computing device is provided, including: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement any of the above data synchronization methods.

[0032] According to a fourth aspect of the embodiments of this application, a communication device is provided, including a transceiver and a processor, wherein the transceiver is used to receive or send data, and the processor is used to execute any data synchronization method of the embodiments of this application.

[0033] According to a fifth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements any data synchronization method.

[0034] According to a sixth aspect of the embodiments of this application, a computer product is provided, comprising: a computer program that, when executed by a processor, implements the steps of any data synchronization method.

[0035] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0036] The above and other objects, features, and advantages of the embodiments of this application will become more apparent from the more detailed description of the embodiments in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the embodiments of this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0037] Figure 1 The figure shows an application example of a data source management system according to an embodiment of this application; Figure 2 The figure shows an application example of a data synchronization method according to an embodiment of this application; Figure 3 The figure shows a flowchart of a data synchronization method according to an embodiment of this application; Figure 4 The illustration shows an example of dynamic management of a data source according to an embodiment of this application; Figure 5 The figure shows yet another flowchart of a data synchronization method according to an embodiment of this application; Figure 6 The figure shows a schematic diagram of the structure of a data processing rule according to an embodiment of this application; Figure 7 The figure shows another application example of a data synchronization method according to an embodiment of this application; Figure 8 The figure shows a schematic diagram of a data synchronization device according to an embodiment of this application; Figure 9 The figure shows a hardware block diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application more apparent, exemplary embodiments according to the embodiments of this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the embodiments of this application, and not all embodiments of the embodiments of this application. It should be understood that the embodiments of this application are not limited to the exemplary embodiments described herein.

[0039] The technical solution of this application embodiment can be applied to the computing field. By pre-configuring a driver repository, one or more drivers can be centrally managed, improving driver management efficiency and providing strong scalability. Based on the driver repository, automatic data source access is achieved through the compatibility between the data source type and the driver, improving data source access efficiency and stability. Subsequently, through dynamic data source access and automatic data synchronization, dynamic data source access and data query are realized, expanding data synchronization scenarios and improving data synchronization efficiency.

[0040] The technical solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0041] like Figure 1 The diagram shown is an application example of a data source management system provided in an embodiment of this application. The time processing system may include one or more data sources 10, a computing device 20, and an application management system 30.

[0042] It is understandable that data source 10 can refer to a database or system that stores data, such as databases like Kafka or Oracle, data sources corresponding to IoT (Internet of Things) streaming data, or data sources corresponding to API (Application Programming Interface) interfaces.

[0043] Application Management System 30 can refer to an in-memory computing platform that provides high-performance data query capabilities and can be deployed on a computing platform. Users, generally referring to enterprise users, can use Application Management System 30 to perform real-time queries and analysis of large amounts of business data, possessing powerful data processing capabilities.

[0044] The computing device 20 can act as a relay station between one or more data sources 10 and the application management system 30, synchronizing data from one or more data sources 10 to the application management system 30.

[0045] In this embodiment, a driver repository is configured in the computing device. The driver repository includes at least one driver program, which centrally manages at least one driver program. Different drivers can be used to drive different types of data sources. The computing device 20 receives a synchronization request, which includes the data source type and the field to be synchronized. By utilizing the association between the data source type and the driver program, the target driver program for the data source to be synchronized is determined from at least one driver program. The target driver program then reads the data from the field to be synchronized. This improves the standardization, maintainability, and scalability of data source access, reduces the difficulty of data source access, and increases access efficiency. Furthermore, by setting preset data processing rules, the data in the field to be synchronized is processed in a regularized manner, making the target data more standardized and normalized, providing higher quality data to the target device, and improving data validity. In addition, regardless of whether there is one or more data sources to be synchronized, or whether it is a single type of data source or multiple types of data sources, data querying and synchronization can be completed through the automatic adaptation of the target driver programs for each data source, solving the limitations of traditional data synchronization methods and making data synchronization highly scalable and efficient.

[0046] In one possible implementation, the computing device 20 can execute the data synchronization method of this application, including: 101. Determine the driver repository, that is, configure the driver repository on the computing device. The driver repository includes at least one driver.

[0047] 102. Receive synchronization requests. These requests include the data source type and fields to be synchronized. The data source to be synchronized can be at least one of one or more data sources (10).

[0048] 103. Driver matching, which means determining the target driver that matches the data source to be synchronized from at least one driver in the driver repository, based on the data source type of the data source to be synchronized.

[0049] 104. Data acquisition, which is to acquire the data of the fields to be synchronized based on the target driver.

[0050] 105. Data processing, which is to process the data of the fields to be synchronized based on preset data processing rules to obtain the target data.

[0051] 106. Data synchronization, which means synchronizing the target data to the target device, which is the device corresponding to the application management system 30.

[0052] like Figure 2 The figure shown is an application example diagram of a data synchronization method provided in an embodiment of this application.

[0053] Taking driver repository 201, which includes database driver 2011, API driver 2012, and IoT driver 2013, as an example, computing device 20 receives synchronization request 202. Synchronization request 202 may include: the data source type of the data source to be synchronized and the fields to be synchronized. Then, driver matching 203 can be performed, that is, determining the target driver corresponding to the data source type in the synchronization request. The target driver is then used to connect to the data source to be synchronized.

[0054] For example, if the data source type of the data source to be synchronized is a database type, then execute 204 to load the database driver; if the data source type of the data source to be synchronized is an API type, then execute 205 to load the API driver; if the data source type of the data source to be synchronized is an IoT type, then execute 206 to load the IoT driver.

[0055] By executing any of steps 204, 205, or 206, step 207 is performed to connect to the data source to be synchronized. Afterward, computing device 20 can perform step 208: data reading, processing, and synchronization; reading data from the data source to be synchronized for the field to be synchronized; processing the data based on preset data processing rules to obtain the target data; and synchronizing the target data to the target device.

[0056] like Figure 3 The diagram shown is a flowchart of a data synchronization method provided in an embodiment of this application, applied to a computing device, which includes at least one driver. The data synchronization method may include the following steps: S301. Receive a synchronization request. The synchronization request includes the type of the data source to be synchronized and the fields of the data source to be synchronized.

[0057] Optionally, after S301, it also includes: parsing the synchronization request to obtain the type of the data source to be synchronized and the fields of the data source to be synchronized.

[0058] Optionally, the method further includes: determining a configured driver repository, the driver repository including at least one driver; and configuring at least one driver from the driver repository in the computing device.

[0059] Understandably, a driver can refer to a software component used to connect (the driver can be responsible for the network connection between the application and the database) and operate the database. Through defined interface programs or functions, it acts as a bridge between the upper-layer application (such as the application that generates synchronization requests) and the database, enabling the upper-layer application to access the corresponding database using a standardized programming language. For example, a driver can translate the requests from the upper-layer application into instructions that the database can understand, and then translate the query results returned by the database into data objects that the upper-layer application can understand.

[0060] The driver repository can be updated dynamically, and at least one driver in the repository can be updated at any time. A driver can refer to a program used to access a data source. For example, if the data source is Kafka, the driver can be a database driver.

[0061] In one possible implementation, a data source access interface can be displayed, the relevant access operations performed by the user on the access interface can be detected, and a data source synchronization request can be generated. The relevant access operations performed by the user on the access interface may include, for example, the user entering relevant information about the data source to be synchronized on the access interface, and clicking on a confirmation control on the access interface.

[0062] After receiving a synchronization request, keywords indicating the data source type can be parsed. For example, a synchronization request can be represented as a URL link, and the URL link can be parsed to obtain keywords indicating the data source type to be synchronized, such as "jdbc:oracle", "https: / / api.", ".kafka.", etc.

[0063] After parsing the keywords in the synchronization request, the target driver that matches the keyword can be queried based on the keyword that represents the data source type to be synchronized.

[0064] S302. Based on the data source type of the data source to be synchronized, determine the target driver that matches the data source to be synchronized from at least one driver.

[0065] Optionally, the data source type can be represented using keywords for the data source type. Each driver can then be associated with a corresponding keyword for the data source type. Therefore, S302 can specifically refer to: determining the keyword used to represent the data source type to be synchronized; querying from the keywords corresponding to at least one driver the target keyword that is the same as the keyword used for the data source type to be synchronized; and determining the driver corresponding to the target keyword as the target driver.

[0066] Here, url is the synchronization request, and startswith refers to checking if there are keywords in the URL that represent the data source type to be synchronized, such as keywords like "jdbc:oracle", "https: / / api.", or ".kafka."

[0067] For example, if the URL representing a synchronization request includes "jdbc:oracle", the keyword indicating the data source type is "oracle". The driver associated with "oracle" is the target driver.

[0068] If the URL indicating a synchronous request includes https: / / api., the keyword representing the data source type is "rest". The driver associated with "rest" is the target driver.

[0069] If the URL indicating a synchronization request includes "kafka: / / ", the keyword indicating the data source type is "kafka". The driver associated with "kafka" is the target driver.

[0070] If no target driver matching the data source to be synchronized is found in at least one driver from the driver repository, an access exception message can be displayed, indicating that the data source type to be synchronized is not supported.

[0071] S303. Based on the target driver, obtain the data of the field to be synchronized.

[0072] Optionally, S303 may include executing a target driver, connecting to the data source to be synchronized, and reading data of the fields to be synchronized from the data source through the target driver. Executing the target driver to connect to the data source specifically refers to establishing a network connection with the data source through the target driver, so as to achieve data communication with the data source through this network connection. When the computing device communicates with the data source through the target driver, it can communicate using a communication protocol adapted to the data source type of the data source. The communication protocols adapted to different types of data sources are known or pre-configured.

[0073] For example, the data source can be a database such as Kafka or Oracle, and the data source type can be any database type. The compatible communication protocol for the database can be, for example, JDBC (Java Database Connectivity). The communication protocol between the target driver and the data source to be synchronized can also be the JDBC protocol.

[0074] For example, the data source is the data source corresponding to the API interface, the data source type is the API type, and the communication protocol for the API type is the REST (Representational State Transfer) protocol. Communication between the API driver and the data source corresponding to the API interface on the computing device uses the REST protocol.

[0075] For example, the data source is an IoT data source, the data source type is IoT, and the communication protocol for IoT is MQTT (Message Queuing Telemetry Transport). When a computing device uses an IoT driver to communicate with the IoT data source, it must do so via the MQTT protocol.

[0076] Optionally, the synchronization request may also include: the source address configured for the data source to be synchronized. Connecting to the data source to be synchronized can be done through the configured source address. Taking the data source of an ITO device as an example, the configured source address could be: mqtt: / / iot-broker:1883 / sensors (the address for reading data from the ITO device's data source). Data in the fields to be synchronized can be read from the data source to be synchronized through the configured source address.

[0077] Optionally, the field to be synchronized can be a field in the data source to be synchronized. The target driver can be used directly to read the data of the field to be synchronized from the data source.

[0078] S304. Based on preset data processing rules, process the data of the field to be synchronized to obtain the target data.

[0079] Data processing rules can process the data in the fields to be synchronized to obtain the target data. For example, they can perform data filtering and validation, and semantic processing on the data in the fields to be synchronized.

[0080] Specifically, preset interception rules and / or validation rules can be used to filter and validate the data in the fields to be synchronized. Interception rules refer to intercepting erroneous data in the data stream. Validation rules include at least one of the following: logical validation rules, correlation validation rules, and format validation rules.

[0081] Semantic mapping rules can also be used to perform semantic processing on data. Semantic mapping rules can include at least one of the following: semantic unification rules, threshold conversion rules, dynamic calculation rules, lineage tracing rules, or dynamic conversion rules. Semantic unification rules map fields from different data sources to standard semantic fields. Threshold conversion rules map different values ​​of the same field from different data sources to standard values. Dynamic calculation rules dynamically calculate the value of a field based on its meaning. Lineage tracing rules ensure that fields in the target data stream are updated synchronously when fields in any data source are updated.

[0082] S305. Synchronize the target data to the target device.

[0083] Optionally, to reduce transmission costs, the same communication protocol as the data source to be synchronized can be used to communicate with the target device. Specifically, the communication protocol with the target device can be determined based on the data source type of the data source to be synchronized. Based on the communication protocol with the target device, the target data is encapsulated and sent to the target device. As mentioned above, the communication protocol with the target device is determined according to the mapping relationship between the data source type and the communication protocol. If the data source to be synchronized is of type API, the communication protocol can be determined to be the REST protocol. If the data source is of type IoT, the communication protocol can be determined to be the MQTT protocol. A mapping relationship between the data source type and the communication protocol with the target device can be established in advance, and the communication protocol corresponding to the data source type of the data source to be synchronized can be queried through this mapping relationship.

[0084] Alternatively, if the target data has a uniform format and content after data processing, the computing device and the target device can negotiate or define a custom communication protocol. Specifically, the target data can be encapsulated based on a communication protocol negotiated with or pre-agreed upon with the target device, and the encapsulated target data can be sent to the target device.

[0085] In this embodiment, at least one driver is first configured in the computing device. This driver can be used to access and query data from a data source. A synchronization request is received, which may include the data source type and the fields to be synchronized. The data source type is related to the driver of the data source. Therefore, a target driver matching the data source to be synchronized can be determined from at least one driver. Based on the target driver, the data source to be synchronized is connected, and the data of the fields to be synchronized is read. This achieves automated matching and rapid connection for data source access, as well as reading the data of the fields to be synchronized, thereby improving the standardization, maintainability, and scalability of data source access, reducing the difficulty of data source access, and improving access efficiency. Furthermore, by setting preset data processing rules, the data of the fields to be synchronized is processed in a regularized manner, making the target data more standardized and normalized, providing higher quality data to the target device, and improving data validity. In addition, regardless of whether there is one or more data sources to be synchronized, or whether it is a single type of data source or multiple types of data sources, data querying and synchronization can be completed through the automatic adaptation of the target drivers for each data source, solving the limitations of traditional data synchronization methods and making data synchronization highly scalable and efficient.

[0086] To improve the connection efficiency of the data source to be synchronized, this application embodiment designs an optimized connection method. Specifically, configuration parameters corresponding to each data source type can be pre-configured, and the data source access can be automatically completed through the configuration parameters.

[0087] Therefore, obtaining the data of the field to be synchronized based on the target driver can include the following steps: A1. Determine the configuration parameters corresponding to the data source type of the data source to be synchronized.

[0088] A2. Based on the configuration parameters, run the target driver and use the target driver to obtain the data of the fields to be synchronized.

[0089] Optionally, configuration parameters can be entered into the target driver to run the target driver and complete the connection to the data source to be synchronized.

[0090] Optionally, configuration parameters may include, for example, timeout duration, maximum number of reconnections, encryption algorithm, maximum data size, etc.

[0091] It is understandable that the configuration parameters for each data source type are pre-configured. For example, the encryption algorithm and the maximum number of reconnections are pre-configured.

[0092] Obtaining data of fields to be synchronized using the target driver can include: obtaining data of fields to be synchronized through program modules such as field parsing, interface calls, and data processing in the target driver.

[0093] In this embodiment, configuration parameters corresponding to the data source type to be synchronized are determined, and these parameters are input into the target driver program, which then runs. This automates the driver program's operation, avoiding manual parameter configuration and ensuring efficient operation based on the adapted parameters, thus improving the stability and success rate of the data source connection. Associating the data source type with configuration parameters enables automatic parameter configuration, facilitating unified management and maintenance of connection rules for different data sources, and further enhancing the standardization and scalability of the multi-data source access process.

[0094] To improve the management efficiency of configuration parameters for different data source types, a mapping table can be used to store the configuration parameters for each data source type. Accordingly, A1, determine the configuration parameters corresponding to the data source type to be synchronized, which may specifically include: Based on a pre-set mapping table, query the configuration parameters corresponding to the data source type of the data source to be synchronized. The mapping table includes at least one set of mapping relationships, which include the data source type and the configuration parameters corresponding to the data source type.

[0095] Optionally, the mapping table may include configuration parameters corresponding to multiple data source types.

[0096] Furthermore, the mapping table (Map) associates the corresponding driver with each data source type, such as naming it ConnectionConfig. The driver's configuration parameters can include the communication protocol and the parameter DEFAULT_CONFIG.

[0097] For example, the driver for the "oracle" data source is: The function `new ConnectionConfig(maxWait=5000, encrypt=true)` creates a new ConnectionConfig instance. Here, ConnectionConfig is the driver to be created, and `maxWait(Max Wait Time)=5000, encrypt(Enable Encryption)=true` are the configuration parameters corresponding to the data source type.

[0098] The driver for the "kafka" data source is: The function `new ConnectionConfig(maxPoll=1000, autoCommit=false)` is defined as follows: `newConnectionConfig` is the driver program, and `maxPoll(Max Poll Records, maximum number of records to pull)=1000, autoCommit(disable autocommit)=false` are the configuration parameters corresponding to the data source type.

[0099] In the above embodiments, ConnectionOptimizer is a connection optimization knowledge base, and RULES = .of{} In this embodiment, by pre-constructing a mapping table containing the mapping relationship between data source types and corresponding configuration parameters, configuration parameters matching the data source type can be directly queried when accessing the data source. This achieves fast and accurate matching and automated acquisition of configuration parameters, improving the efficiency and accuracy of parameter acquisition. The mapping table unifies and makes transparent the configuration rules for different data source types. When adding new data source types or adjusting configuration parameters, only the mapping table needs to be maintained, without modifying the core business logic. This significantly enhances the flexibility and scalability of configuration management, providing crucial support for efficient and stable data source connections.

[0100] As mentioned above, a driver repository can include one or more drivers. To improve the management efficiency of the driver repository, it also includes: Receive a registration request, which includes driver information for unregistered data source types; In response to the registration request, the driver corresponding to the driver information is registered and added to the driver repository.

[0101] Optionally, driver information may include, for example, the data source type. The registration method for each data source type has been pre-set.

[0102] Specifically, registering the driver information can include taking the driver information and the registration method of the corresponding driver as parameters and inputting them into the registration function to complete the driver registration.

[0103] Taking a data source type of "graphql" as an example, the corresponding registration method for this data source type is GraphqlDriver(). Then, "graphql" and GraphqlDriver() are passed as parameters to the registration function, which is then executed to complete the driver registration. For example, registerDriver("graphql", new GraphQLDriver()), where registerDriver is a pre-written registration function.

[0104] In this embodiment, a registration request can be received, which may contain driver information corresponding to an unregistered data source type. The driver is then registered to the driver repository in response to the request. This enables dynamic expansion of the driver and efficient management of the driver repository, allowing for rapid adaptation to the access requirements of new data types and significantly improving the scalability and adaptability of the entire data source access system.

[0105] During the use of a data source, the producer of the data source (such as the data acquisition party) and the consumer (such as...) Figure 1 The application management system (30) relies on unified metadata, such as schema format, to avoid data anomalies such as parsing errors and interception caused by inconsistent metadata between the two parties. Therefore, the schema format of the data source producer and consumer should be consistent.

[0106] Understandably, data source metadata can be used to describe the data source's own attributes, structure, relationships, and other characteristics. This may include at least one of the following: data source type (e.g., MySQL, Oracle, Kafka, REST API), connection address, port, access account permissions, creation time, server, field names, field types (e.g., int, varchar, datetime), field length, primary key, foreign key, index information, data storage path, transmission protocol, data format (e.g., JSON, CSV), compression method, etc.

[0107] Because data source metadata has many components and types, it changes frequently. However, if consumers cannot promptly detect changes in the metadata of the data source producer during or after data retrieval, it can easily lead to abnormal data usage. Therefore, this application embodiment designs a metadata update mechanism.

[0108] Specifically, it also includes: Read the current metadata of the connected data source; The current metadata of the connected data source is compared with the historical metadata to obtain the comparison result, which indicates whether the metadata has changed or not. If the comparison result indicates that the metadata has changed, delete the historical metadata and save the current metadata.

[0109] Furthermore, if the comparison result indicates that the metadata has changed, the target device can be notified to update the metadata of the connected data source to the current metadata.

[0110] Optionally, the current metadata of each connected data source can be read periodically. The historical metadata of the connected data source can be the metadata of the last time it was read.

[0111] Saving current metadata can refer to storing it locally. Specifically, it means synchronizing the current metadata of the connected data source with updates in the data processing rules. For example, if the port of the metadata changes, the port value in the metadata in the data processing rules will be replaced with the current port value.

[0112] For ease of understanding, such as Figure 4 The diagram shown is an example of dynamic management of a data source provided in an embodiment of this application.

[0113] After determining the target driver for the data source, you can execute step 401, Smart Parameter Configuration and Data Acquisition, which corresponds to steps A1 and A2 above. Next, execute step 402, Data Processing and Synchronization. Then, execute step 403, Continuous Monitoring of the Metadata of the Connected Data Source, which involves reading the current metadata of the connected data source at a preset reading frequency. After comparing the current metadata of the connected data source with historical metadata, if the comparison result shows that the metadata has changed, execute step 404, Metadata Modification, such as deleting historical metadata and saving the current metadata. Metadata changes can be reported back to step 401 to ensure the consistency and integrity of the data source.

[0114] For example, the historical metadata of an IoT data source might be: "Device ID": {"type": "string"} and "Temperature": {"type": "number"}.

[0115] The current metadata for the IoT data source is: "Device ID": {"type": "string"}, "Temperature": {"type": "number"}, "Timestamp": {"type": "integer"}.

[0116] By comparing the historical metadata of the IoT data source with the current metadata, a "timestamp" field can be added to the current metadata of the IoT data source. Therefore, computing devices can delete the historical metadata of the IoT data source and save the current metadata. Furthermore, target devices can be notified to update the metadata of the IoT data source to the current metadata.

[0117] In this embodiment, the current metadata of the connected data source can be read and compared with historical data to determine whether the metadata of the connected data source has changed. When a metadata change is confirmed, the metadata is updated, achieving real-time awareness and dynamic updating of the metadata of the connected data source. This ensures that the stored metadata always remains consistent with the actual state of the data source, providing an accurate data foundation for data source-related operations.

[0118] It's understandable that fields from different data sources may have inconsistent formats. Therefore, after retrieving the data to be synchronized, the format needs to be converted to the fields that require synchronization. Thus, for the data in the fields to be synchronized, one or both of data governance and semantic consistency rules can be used to improve data consistency and effectiveness. For example... Figure 5 The diagram shown is another flowchart of a data synchronization method provided in this application. The difference from the previous embodiments lies in that, based on preset data processing rules, the data of the field to be synchronized is processed to obtain the target data, including: S501. Based on preset data governance rules and / or semantic mapping rules, perform data processing on the data of the field to be synchronized to obtain the target data.

[0119] In this embodiment, data integration is performed according to preset data processing rules to obtain the target data stream, achieving unified logical processing of the data and ensuring the accuracy and applicability of data delivery. The obtained target data stream is then sent to the application management system, providing a stable channel for the application management system to obtain reliable and standardized data support, effectively connecting the data source and the application layer, and enabling efficient data querying and display.

[0120] like Figure 6 The diagram shown is an example of a data processing rule provided according to an embodiment of this application. The preset data processing rule 601 may include: a data governance rule 602 and / or a semantic mapping rule 603.

[0121] Among them, data governance rule 602 may include at least one of the following: interception rule 6021, verification rule 6022, or dynamic conversion rule 6023.

[0122] Optionally, interception rule 6021 is used to intercept erroneous data. An interception rule can include one or more interception conditions / formulas. For data in the field to be synchronized or data obtained through other data processing rules, erroneous data can be intercepted using one or more interception conditions / formulas. For example, the interception formula could be "amount<0", meaning that for the amount field, data less than 0 will be filtered or intercepted to avoid data contamination.

[0123] Optionally, the verification rule 6022 may include at least one of the following: logical verification rule S1, correlation verification rule S2, and format verification rule S3.

[0124] The logical validation rule S1 can be used to filter or block data that does not conform to a preset basic logic. This basic logic can also include a preset logical validation formula or condition. For example, in the e-commerce field, fields include sales volume and inventory. If a merchant provides data showing that sales volume is greater than inventory, this data clearly does not conform to basic data rules. This can be used as a logical validation condition to filter or block data where "sales volume is greater than inventory".

[0125] The correlation validation rule S2 can be used to validate two or more parameters that are correlated, determining whether the correlation between the two or more parameters is valid. For example, consider two parameters: a user's VIP level and their spending amount. A higher VIP level corresponds to a higher spending amount, and a decrease in spending amount should result in a lower VIP level. For instance, if a user's spending amount parameter total_spent < 1,000,000, then that user's VIP level can be lowered.

[0126] Format validation rule S3 can be used to filter or block data that does not conform to the preset data format. For example, when a mobile phone number is used as a field, the format of the mobile phone number can be used as a format validation rule. If the format validation rule for the mobile phone number is "\\+86[0-9]{11}", the mobile phone number can be matched with this format validation rule. If the mobile phone number does not match any of the format validation rules, it means that the mobile phone number format is incorrect, and the mobile phone number can be filtered or blocked.

[0127] Dynamic conversion rule 6023 can be used to convert fields with the same meaning but different expressions into standard semantic fields according to a standard format. Specifically, fields with the same meaning but different expressions can all be converted to standard format fields. For example, addresses from multiple data sources can be represented using strings like "street," "street," and "ST," all of which can be mapped to the standard field "address." Similarly, mobile phone numbers, telephone numbers, and contact information have the same meaning and can all be mapped to mobile phone numbers.

[0128] The semantic mapping rule 603 may include at least one of the following: semantic unification rule 6031, threshold conversion rule 6032, dynamic calculation rule 6033, and lineage tracing rule 6034.

[0129] Among them, semantic unification rule 6031 refers to mapping fields from different data sources to standard semantic fields.

[0130] Optionally, threshold conversion rule 6032 refers to mapping different values ​​of the same field from different data sources to a standard value. For example, the status (field) value of data source A is represented by 1 / 0. The status value of data source B is represented by ACTIVE / INACTIVE. 1 and ACTIVE can be converted, and 0 and INACTIVE can be converted.

[0131] Dynamic calculation rule 6033 refers to dynamically calculating the value of a field based on its meaning. For example, if different departments repeatedly calculate customer ages, the age calculation only needs to be performed once, eliminating data discrepancies from different data sources. The customer whose age is being calculated can be called a data object, and related data for the object can refer to, for example, age. The calculation result of the data object is the calculated customer age.

[0132] Lineage tracing rule 6034 means that when a field in any data source is updated, the field in the target data stream is updated synchronously. Lineage tracing rules can, for example, track the entire data chain from generation, intermediate processing and transformation, to downstream consumption, accurately recording which downstream tables / tasks / interfaces / applications / reports directly / indirectly reference a field, whether the field has been renamed / calculated / aggregated, or the final business node to which the field flows. For example, the source system fields and the business system fields are consistent. For example, when the source system fields are updated, the target system fields can be updated accordingly to ensure field consistency.

[0133] In this embodiment, the data governance rules include at least one of interception rules, verification rules, and dynamic transformation rules. Interception rules can filter erroneous data to reduce noise, verification rules can ensure data compliance from different dimensions, and dynamic transformation rules unify data into a standard format. These three layers of rules work together to improve data quality from the source. Semantic mapping rules include at least one of semantic unification rules, threshold transformation rules, dynamic calculation rules, and lineage tracing rules. Semantic unification rules can eliminate naming differences between different fields, threshold transformation rules can resolve value conflicts for the same field, dynamic calculation rules can dynamically calculate accurate field values ​​based on field meaning, and lineage tracing rules can ensure end-to-end consistency during data changes. Through one or more of these rules, deep alignment of data semantics and business unification can be achieved. By ensuring data qualification and availability through data governance rules and achieving data semantic unification through mapping rules, a high-quality, highly consistent, and highly business-adaptable target data stream is provided to the application management system, significantly reducing the complexity of data integration and improving data application efficiency.

[0134] In one possible design, depending on the data processing rules adopted, the data processing steps for the fields to be synchronized can vary in the following ways: Scenario 1: Based on preset data governance rules, perform data governance on the data of the fields to be synchronized to obtain the target data.

[0135] Specifically, based on preset data governance rules, the data of the fields to be synchronized is filtered and validated to obtain the target data.

[0136] Based on scenario 1, and based on preset data governance rules, data governance is performed on the data of the fields to be synchronized to obtain the target data, including: Filter out erroneous data and data that does not conform to preset validation rules from the data in the fields to be synchronized, and obtain the filtered data; Data that does not conform to the preset standard format in the filtered data is converted into data that conforms to the standard format to obtain the target data.

[0137] Alternatively, it can be done through, for example Figure 6 The interception rule 6021 and verification rule 6022 shown filter out erroneous data and data that do not conform to the preset verification rules in the data of the field to be synchronized, and obtain the filtered data.

[0138] It can be done as follows Figure 6 The dynamic conversion rule 6023 shown converts data that does not conform to the preset standard format into data that conforms to the standard format after filtering, thereby obtaining the target data.

[0139] Therefore, by using data governance rules, erroneous data and data that do not conform to preset verification rules in the data of the fields to be synchronized are filtered out. Data that does not conform to the preset standard format in the filtered data is then converted to obtain target data that conforms to the standard format. By filtering invalid data and verifying data, data quality is guaranteed from the source, redundant and abnormal data are eliminated, and more reliable and secure data is provided for subsequent application management systems.

[0140] In this embodiment of the application, it is specified that data processing rules may include data governance rules. By using data governance rules to govern the data stream and obtain target data that conforms to the data governance rules, dirty data that does not conform to the data governance rules can be prevented from flowing into subsequent processes, thereby improving the overall data synchronization efficiency and quality and providing reliable data support for subsequent data.

[0141] Scenario 2: Based on preset semantic mapping rules, perform semantic mapping processing on the data of the fields to be synchronized to obtain the target data.

[0142] Based on scenario 2, and based on preset semantic mapping rules, semantic mapping processing is performed on the data of the fields to be synchronized to obtain the target data, including: The different names of the fields to be synchronized in different data sources are uniformly mapped to standard semantic fields, and the values ​​of the standard semantic fields are obtained. The target data includes the values ​​of the standard semantic fields.

[0143] Among them, it can be achieved through, for example Figure 6 The semantic unification rule 6031 shown maps the different names of the fields to be synchronized in different data sources to a standard semantic field.

[0144] And / or, map different values ​​of the same field to be synchronized in different data sources to a standard value to obtain the standard value of the field to be synchronized, and the target data includes the standard value of the field to be synchronized.

[0145] Among them, it can be achieved through, for example Figure 6 The dynamic conversion rule 6023 shown maps different values ​​of the same field to be synchronized in different data sources to a standard value, thereby obtaining the standard value of the field to be synchronized.

[0146] And / or, if the field to be synchronized is updated in the data source to be synchronized, the value of the field to be synchronized is updated locally to obtain the updated value of the field to be synchronized. The target data includes the updated value of the field to be synchronized.

[0147] Among them, the lineage tracking rule 6034 can be used to execute the following: if the field to be synchronized is updated in the data source to be synchronized, the value of the field to be synchronized will be updated locally to obtain the value of the field to be synchronized after synchronization.

[0148] And / or, if the field to be synchronized involves multiple calculations of the same data object, then based on the data related to the data object in the data of the field to be synchronized, perform a calculation of the target value of the data object once to obtain the calculation result of the data object. The target data includes the calculation result of the data object.

[0149] Among them, if the field to be synchronized involves multiple calculations of the same data object, then the target value of the data object can be calculated once based on the data related to the data object in the data of the field to be synchronized, and the calculation result of the data object can be obtained.

[0150] The target data may include: the values ​​of standard semantic fields, the standard values ​​of the fields to be synchronized, the values ​​of the fields to be synchronized after synchronization, and / or the calculation results of data objects.

[0151] Therefore, unifying the semantics of fields across different data sources can eliminate naming differences. Mapping the same field value to a standard value across different data sources resolves value conflicts. When a field to be synchronized is updated in the data source, its value is updated locally, enabling traceable updates and ensuring end-to-end data consistency during data changes. Furthermore, when the same data object involves multiple calculations, only one calculation is needed, reducing computational burden. Through these semantic mapping rules, deep alignment of data semantics and business consistency can be achieved, ensuring data quality and availability while maintaining consistent data meaning. This provides application management systems with a high-quality, highly consistent, and highly business-adaptable target data stream, significantly reducing the complexity of data integration and improving data application efficiency.

[0152] In this embodiment, semantic mapping rules are used to perform semantic analysis on the data of the fields to be synchronized to obtain the target data. By using semantic mapping, the semantic rules of the data are unified, which ensures the accuracy and compliance of the target data stream, and also achieves semantic standardization and unification. This provides high-quality data for the application management system and improves the usability and value of the data.

[0153] Scenario 3: Based on preset data governance rules and semantic mapping rules, perform data integration on the fields to be synchronized to obtain the integrated target data.

[0154] Based on scenario 3, data governance rules and semantic mapping rules can be executed sequentially. For example, data governance rules can be executed first, followed by semantic mapping rules. Alternatively, semantic mapping rules can be executed first, followed by data governance rules.

[0155] Taking the execution of data governance rules followed by semantic mapping rules as an example, based on preset data processing rules, the data streams of the fields to be synchronized are integrated to obtain the integrated target data stream, including: Based on preset data governance rules, the data of the fields to be synchronized is filtered and validated to obtain the first data; based on preset semantic mapping rules, the first data is semantically processed to obtain the target data.

[0156] Taking the execution of semantic mapping rules followed by data governance rules as an example, based on preset data processing rules, the data streams of the fields to be synchronized are integrated to obtain the integrated target data stream, including: Based on preset semantic mapping rules, semantic processing is performed on the data of the field to be synchronized to obtain the second data; based on preset data governance rules, data governance rules are applied to the second data to obtain the target data.

[0157] In this embodiment, the data processing rules are specified to include data governance rules and semantic mapping rules. Data is processed in stages through these rules. Data governance rules filter and validate the data stream, ensuring data quality from the source by filtering invalid data and validating data, eliminating redundant and abnormal data, and providing more reliable and secure data for subsequent application management systems. Semantic mapping rules perform semantic analysis on the data, achieving semantic alignment and unified interpretation, eliminating semantic differences between multi-source data, and ensuring consistent data expression across different data sources. Therefore, overall, this approach guarantees the accuracy and compliance of the target data stream while achieving semantic standardization and unification, providing high-quality data for application management systems and enhancing data usability and value.

[0158] like Figure 7 The diagram shown is another application example of a data synchronization method provided in this application. (See reference...) Figure 7 There can be several connected data sources 'a', and the driver repository can include multiple drivers such as database drivers, API drivers, and IoT drivers. Based on this, the following steps can be performed: S701. Receive a synchronization request. The synchronization request includes the data source type of the data source to be synchronized and the fields of the data source to be synchronized.

[0159] S702, Driver Matching: Based on the data source type of the data source to be synchronized, determine the target driver that matches the data source to be synchronized from at least one driver in the driver repository.

[0160] Optionally, after determining the target driver that matches the data source to be synchronized, the target driver can be loaded.

[0161] If a database type is matched, then execute S703 and load the database driver.

[0162] If an API type is matched, execute S704 and load the API driver.

[0163] If an IoT type is matched, execute S705 and load the IoT driver.

[0164] The target driver is loaded to execute S707 and connect to the data source to be synchronized.

[0165] Assume there is an existing data source a, and the data source to be synchronized is data source b. Data source b can be connected as a new data source. Figure 7 The data sources a and b shown are both already connected.

[0166] Based on this, S708, data reading, can be executed to read the data of the field to be synchronized from the data source to be synchronized (such as data source b) based on the target driver. Then, S709, data processing, can be executed, that is, based on preset data processing rules, the data of the field to be synchronized is processed to obtain the target data. The target data can then be sent to the application management system 30 in S710.

[0167] As an example, if the data stream is interrupted during the process of reading the corresponding data stream from the connected data source, the connected data source will be reconnected.

[0168] Optionally, if the data stream is interrupted, the connected data source is disconnected and needs to be reconnected. Specifically, this can be achieved through the above... Figure 3 Related embodiments can reconnect to the disconnected data source.

[0169] Optionally, when reconnecting to or accessing a disconnected data source, the number of reconnection attempts can be obtained, and the data source can be reconnected within the limit of the number of reconnection attempts until the data source is successfully reconnected.

[0170] In this embodiment, if a data stream interruption is detected during the reading process from an already connected data source, the data source is automatically reconnected, achieving fault self-healing and stability assurance in the data stream reading process. Furthermore, by real-time detection of abnormal interruptions in the data reading link and automatic retrying, data reading failures caused by link fluctuations or temporary data source failures are avoided, ensuring the continuity of data stream acquisition. This effectively reduces the impact of data interruptions on subsequent data integration and downstream application data supply, ensuring end-to-end stability from data acquisition to application.

[0171] like Figure 8 The diagram shown is a structural schematic of a data synchronization device provided in an embodiment of this application. The data synchronization device 800 includes: The request receiving unit 801 is used to receive synchronization requests, which include the data source type and fields to be synchronized.

[0172] The driver matching unit 802 is used to determine, from at least one driver, a target driver that matches the data source to be synchronized, based on the data source type of the data source to be synchronized.

[0173] The data acquisition unit 803 is used to acquire data of the fields to be synchronized based on the target driver.

[0174] The data processing unit 804 is used to process the data of the field to be synchronized based on preset data processing rules to obtain the target data.

[0175] The data synchronization unit 805 is used to synchronize target data to the target device.

[0176] As one embodiment, the data acquisition unit 803 may include: The parameter determination module is used to determine the configuration parameters corresponding to the data source type of the data source to be synchronized.

[0177] The program driver module is used to run the target driver based on configuration parameters, so as to obtain the data of the data field to be synchronized using the target driver.

[0178] As another embodiment, the parameter determination module can be specifically used to: query the configuration parameters corresponding to the data source type of the data source to be synchronized based on a pre-set mapping table. The mapping table includes at least one mapping relationship corresponding to each data source type, and the mapping relationship includes the data source type and the configuration parameters corresponding to the data source type.

[0179] As yet another embodiment, it also includes: The request unit is used to receive registration requests, which include driver information corresponding to unregistered data source types.

[0180] The registration unit is used to register the driver corresponding to the driver information in response to a registration request and store the driver in the computing device.

[0181] As yet another embodiment, it also includes: The schema reading unit is used to read the current schema metadata of the data source to be synchronized; The pattern synchronization unit is used to update the historical metadata based on the current metadata if the current metadata of the data source is different from the historical metadata that the data source has stored locally.

[0182] As another embodiment, the data processing rules include: data governance rules; the data processing unit 804 includes: The first processing module is used to perform data governance on the data of the fields to be synchronized based on preset data governance rules, so as to obtain the target data.

[0183] As another embodiment, the first processing module is specifically used to: filter out erroneous data and data that do not conform to preset verification rules in the data of the field to be synchronized, and obtain filtered data; convert data that does not conform to the preset standard format in the filtered data into data that conforms to the standard format, and obtain target data.

[0184] As another embodiment, the data processing rules include: semantic mapping rules. Data processing unit 804 includes: The second processing module is used to perform semantic mapping processing on the data of the fields to be synchronized based on preset semantic mapping rules, so as to obtain the target data.

[0185] As another embodiment, the second processing module is specifically used for: The different names of the fields to be synchronized in different data sources are uniformly mapped to standard semantic fields, and the values ​​of the standard semantic fields are obtained. The target data includes the values ​​of the standard semantic fields. And / or, map different values ​​of the same field to be synchronized in different data sources to standard values ​​to obtain the standard values ​​of the field to be synchronized, and the target data includes the standard values ​​of the field to be synchronized; And / or, if the field to be synchronized is updated in the data source to be synchronized, the value of the field to be synchronized is updated locally to obtain the updated value of the field to be synchronized. The target data includes the updated value of the field to be synchronized. And / or, if the field to be synchronized involves multiple calculations of the same data object, then based on the data related to the data object in the data of the field to be synchronized, perform a calculation of the target value of the data object once to obtain the calculation result of the data object. The target data includes the calculation result of the data object.

[0186] In the embodiments of this application, Figure 8 The device shown can also be a chip or a chip system, such as a system on chip (SoC) or a baseboard management controller (BMC).

[0187] Figure 9 This is a hardware block diagram of a computing device provided in an embodiment of this application. The computing device 900 according to an embodiment of this application includes at least a memory 901, a processor 902, and a transceiver 903. The memory 901 is used to store computer programs, and the transceiver 903 is used to communicate with data sources, application management systems, etc. The processor 902 is used to execute the computer program to implement the data synchronization method of any of the above embodiments.

[0188] In addition, the memory 901, processor 902, and transceiver 903 are all electrically connected to the bus 904.

[0189] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program. When executed by a processor, the computer program implements the data synchronization method of any of the preceding embodiments of this application.

[0190] Computer-readable storage media include, but are not limited to, volatile storage media and / or non-volatile storage media. Volatile storage media may include, for example, random access storage media (RAM) and / or cache storage media. Non-volatile storage media may include, for example, read-only storage media (ROM), hard disks, flash memory, optical disks, magnetic disks, etc.

[0191] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the data synchronization method of any of the preceding embodiments of this application.

[0192] The basic principles of the embodiments of this application have been described above with reference to specific examples. However, it should be noted that the advantages, benefits, and effects mentioned in the embodiments of this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the embodiments of this application from necessarily employing the aforementioned specific details.

[0193] The block diagrams of devices, apparatuses, devices, and systems involved in the embodiments of this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context explicitly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0194] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0195] It should also be noted that in the systems and methods of this application embodiment, each component or step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the embodiments of this application.

[0196] Various changes, substitutions, and modifications can be made to the technology herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the embodiments of this application is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0197] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use embodiments of this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of embodiments of this application. Therefore, embodiments of this application are not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0198] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A data synchronization method, characterized in that, The method is applied to a computing device, the computing device including at least one driver, and the method includes: Receive a synchronization request, the synchronization request including the data source type of the data source to be synchronized and the fields of the data source to be synchronized; Based on the data source type of the data source to be synchronized, a target driver matching the data source to be synchronized is determined from the at least one driver; Based on the target driver, obtain the data of the field to be synchronized; Based on preset data processing rules, the data of the fields to be synchronized is processed to obtain the target data; Synchronize the target data to the target device.

2. The method according to claim 1, characterized in that, The step of obtaining the data of the field to be synchronized based on the target driver includes: Determine the configuration parameters corresponding to the data source type of the data source to be synchronized; Based on the configuration parameters, the target driver is run to obtain the data of the field to be synchronized.

3. The method according to claim 2, characterized in that, The step of determining the configuration parameters corresponding to the type of the data source to be synchronized includes: Based on a pre-set mapping table, the configuration parameters corresponding to the data source type of the data source to be synchronized are queried. The mapping table includes at least one mapping relationship corresponding to each data source type, and the mapping relationship includes the data source type and the configuration parameters corresponding to the data source type.

4. The method according to any one of claims 1-3, characterized in that, Also includes: Receive a registration request, which includes driver information corresponding to an unregistered data source type; In response to the registration request, the driver corresponding to the driver information is registered, and the driver is stored in the computing device.

5. The method according to any one of claims 1-4, characterized in that, Also includes: Read the current metadata of the data source to be synchronized; If the current metadata is different from the historical metadata of the data source to be synchronized, update the historical metadata based on the current metadata.

6. The method according to any one of claims 1-5, characterized in that, The data processing rules include: data governance rules; The process of processing the data of the field to be synchronized based on preset data processing rules to obtain target data includes: Based on preset data governance rules, the data of the field to be synchronized is processed to obtain the target data.

7. The method according to claim 6, characterized in that, The process of performing data governance on the data of the field to be synchronized based on preset data governance rules to obtain the target data includes: Filter out erroneous data and data that do not conform to preset verification rules from the data of the field to be synchronized to obtain filtered data; Data that does not conform to the preset standard format in the filtered data is converted into data that conforms to the standard format to obtain the target data.

8. The method according to any one of claims 1-7, characterized in that, The process of processing the data of the field to be synchronized based on preset data processing rules to obtain target data includes: Based on preset semantic mapping rules, semantic mapping processing is performed on the data of the field to be synchronized to obtain the target data.

9. The method according to claim 8, characterized in that, The step of performing semantic mapping processing on the data of the field to be synchronized based on preset semantic mapping rules to obtain the target data includes: The different names of the fields to be synchronized in different data sources are uniformly mapped to standard semantic fields, and the values ​​of the standard semantic fields are obtained. The target data includes the values ​​of the standard semantic fields. And / or, map different values ​​of the same field to be synchronized in different data sources to standard values ​​to obtain the standard values ​​of the field to be synchronized, wherein the target data includes the standard values ​​of the field to be synchronized; And / or, if the field to be synchronized is updated in the data source to be synchronized, the value of the field to be synchronized is updated locally to obtain the updated value of the field to be synchronized, and the target data includes the updated value of the field to be synchronized. And / or, if the field to be synchronized involves multiple calculations of the same data object, then based on the data related to the data object in the data of the field to be synchronized, perform a calculation of the target value of the data object once to obtain the calculation result of the data object, wherein the target data includes the calculation result of the data object.

10. A computing device, characterized in that, include: A processor and a memory, the memory storing a computer program that is invoked by the processor to perform the data synchronization method according to any one of claims 1-9.

Images