Data Processing Method, Apparatus, Electronic Device, and Storage Medium

Abstract

The present application relates to a data processing method, apparatus, electronic device and storage medium, which can be applied to data synchronization of application programs such as maps and traffic. The method includes: if it is monitored based on a first preset path that a second data service system has a data change, updating historical service data in the first data service system according to first incremental data transmitted through the first preset path, and if there is a data conflict, performing data conflict processing according to a unified conflict processing policy to obtain first service data; if it is monitored based on a first data subscription path that the second data service system has a data change, updating the first service data according to second incremental data pushed through the first data subscription path, and if there is a data conflict, determining a target data provider of the conflict data; performing conflict processing on the conflict data based on a custom conflict processing policy of the target data provider. The technical solution of the present application can meet the dual requirements of data processing efficiency and custom conflict processing.

Description

Technical Field

[0001] The present application relates to the field of computer technologies, and in particular, to a data processing method, apparatus, electronic device, and storage medium. Background Art

[0002] Data providers generally store data in multiple databases, such as storing data in multiple cloud databases, so that traffic can be flexibly switched among multiple cloud databases. Based on this, it is generally necessary to synchronize data of multiple cloud databases. Data conflicts often occur during data synchronization. Cloud databases of different manufacturers have their own data conflict handling strategies, and the data conflict handling strategies of each manufacturer are relatively fixed to uniformly and stably provide data conflict handling for different data providers, and this can also meet the conflict handling requirements of most types of data.

[0003] The above unified method for handling data conflicts not only makes the data conflict handling methods of different data providers single, but also results in inaccurate data synchronization for some data. The existing data synchronization method of cloud databases aims at synchronization speed to meet the need for rapid traffic switching. Therefore, how to improve data synchronization accuracy on the basis of ensuring the synchronization speed of the existing data synchronization method is an urgent problem to be solved. Summary of the Invention

[0004] In view of the above technical problems, the present application provides a data processing method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of the present application, a data processing method is provided. The method includes:

[0006] When it is detected based on a first preset path that a second data service system has a data change, update the historical service data in the first data service system according to the first incremental data transmitted through the first preset path. If there is a data conflict, perform data conflict handling according to a first preset conflict handling strategy to obtain first service data; the first preset conflict handling strategy is the unified conflict handling strategy corresponding to the first data service system;

[0007] When it is detected based on a first data subscription path that the second data service system has a data change, update the first service data according to the second incremental data pushed through the first data subscription path. If there is a data conflict, determine the target data provider of the conflict data;

[0008] Obtain the custom conflict handling strategy corresponding to the target data provider;

[0009] Based on the custom conflict handling strategy, perform conflict handling on the conflict data.

[0010] According to another aspect of the present application, a data processing device is provided, including:

[0011] A first synchronization module, configured to, when it is detected based on a first preset path that data changes occur in a second data service system, update historical service data in the first data service system according to first incremental data transmitted through the first preset path, and if there is a data conflict, perform data conflict processing according to a first preset conflict processing policy to obtain first service data; the first preset conflict processing policy is a unified conflict processing policy corresponding to the first data service system;

[0012] A second synchronization module, configured to, when it is detected based on a first data subscription path that data changes occur in the second data service system, update the first service data according to second incremental data pushed through the first data subscription path, and if there is a data conflict, determine a target data provider of the conflicting data;

[0013] A conflict processing policy acquisition module, configured to acquire a custom conflict processing policy corresponding to the target data provider;

[0014] A conflict processing module, configured to perform conflict processing on the conflicting data based on the custom conflict processing policy.

[0015] According to another aspect of the present application, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the above method.

[0016] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein, when the computer program instructions are executed by a processor, the above method is implemented.

[0017] According to another aspect of the present application, a computer program product is provided, including computer instructions, which, when executed by a processor, cause a computer to execute the above method.

[0018] By monitoring data changes in the second data service system through two paths, the data synchronization processing of the first data service system can not only ensure the synchronization rate based on the first preset path, providing guarantee for the smooth traffic switching of the data provider; but also meet the custom processing requirements of the data provider based on the first data subscription path, enabling the data synchronization of different data providers to be customized according to different data providers' custom conflict processing, realizing refined and fine-grained data synchronization, so as to meet the dual requirements of the data provider's data synchronization efficiency and custom processing;

[0019] Moreover, the synchronization speed of the first data subscription path is relatively low, enabling custom conflict handling to occur after unified conflict handling, allowing the custom conflict handling of the data provider to override the unified conflict handling and reasonably meet the custom conflict handling requirements of the data provider.

[0020] In addition, this dual-path data synchronization method ensures that the custom conflict handling strategy of the data provider does not cause code intrusion into the unified conflict handling strategy of the data service system, safeguarding the security of the data service system and the unity of data management.

[0021] Other features and aspects of the present application will become clear from the following detailed description of the exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present application and are used to explain the principles of the present application.

[0023] Figure 1 A schematic diagram of an application system provided according to an embodiment of the present application is shown.

[0024] Figure 2 A flowchart of a data processing method provided according to an embodiment of the present application is shown.

[0025] Figure 3 A schematic diagram of the operating architecture of a two-way data synchronization provided according to an embodiment of the present application is shown.

[0026] Figure 4 A flowchart of a method for obtaining a custom conflict handling strategy corresponding to a target data provider provided according to an embodiment of the present application is shown.

[0027] Figure 5 A schematic diagram of the logical architecture of a two-way data synchronization provided according to an embodiment of the present application is shown.

[0028] Figure 6 A block diagram of a data processing device provided according to an embodiment of the present application is shown.

[0029] Figure 7 A block diagram of an electronic device for data processing provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. Like reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0031] As used herein, the term "exemplary" means "serving as an example, embodiment, or illustration". Any embodiment described as "exemplary" herein need not be construed as superior to or better than other embodiments.

[0032] In addition, for a better illustration of the present application, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present application can be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present application.

[0033] Please refer to Figure 1 , Figure 1 , showing a schematic diagram of an application system provided according to an embodiment of the present application. The application system can be used for the data processing method of the present application, that is, data synchronization processing. As Figure 1 shown, the application system can at least include a first data service system 01, a second data service system 02, a data provider 03, and a terminal 04 (a general term for terminals).

[0034] In an embodiment of the present application, the first data service system 01 can be used for data processing. The first data service system 01 can be a cloud service system, such as a public cloud, a private cloud, etc. The first data service system 01 can provide functions such as a data transmission service DTS (Data Transmission Service) and data subscription. For example, the first data service system 01 can include cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms.

[0035] In an embodiment of the present application, the second data service system 02 can be used to provide synchronized data. In an embodiment of the present application, the second data service system 02 can be a non-cloud system, such as a self-built IDC computer room, etc. Alternatively, the second data service system 02 can also be a cloud service system, such as a public cloud, a private cloud, etc. Correspondingly, the second data service system 02 can also provide functions such as a data transmission service DTS and data subscription. The second data service system 02 can include cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms.

[0036] In the embodiments of this specification, the above-mentioned first data service system 01 and the second data service system 02 may be connected by a dedicated line, and this application does not limit this.

[0037] Optionally, as Figure 1 shown, the data provider 03 may publish service data in the first data service system 01 and the second data service system 02, that is, the data provider 03 may provide data services for the terminal 04 by accessing the first data service system 01 and the second data service system 02. As an example, the data provider 03 may be a target application, and the terminal 04 using the target application may obtain the service of the target application through the first data service system 01 or the second data service system 02, or may also switch between the first data service system 01 and the second data service system 02. Based on this, in order to ensure the data service after traffic switching, data synchronization in the first data service system 01 and the second data service system 02 needs to be performed. Optionally, the terminal 04 may include physical devices such as in-vehicle terminals, smart phones, desktop computers, tablet computers, laptop computers, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices. It may also include software running on the physical device, such as application programs, etc. The operating system running on the terminal 04 in the embodiments of this application may include, but is not limited to, Android system, IOS system, Linux, Windows, etc.

[0038] Among them, the number of data providers 03 may be multiple, and the number of terminals 04 served by each data service system may also be multiple. The number of data service systems may also be multiple, that is, it may include a third data service system, a fourth data service system, etc. The data processing method of this application may equally apply between the first data service system and the third data service system or the fourth data service system, or may equally apply between the second data service system and the third data service system or the fourth data service system. Figure 1 It is only an example of data processing between any two data service systems.

[0039] Exemplarily, in the case where the target application is a traffic service application, multiple terminals can obtain real-time traffic services in the traffic service application. The traffic service application can be connected to the first data service system 01 and the second data service system 02 at the same time. In this way, the traffic service application can provide traffic service data for the multiple terminals (here, it can refer to in-vehicle terminals in multiple vehicles that communicate with the first data service system 01 and the second data service system 02), such as recording interaction data in the traffic service, deleting or adding data, etc. Based on this, the multiple terminals can switch between the first data service system 01 and the second data service system 02, or the first data service system 01 or the second data service system 02 fails, so data synchronization is selected between the first data service system 01 and the second data service system 02. However, the data synchronization between existing data service systems is based on a unified path and a unified conflict strategy. For example, the data synchronization between cloud databases can be achieved based on the DTS path and a unified conflict strategy. Although the synchronization rate is high and the smooth switching of traffic can be guaranteed, the data provider can only use the unified conflict handling strategy of the accessed data service system, and the unified conflict handling strategies of different data service systems may be different, which results in the data provider being unable to achieve the consistency processing of its own data.

[0040] In addition, considering directly coupling the custom conflict handling strategy of the data provider on the DTS, on the one hand, it will cause the synchronization rate of the DTS path to decrease, and on the other hand, since the data service system serves a large number of data providers, the unified conflict handling strategy of the data service system will be invaded by a large number of data providers' codes, which is not safe and not convenient for the unified management of data services. Based on these, the dual-path data synchronization method of this application is designed.

[0041] It should be noted that in the specific implementation of this application, when it comes to user-related data, when the following embodiments of this application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0042] Figure 2 The flowchart showing a data processing method provided by an embodiment of this application, that is, data synchronization processing, can be applied to the first data service system. It can be understood that the first data service system is any data service system that requires data synchronization. As Figure 2 shown, the data processing method can include:

[0043] S201. When it is detected based on the first preset path that data changes have occurred in the second data service system, update the historical service data in the first data service system according to the first incremental data transmitted through the first preset path. If there are data conflicts, handle the data conflicts according to the first preset conflict handling strategy to obtain the first service data.

[0044] Among them, the first preset conflict handling strategy can be the unified conflict handling strategy corresponding to the first data service system. The existence of data conflicts may refer to the existence of data with the same data index, such as data with the same Key, or data with the same library table identifier.

[0045] In practical applications, it is possible to monitor the data changes in the second data service system based on the first preset path. Thus, when it is detected that data changes have occurred in the second data service system, the first incremental data in the second data service system where the data changes have occurred can be obtained, and the first incremental data can be obtained using the first preset path. Correspondingly, the historical service data in the first data service system can be updated according to the first incremental data transmitted through the first preset path, that is, the data synchronization of the first incremental data in the second data service system to the first data service system is realized.

[0046] Correspondingly, if there are data conflicts, the first preset conflict handling strategy corresponding to the first data service system can be obtained, so that the data conflicts can be handled according to the first preset conflict handling strategy to obtain the first service data, that is, the historical service data and the first incremental data after the data conflicts are handled. That is to say, the first service data can be the full volume of service data in the first data service system after the data conflicts are handled.

[0047] Optionally, if there are no data conflicts, the first incremental data can be directly written into the first data service system.

[0048] In an alternative implementation manner, it is also possible to synchronize the historical full volume of data in the second data service system. Based on this, when it is detected based on the first preset path that data changes have occurred in the second data service system, the historical full volume of data in the second data service system can also be obtained through the first preset path for processing the historical full volume of data. In this way, in the case of a failure of the second data service system, it can be ensured that the traffic of the data provider can be smoothly switched to the first data service system for seamless connection of data services.

[0049] S203. When it is detected based on the first data subscription path that data changes have occurred in the second data service system, update the first service data according to the second incremental data pushed by the first data subscription path. If there are data conflicts, determine the target data provider of the conflicting data.

[0050] In one example, the synchronization speed of the first preset path can be higher than that of the first data subscription path, enabling the data synchronization process of the first data service system to ensure the synchronization rate based on the first preset path and providing guarantee for the smooth traffic switching of the data provider. For example, the first preset path can be a path based on the Data Transmission Service (DTS). The DTS path can synchronize data by simulating the replication protocol based on the binary log file (binlog), which is a synchronization based on binary data and can perform batch synchronization with high synchronization efficiency and low latency, approaching the synchronization rate in the MySQL master-slave mode. The first data subscription path can be a path based on the data subscription method, and data message analysis, etc. will be performed during data synchronization, resulting in a relatively slow synchronization speed.

[0051] As an example, the data transmission service and data subscription can be implemented based on cloud services, which is not limited in this disclosure.

[0052] In the embodiments of this specification, the data changes of the second data service system can be monitored based on the first data subscription path. Thus, in the case where data changes in the second data service system are detected based on the first data subscription path, the data with data changes in the second data service system, i.e., the second incremental data, can be obtained, and the second incremental data pushed based on the first data subscription path can also be obtained. Correspondingly, the first service data can be updated according to the second incremental data pushed by the first data subscription path, that is, the second incremental data can be written into the first data service system. During the writing process, if there is a data conflict, the target data provider of the conflict data can be determined. The conflict data can refer to the service data corresponding to the data conflict. Correspondingly, it can include the first conflict data in the first service data and the second conflict data in the second incremental data, that is, the first conflict data in the first service data and the second conflict data in the second incremental data have a data storage conflict, that is, the first conflict data and the second conflict data correspond to the same data index.

[0053] In the embodiments of this specification, a data index can be used to uniquely identify a piece of business data. Since in actual applications, the status or specific information of a piece of business data may change over time, conflicts may occur during the update of this piece of business data. As an example, the data index can be a library table identifier. For instance, the first data service system and the second data service system can set up multiple library tables to store different business data, so that the library table identifier can be used as the data index to identify a piece of business data. For example, a piece of business data can be the data of order A. When the data of order A is constantly changing, the incremental data of order A and the historical data of order A correspond to the same data index, such as order A. Then, it can be considered that there is a data conflict between the incremental data of order A and the historical data of order A. Correspondingly, the incremental data of order A and the historical data of order A can be regarded as conflicting data. Alternatively, a time dimension can be added to determine whether there is conflicting data. For example, when the data index and the data change time are both the same, it can be considered that there is a data conflict.

[0054] Furthermore, if there is a data conflict, the target data provider of the conflicting data, that is, the party to which the conflicting data belongs, such as the target application to which it belongs, can be determined. Since the first data service system and the second data service system provide functions such as data storage and data reading for data providers. The data processing of a terminal in a target application can be to read data and other processing from any data service system of the target application. Among them, any data service system of the target application can refer to any one of the data systems that provide data services for the target application. Based on this, the target application to which the conflicting data belongs can be determined as the target data provider.

[0055] Optionally, if there is no data conflict, the second incremental data can be directly written into the first data service system to achieve data synchronization between different data service systems.

[0056] In a possible implementation, to save resources, the data provider can preset the data monitored by the first data subscription path, so that not all service data needs to be monitored, which can save the bandwidth resources of the data provider. Specifically, the first data service system can provide an interface or interface for the data provider to set the preset subscription data, so that the data provider can set or change the preset subscription data. Based on this, step S203 may include: monitoring the preset subscription data in the second data service system based on the first data subscription path; the preset subscription data is the data subscribed in advance by at least one data provider; when the preset subscription data changes (the change of the preset subscription data may mean that all or part of the preset subscription data changes), the second incremental data pushed based on the first data subscription path can be obtained, and the second incremental data may refer to the preset subscription data that has changed in the second data service system; and updating the first service data according to the second incremental data. Among them, the changed preset subscription data may belong to the service data in which the second data service system is monitored to have data changes based on the first preset path, that is, the changed preset subscription data may be all or part of the service data in which the second data service system is monitored to have data changes based on the first preset path.

[0057] S205. Obtain the custom conflict handling policy corresponding to the target data provider.

[0058] In the embodiments of this specification, in order to enable the data provider to customize the handling of data conflicts while ensuring the data synchronization speed of the data provider, the custom conflict handling policy corresponding to the target data provider can be obtained. The custom conflict handling policy can be set by the target data provider based on the conflict policy setting interface or conflict policy setting interface provided by the first data service system. Based on this, the custom conflict handling policy corresponding to the target data provider can be screened out from the stored custom conflict handling policies of different data providers. Here, the custom conflict handling policies of different data providers can be stored in the form of custom conflict handling configuration information, that is, the custom conflict handling configuration information can include the corresponding relationship between different data providers and their respective custom conflict handling policies.

[0059] As an example, the custom conflict handling policy may refer to a custom conflict handling method, such as conflict handling methods such as overwriting and correction, which are not limited in this disclosure.

[0060] S207. Perform conflict handling on the conflict data based on the custom conflict handling policy.

[0061] In the embodiments of this specification, conflict processing can be directly performed on conflict data based on a custom conflict processing policy. For example, if the custom conflict processing method is the overwrite method, the first conflict data can be directly overwritten with the second conflict data.

[0062] Optionally, the traffic information of the first data service system can also be obtained. Correspondingly, S207 can include: when the traffic information is lower than the traffic threshold, performing conflict processing on the conflict data based on the custom conflict processing policy; when the traffic information is not lower than the traffic threshold, custom conflict processing may not be performed. In this way, in the case of high traffic, the processing resources can support high-traffic processing.

[0063] In an optional implementation manner, the target state of the custom conflict processing policy can be obtained. The target state can be the closed state or the open state. When the target state is the open state, conflict processing can be performed on the conflict data based on the custom conflict processing policy; when the target state is the closed state, custom conflict processing may not be performed. In this way, the data service system can be based on the target state of the custom conflict processing policy. Based on this, the method can further include: in response to a status setting request from a target data provider, extracting status setting indication information from the status setting request; and thus the status of the custom conflict processing policy can be updated based on the status setting indication information. The status setting indication information can be used to indicate the status of the custom conflict processing configuration information, such as the closed state or the open state. In this way, the data provider can dynamically enable or disable custom conflict processing to meet dynamic requirements.

[0064] By monitoring data changes in the second data service system through two channels, and the synchronization speed of the first preset channel is higher than that of the first data subscription channel, the data processing of the first data service system can not only ensure the synchronization rate based on the first preset channel, providing guarantee for the smooth traffic switching of the data provider; but also meet the custom processing requirements of the data provider based on the first data subscription channel, enabling the data synchronization of different data providers to be based on the custom conflict processing of different data providers, realizing refined and fine-grained data synchronization, and thus meeting the dual requirements of the data synchronization efficiency and custom processing of the data provider;

[0065] Moreover, the synchronization speed of the first data subscription channel is relatively low, enabling the custom conflict processing to be performed after the unified conflict processing, so that the custom conflict processing of the data provider can overwrite the unified conflict processing, reasonably meeting the custom conflict processing requirements of the data provider;

[0066] In addition, this dual-channel data synchronization method ensures that the custom conflict handling strategy of the data provider does not cause code intrusion into the unified conflict handling strategy of the data service system, thus guaranteeing the security of the data service system and the unity of data management.

[0067] Figure 3 FIG. shows a flowchart of a method for obtaining a custom conflict handling strategy corresponding to a target data provider according to an embodiment of the present application. As Figure 3 shown, in a possible implementation, the above step S205 may include:

[0068] S301, obtaining custom conflict handling configuration information corresponding to the target data provider; the custom conflict handling configuration information may include a mapping relationship between a service type and a first preset conflict handling strategy.

[0069] In practical applications, in order to perform custom conflict handling more precisely and with finer granularity, the custom conflict handling configuration information may include a mapping relationship between a service type and a first preset conflict handling strategy, so as to implement differential conflict handling based on the service type.

[0070] The custom conflict handling configuration information may be pre-set and stored in the first data service system by the target data provider. Based on this, the custom conflict handling configuration information corresponding to the target data provider can be directly found from the stored custom conflict handling configuration information of different data providers.

[0071] Optionally, the custom conflict handling configuration information can be dynamically updated. Based on this, in response to a conflict handling configuration request from the target data provider, the mapping relationship between the service type to be configured and the preset conflict handling strategy to be configured can be extracted from the conflict handling configuration request; thus, the custom conflict handling configuration information can be updated based on the mapping relationship between the service type to be configured and the preset conflict handling strategy to be configured. For example, modifying the original historical mapping relationship, or adding the mapping relationship to be configured, etc.

[0072] S303, determining the target service type of the conflict data;

[0073] S305, according to the mapping relationship, determining the first preset conflict handling strategy corresponding to the target service type as the custom conflict handling strategy.

[0074] In the embodiments of this specification, the target service type of the conflict data can be determined. The target service type can be any one of multiple service types set by the target data provider. The service type can be divided based on multiple dimensions of service data, and the present disclosure does not limit this. Further, the first preset conflict handling strategy corresponding to the target service type in the mapping relationship can be determined as the custom conflict handling strategy.

[0075] By introducing business types to obtain corresponding custom conflict handling strategies, differential conflict handling for different business types can be achieved, making data synchronization more accurate and flexible.

[0076] Optionally, it is possible to determine whether to perform custom conflict handling based on the business status of the conflict data. The conflict data may include first conflict data in the first business data and second conflict data in the second incremental data. That is, the business data corresponding to the data with a conflict in the first business data can be called the first conflict data, and the business data corresponding to the data with a conflict in the second incremental data can be called the second conflict data. Specifically, based on the custom conflict handling strategy, conflict handling of the conflict data may include:

[0077] Obtain the first business status of the first conflict data, the second business status of the second conflict data, and the preset status conversion information set by the target data provider for the second conflict data; the preset status conversion information represents the conversion relationship between multiple business statuses. For example, the multiple business statuses of order business data may be: order received status, order processing, order end (order completed, order cancelled), and the conversion relationship of these multiple business statuses is linearly sequential, that is, order received status → order processing → order end, and the business status can be converted in the manner of this arrow.

[0078] Optionally, the conversion relationship of multiple business statuses may also be circular, which is a cyclic relationship, and the present disclosure does not limit this.

[0079] Furthermore, if in the preset status conversion information, the second business status is after the first business status, then based on the custom conflict handling strategy, conflict handling of the conflict data can be performed. If the second business status is before the first business status, custom conflict handling may not be performed. Here, before and after may refer to the sequence in the conversion relationship. For example, if the second business status is order end and the first business status is order processing, then it can be considered that the second business status is after the first business status, and thus conflict handling of the conflict data can be performed based on the custom conflict handling strategy. By setting the business status, the accuracy of custom conflict handling can be improved, and processing resources can also be saved.

[0080] Optionally, the above step S207 may include: if the second business status is the business status at the end in the preset status conversion information, the first conflict data may be overwritten with the second conflict data. Taking the multiple business statuses of the above order business data "order received status → order processing → order end" as an example, order end can be regarded as the business status at the end in the preset status conversion information. Further, if it is order end, the first conflict data may be overwritten with the second conflict data.

[0081] As an alternative implementation, the service status can also be set in the custom conflict handling configuration information, so that the custom conflict handling configuration information can include the mapping relationship between the service type, the service status, and the first preset conflict handling strategy. In this way, based on the service type, the custom conflict handling strategy can be determined in combination with the service status. This application does not make any restrictions in this regard. For example, the service type and service status of the conflict data can be obtained, so that the first preset conflict handling strategy can be matched from the custom conflict handling configuration information as the custom conflict handling strategy.

[0082] In one example, taking the cloud data service system as an example, refer to Figure 4 , where DTS, data subscription, and the custom conflict handling application service (or referred to as the custom data processing module) can be deployed in the first data service system, which is to reflect the interactive operation logic in data synchronization. As Figure 4 shown, to implement the data synchronization function, the DTS data synchronization task can be configured in DTS first, and the data subscription task can be configured in the data subscription service to monitor the data changes in the second data service system. In addition, the service deployment needs to be started in the custom conflict handling application service to subscribe to the business data it is interested in through the subscription message in the data subscription. Thus, the data subscription can add the concerned business data to the data subscription task to achieve the monitoring of the concerned business data.

[0083] Based on the above configurations and deployments, data changes can be monitored through DTS and data subscription respectively. If the second data service system stores data in the form of database tables, whether the database tables have changed can be monitored. When the database table data in the second data service system changes, the change can be synchronized to DTS and the data subscription service. Correspondingly, DTS can directly synchronize the change to the database of the first data service system. When there are data conflicts, unified conflict strategy processing can be performed; the data subscription service can encapsulate the changed data into a message and push the message to the custom conflict handling application service. Since DTS has completed the change synchronization, when the custom conflict handling application service writes the changed data into the database of the first data service system, data conflicts generally occur. Then, based on business rules and custom conflict handling, for example, according to the first preset conflict handling strategy corresponding to the target business type of the conflict data and the service status, the conflict data can be processed. For example, in the final state (the terminal service status), the previous operations can be directly overwritten (such as Figure 4The data processing shown overwrites the previous operation), that is, the second conflicting data can be used to overwrite the first conflicting data, so as to achieve the correction and compensation of the custom conflict processing of the data provider on the basis of unified conflict processing. Specifically, for example, taking order data as an example, the business status of the order data can be indicated by the value of the type field in the order message. If the type value is order completed, order cancelled or order refunded, the data overwrite logic can be performed. For example, the order message can be structurally converted into an order object, so that the data in the first data service system can be updated through the update() method of the order access object to complete the overwrite strategy.

[0084] It should be noted that in the case of preset subscription data, the change synchronization of data subscription can be triggered based on detecting that the preset subscription data has changed.

[0085] In a possible implementation, referring to Figure 5 , two-way data processing can be performed, that is, two-way data synchronization, which can ensure the two-way smooth switching of traffic, and the data provider can implement custom conflict processing in both directions. Based on this, the method may further include:

[0086] In the case where data changes occur in the first data service system, the target incremental data of the first data service system can be sent to the second data service system through the second preset path and the second data subscription path respectively, so that the second data service system can perform data processing on the target incremental data based on the second preset conflict processing strategy and the custom conflict processing strategy corresponding to the data provider of the target incremental data; wherein, the second preset conflict processing strategy is the unified conflict processing strategy corresponding to the second data service system. Here, the data processing of the second data service system can be the same as that of the first data service system, as Figure 5 shown, which will not be elaborated here.

[0087] As Figure 5 shown, taking the first data service system and the second data service system as cloud service systems as an example, the first data service system and the second data service system can each include a DTS module, a data subscription module, a custom data conflict processing module, and a database (such as mysql). Among them, the custom data conflict processing module can be the business implementation of a kafka consumer, and it can select the library tables to be concerned about to subscribe to their changes, and perform custom conflict processing implementation on the data of the library tables subscribed to by the consumer.

[0088] Referring to Figure 5, both the first data service system and the second data service system can be synchronized by adopting a dual-channel method. Among them, the DTS channel can perform data synchronization based on the binlog-simulated replication protocol, with high synchronization efficiency, low latency, and a synchronization rate close to that in the mysql master-slave mode. For example, historical full-volume data and real-time incremental data can be obtained based on the DTS channel. The historical full-volume data can be stored in a file storage, and the real-time incremental data can be stored in a message queue. Further, the historical full-volume data and the real-time incremental data can be parsed, filtered, converted (data format conversion to match the data format in the database to be written), conflict processed, etc. Here, a unified conflict strategy is adopted, that is, the first preset conflict handling strategy or the second preset conflict handling strategy, that is to say, the preset conflict handling strategy corresponding to the database to be written is adopted.

[0089] The data subscription channel can synchronize the changes in the library table data of interest to Kafka for the requester to subscribe and consume, and can play a role in post-event compensation and repair through the data subscription channel. Since the latency of the data subscription channel is within a controllable range, the impact on the business scenario is usually acceptable; therefore, integrating the DTS channel and the data subscription channel to meet the library tables with different data synchronization requirements can balance the rate and custom requirements.

[0090] Among them, the custom data conflict handling module can include a consumer module and a custom data processing strategy module. As an example, the consumer module can obtain the real-time incremental data of the second data service system from the message queue through the data subscription channel, and can encapsulate the real-time incremental data into a message and push it to the custom data processing strategy module. Further, the message can be written into the database for data compensation of the custom conflict resolution strategy.

[0091] It should be noted that the custom data processing module of the first data service system can be deployed in the first data service system, and the custom data processing module of the second data service system can be deployed in the second data service system.

[0092] As an application example, the data provider can be a transportation service application. Based on this, the transportation service application, the vehicles under the transportation service application, the first data service system, and the second data service system can form a vehicle networking. Users can operate the vehicles based on the transportation service application. The data generated in the transportation service application can be stored in the first data service system or the second data service system to implement data access services through the first data service system and the second data service system. Based on this, the data generated in the transportation service can be synchronized between the first data service system and the second data service system to support smooth data switching. For example, it can be applicable to situations where one of the data service systems fails or the vehicle switches from the second data service system to the first data service system. For example, if the transportation service application H selects to store and provide data services through the first data service system and the second data service system, then H can set the data to be subscribed (i.e., the preset subscribed data) and set the custom conflict handling policy in the conflict policy setting interfaces or conflict policy setting interfaces provided by the first data service system and the second data service system respectively. The first data service system and the second data service system can store the preset subscribed data, the custom conflict handling policy, and the identifier of H in an associated manner.

[0093] In the case where a data change in the second data service system is detected based on the first preset path, the historical service data in the first data service system can be updated according to the first incremental data transmitted through the first preset path. If there is a data conflict, data conflict handling is performed according to the first preset conflict handling policy to obtain the first service data. Moreover, the preset subscribed data in the second data service system can be monitored based on the first data subscription path; in the case where the preset subscribed data changes, the second incremental data pushed based on the first data subscription path can be obtained; and the first service data can be updated according to the second incremental data. If there is a data conflict in this update, the custom conflict handling policy of H can be obtained, so that the conflict data can be conflict-handled based on the custom conflict handling policy to implement the custom conflict handling of H.

[0094] Furthermore, if vehicle 1 under H was originally under the second data service system and switches to the first data service system, and the first data service system provides data services, it can not only provide data services for vehicle 1 in a timely manner, provide seamless data services for the switching of vehicle 1, and make the switching smoother; but also provide data services for vehicle 1 after custom conflict handling, making the data services more accurate.

[0095] Figure 6 The block diagram of a data processing device provided according to an embodiment of the present application is shown. As Figure 6 shown, the device may include:

[0096] The first synchronization module 601 is configured to, when it is detected based on a first preset path that data changes occur in the second data service system, update the historical service data in the first data service system according to the first incremental data transmitted through the first preset path. If there is a data conflict, data conflict handling is performed according to a first preset conflict handling policy to obtain first service data; the first preset conflict handling policy is the unified conflict handling policy corresponding to the first data service system.

[0097] The second synchronization module 603 is configured to, when it is detected based on a first data subscription path that data changes occur in the second data service system, update the first service data according to the second incremental data pushed through the first data subscription path. If there is a data conflict, determine the target data provider of the conflict data.

[0098] The conflict handling policy acquisition module 605 is configured to acquire the custom conflict handling policy corresponding to the target data provider.

[0099] The conflict handling module 607 is configured to perform conflict handling on the conflict data based on the custom conflict handling policy.

[0100] Wherein, the synchronization speed of the first preset path is higher than the synchronization speed of the first data subscription path.

[0101] By monitoring data changes in the second data service system through two paths, and the synchronization speed of the first preset path is higher than that of the first data subscription path, the data processing of the first data service system can not only ensure the synchronization rate based on the first preset path, providing guarantee for the smooth traffic switching of the data provider; but also meet the custom processing requirements of the data provider based on the first data subscription path, enabling the data synchronization of different data providers to be based on the custom conflict handling of different data providers, realizing refined and fine-grained data synchronization, so as to meet the dual requirements of the data synchronization efficiency and custom processing of the data provider.

[0102] Moreover, the synchronization speed of the first data subscription path is relatively low, enabling the custom conflict handling to be performed after the unified conflict handling, so that the custom conflict handling of the data provider can cover the unified conflict handling, reasonably meeting the custom conflict handling requirements of the data provider.

[0103] In addition, this dual-path data synchronization method ensures that the custom conflict handling policy of the data provider does not cause code intrusion into the unified conflict handling policy of the data service system, guaranteeing the security of the data service system and the unity of data management.

[0104] In a possible implementation manner, the above conflict handling policy acquisition module 605 may include:

[0105] A custom configuration acquisition unit for acquiring custom conflict handling configuration information corresponding to the target data provider; the custom conflict handling configuration information includes a mapping relationship between a service type and a first preset conflict handling policy.

[0106] A service type determination unit for determining the target service type of the conflict data.

[0107] A custom conflict handling policy determination unit for determining, according to the mapping relationship, the first preset conflict handling policy corresponding to the target service type as the custom conflict handling policy.

[0108] In a possible implementation manner, the conflict data includes first conflict data in the first service data and second conflict data in the second incremental data; the above conflict handling module 607 may include:

[0109] A service status acquisition unit for acquiring the first service status of the first conflict data, the second service status of the second conflict data, and preset status conversion information set by the target data provider for the second conflict data; the preset status conversion information represents the conversion relationship between multiple service statuses.

[0110] A first conflict handling unit for, if in the preset status conversion information, the second service status is after the first service status, performing conflict handling on the conflict data based on the custom conflict handling policy.

[0111] In a possible implementation manner, the above conflict handling module 607 may include:

[0112] A second conflict handling unit for, if the second service status is the service status at the end in the preset status conversion information, using the second conflict data to overwrite the first conflict data.

[0113] In a possible implementation manner, the device may further include:

[0114] A configuration request module for, in response to a conflict handling configuration request from the target data provider, extracting a mapping relationship between a service type to be configured and a preset conflict handling policy to be configured from the conflict handling configuration request.

[0115] A custom configuration module for updating the custom conflict handling configuration information based on the mapping relationship between the service type to be configured and the preset conflict handling policy to be configured.

[0116] In a possible implementation manner, the above second synchronization module 603 may include:

[0117] A monitoring unit, configured to monitor preset subscribed data in the second data service system based on the first data subscription path; the preset subscribed data is data subscribed in advance by at least one data provider;

[0118] A second incremental data acquisition unit, configured to acquire second incremental data pushed based on the first data subscription path when the preset subscribed data changes;

[0119] A data synchronization unit, configured to update the first service data according to the second incremental data.

[0120] In a possible implementation, the apparatus may further include:

[0121] A bidirectional synchronization module, configured to, when data in the first data service system changes, send target incremental data of the first data service system to the second data service system through a second preset path and a second data subscription path respectively, so that the second data service system processes the target incremental data based on a second preset conflict handling policy and a custom conflict handling policy corresponding to the data provider of the target incremental data;

[0122] Wherein, the second preset conflict handling policy is a unified conflict handling policy corresponding to the second data service system.

[0123] In a possible implementation, the apparatus may further include:

[0124] A traffic information acquisition module, configured to acquire traffic information of the first data service system;

[0125] Correspondingly, the above conflict handling module 607 may include:

[0126] A third conflict handling unit, configured to handle conflicts of the conflict data based on the custom conflict handling policy when the traffic information is lower than a traffic threshold.

[0127] In a possible implementation, the apparatus may further include:

[0128] A target state acquisition module, configured to acquire a target state of the custom conflict handling policy;

[0129] Correspondingly, the above conflict handling module 607 may include:

[0130] A fourth conflict handling unit, configured to handle conflicts of the conflict data based on the custom conflict handling policy when the target state is an enabled state.

[0131] In a possible implementation, the apparatus may further include:

[0132] A status update response module, configured to extract status update indication information from the status update request in response to a status update request from the target data provider;

[0133] A status update module, configured to update the status of the custom conflict handling policy based on the status update indication information.

[0134] In a possible implementation, the synchronization speed of the first preset path is higher than the synchronization speed of the first data subscription path.

[0135] Regarding the apparatus in the above embodiments, the specific manners in which each module and unit perform operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0136] Figure 7 FIG. shows a block diagram of an electronic device for data processing according to an embodiment of the present application. The electronic device may be a server, and its internal structure diagram may be as Figure 7 shown. The electronic device includes a processor 701, a memory, and a network interface 704 connected through a system bus 702. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory 703. The non-volatile storage medium stores an operating system 705 and a computer program 706. The internal memory 703 provides an environment for the operation of the operating system 705 and the computer program 706 in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0137] Those skilled in the art can understand that Figure 7 the structure shown in is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In an exemplary embodiment, an electronic device is further provided, including: a processor; a memory for storing executable instructions of the processor; wherein, the processor is configured to execute the instructions to implement the data processing method in the embodiments of the present application.

[0139] In an exemplary embodiment, a storage medium is further provided. When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the data processing method in the embodiments of the present application.

[0140] In an exemplary embodiment, a computer program product including instructions is also provided. When it runs on a computer, it causes the computer to execute the data processing method in the embodiments of the present application.

[0141] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0142] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0143] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to the first data service system; the method includes: When it is detected based on the first preset path that the second data service system has data changes, update the historical service data in the first data service system according to the first incremental data transmitted through the first preset path. If there is a data conflict, perform data conflict handling according to the first preset conflict handling policy to obtain the first service data; the first preset conflict handling policy is the unified conflict handling policy corresponding to the first data service system; the first preset path is a path based on data transmission services. Monitor the preset subscription data in the second data service system based on the first data subscription path; the preset subscription data is data subscribed in advance by at least one data provider; when the preset subscription data changes, obtain the second incremental data pushed through the first data subscription path; and update the first service data according to the second incremental data. If there is a data conflict, determine the target data provider of the conflict data. Obtain the custom conflict handling policy corresponding to the target data provider. Based on the custom conflict handling policy, perform conflict handling on the conflict data.

2. The method according to claim 1, characterized in that, The obtaining the custom conflict handling policy corresponding to the target data provider includes: Obtain the custom conflict handling configuration information corresponding to the target data provider; the custom conflict handling configuration information includes the mapping relationship between the service type and the first preset conflict handling policy. Determine the target service type of the conflict data. According to the mapping relationship, determine the first preset conflict handling policy corresponding to the target service type as the custom conflict handling policy.

3. The method according to claim 1 or 2, characterized in that, The conflict data includes the first conflict data in the first service data and the second conflict data in the second incremental data; the performing conflict handling on the conflict data based on the custom conflict handling policy includes: Obtain the first service state of the first conflict data, the second service state of the second conflict data, and the preset state conversion information set by the target data provider for the second conflict data; the preset state conversion information represents the conversion relationship between multiple service states. If in the preset state conversion information, the second service state is after the first service state, perform conflict handling on the conflict data based on the custom conflict handling policy.

4. The method according to claim 3, wherein The performing conflict handling on the conflict data based on the custom conflict handling policy includes: If the second service state is the service state at the end in the preset state conversion information, use the second conflict data to overwrite the first conflict data.

5. The method according to claim 2 or 4, characterized in that, The method further includes: In response to the conflict handling configuration request of the target data provider, extract the mapping relationship between the service type to be configured and the preset conflict handling policy to be configured from the conflict handling configuration request. Based on the mapping relationship between the service type to be configured and the preset conflict handling policy to be configured, update the custom conflict handling configuration information.

6. The method according to claim 4, wherein The method further includes: In the case of data changes occurring in the first data service system, the target incremental data of the first data service system is sent to the second data service system through a second preset path and a second data subscription path respectively, so that the second data service system performs data synchronization processing on the target incremental data based on a second preset conflict handling policy and a custom conflict handling policy corresponding to the data provider of the target incremental data; Wherein, the second preset conflict handling policy is a unified conflict handling policy corresponding to the second data service system.

7. The method according to claim 1 or 2, characterized in that, The method further includes: Obtaining traffic information of the first data service system; Based on the custom conflict handling policy, performing conflict handling on the conflict data, including: In the case where the traffic information is lower than a traffic threshold, performing conflict handling on the conflict data based on the custom conflict handling policy.

8. The method according to claim 1 or 2, characterized in that, The method further includes: Obtaining a target state of the custom conflict handling policy; Based on the custom conflict handling policy, performing conflict handling on the conflict data, including: In the case where the target state is an enabled state, performing conflict handling on the conflict data based on the custom conflict handling policy.

9. The method according to claim 8, wherein The method further includes: Responding to a status update request of the target data provider, and extracting status update indication information from the status update request; Updating the status of the custom conflict handling policy based on the status update indication information.

10. The method according to claim 1, characterized in that, The synchronization speed of the first preset path is higher than the synchronization speed of the first data subscription path.

11. A data processing device, characterized in that, Applied to a first data service system; the device includes: A first synchronization module, configured to, in the case of monitoring data changes occurring in a second data service system based on a first preset path, update historical service data in the first data service system according to first incremental data transmitted through the first preset path, and if there is a data conflict, perform data conflict handling according to a first preset conflict handling policy to obtain first service data; the first preset conflict handling policy is a unified conflict handling policy corresponding to the first data service system; the first preset path is a path based on a data transmission service; A second synchronization module, configured to, in the case of monitoring data changes occurring in the second data service system based on a first data subscription path, update the first service data according to second incremental data pushed through the first data subscription path, and if there is a data conflict, determine a target data provider of the conflict data; A conflict handling policy acquisition module, configured to acquire a custom conflict handling policy corresponding to the target data provider; A conflict handling module, configured to perform conflict handling on the conflict data based on the custom conflict handling policy; Wherein, the second synchronization module includes: A monitoring unit, configured to monitor preset subscription data in the second data service system based on the first data subscription path; the preset subscription data is data subscribed in advance by at least one data provider; A second incremental data acquisition unit, configured to acquire second incremental data pushed based on the first data subscription path when the preset subscribed data changes; A data synchronization unit, configured to update the first service data according to the second incremental data.

12. The device according to claim 11, wherein The conflict handling policy acquisition module includes: A custom configuration acquisition unit, configured to acquire custom conflict handling configuration information corresponding to the target data provider; the custom conflict handling configuration information includes a mapping relationship between a service type and a first preset conflict handling policy; A service type determination unit, configured to determine a target service type of the conflict data; A custom conflict handling policy determination unit, configured to determine, according to the mapping relationship, a first preset conflict handling policy corresponding to the target service type as the custom conflict handling policy.

13. The device according to claim 11 or 12, characterized in that, The conflict data includes first conflict data in the first service data and second conflict data in the second incremental data; the conflict handling module includes: A service state acquisition unit, configured to acquire a first service state of the first conflict data, a second service state of the second conflict data, and preset state transition information set by the target data provider for the second conflict data; the preset state transition information represents a transition relationship between multiple service states; A first conflict handling unit, configured to, if in the preset state transition information, the second service state is after the first service state, perform conflict handling on the conflict data based on the custom conflict handling policy.

14. The device according to claim 13, wherein The conflict handling module includes: A second conflict handling unit, configured to, if the second service state is a service state at the end in the preset state transition information, use the second conflict data to overwrite the first conflict data.

15. The device according to claim 12 or 14, characterized in that, The apparatus further includes: A configuration request module, configured to, in response to a conflict handling configuration request of the target data provider, extract a mapping relationship between a service type to be configured and a preset conflict handling policy to be configured from the conflict handling configuration request; A custom configuration module, configured to update the custom conflict handling configuration information based on the mapping relationship between the service type to be configured and the preset conflict handling policy to be configured.

16. The device according to claim 14, characterized in that, The apparatus further includes: A bidirectional synchronization module, configured to, when data in the first data service system changes, send target incremental data of the first data service system to the second data service system through a second preset path and a second data subscription path respectively, so that the second data service system performs data processing on the target incremental data based on a second preset conflict handling policy and a custom conflict handling policy corresponding to the data provider of the target incremental data; wherein the second preset conflict handling policy is a unified conflict handling policy corresponding to the second data service system.

17. The device according to claim 11 or 12, characterized in that The apparatus further includes: A traffic information acquisition module, configured to acquire traffic information of the first data service system; Correspondingly, the conflict handling module includes: A third conflict handling unit, configured to handle the conflict data based on the custom conflict handling policy when the traffic information is lower than the traffic threshold.

18. The device according to claim 11 or 12, characterized in that, The apparatus further includes: A target status obtaining module, configured to obtain the target status of the custom conflict handling policy; Correspondingly, the conflict handling module includes: A fourth conflict handling unit, configured to handle the conflict data based on the custom conflict handling policy when the target status is in an enabled state.

19. The apparatus according to claim 18, wherein The apparatus further includes: A status update response module, configured to extract status update indication information from the status update request in response to a status update request from the target data provider; A status update module, configured to update the status of the custom conflict handling policy based on the status update indication information.

20. The apparatus according to claim 11, wherein The synchronization speed of the first preset path is higher than the synchronization speed of the first data subscription path.

21. An electronic device, characterized in that, Comprising: A processor; A memory for storing processor-executable instructions; Wherein, the processor is configured to execute the executable instructions to implement the method according to any one of claims 1 to 10.

22. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions, when executed by the processor, implement the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that, Including computer instructions, when the computer instructions are executed by the processor, the computer is caused to execute the method according to any one of claims 1 to 10.

Images