Data synchronization method, apparatus, device, and storage medium

Abstract

The application provides a data synchronization method and device, equipment and a storage medium. The method comprises the following steps: receiving a data write request sent by at least one data provider, wherein the data write request comprises a data provider identifier and at least one piece of to-be-written data; determining a target database group corresponding to each data provider according to each data provider identifier, wherein each database group comprises a relational database and a non-relational database; and writing each to-be-written data into the corresponding relational database and non-relational database. The scheme of the application can efficiently synchronize a large amount of concurrent data.

Description

Technical Field

[0001] This application relates to communication technology, and more particularly to a data synchronization method, apparatus, device, and storage medium. Background Technology

[0002] Producers generate data for their own use or for consumers' use. Producers and consumers exchange data through data synchronization.

[0003] Currently, producers can directly expose their databases or data access interfaces to consumers, or they can register their data on a data exchange platform for consumers to subscribe to, thereby achieving data synchronization between producers and consumers. However, exposing the database or data access interface to consumers can impact database efficiency when there are a large number of concurrent accesses. On the other hand, data synchronization through a data exchange platform is too cumbersome when producers and consumers belong to the same system, involving registration, subscription, and retransmission, resulting in low data synchronization efficiency.

[0004] In summary, existing data synchronization methods cannot efficiently synchronize large amounts of concurrent data. Summary of the Invention

[0005] This application provides a data synchronization method, apparatus, device, and storage medium to solve the problem that existing technologies cannot efficiently synchronize large amounts of concurrent data.

[0006] According to a first aspect of this application, a data synchronization method is provided, comprising:

[0007] Receive at least one data provider sending a data write request, the data write request including a data provider identifier and at least one piece of data to be written;

[0008] The target database group corresponding to each data provider is determined based on the identifier of each data provider. Each database group includes one relational database and one non-relational database.

[0009] Write each piece of data to be written to the corresponding relational database and non-relational database.

[0010] As an optional implementation, writing each piece of data to be written into the corresponding relational database and non-relational database includes: determining the data sequence number of each piece of data to be written, wherein the data sequence number of each piece of data to be written is sequentially incremented; using the data sequence number of each piece of data to be written as an identifier, writing each piece of data to be written into the corresponding relational database and non-relational database.

[0011] As an optional implementation, determining the data sequence number of each piece of data to be written includes:

[0012] Obtain the maximum identifier number in each data record of each database group, and determine the maximum identifier number as the cache sequence number; for each data write request, perform the following operations: determine the data sequence number of each data to be written in the data write request based on the cache sequence number, wherein the data sequence numbers of each data to be written in the data write request are sequentially increased and are all greater than the cache sequence number; update the cache sequence number to the maximum item among the data sequence numbers of each data to be written in the data write request.

[0013] As an optional implementation, the data synchronization method further includes: receiving a data query request sent by at least one data consumer, wherein the data query request includes a query interface type, query authorization information, and identification information of at least one piece of data to be queried; determining the type of each data query request according to each query interface type; determining at least one authorized database group corresponding to each data query request according to each query authorization information; and querying each piece of data to be queried in each authorized database group corresponding to each data query request according to the identification information of the data to be queried, based on the type of each data query request.

[0014] As an optional implementation, the data query request types include complex queries and simple queries; the step of querying each data to be queried in each authorized database group corresponding to each data query request according to the identification information of the data to be queried includes: if the data query request type is determined to be a simple query, then querying each data to be queried in the non-relational database of each authorized database group corresponding to the data query request according to the identification information of the data to be queried; if the data query request type is determined to be a complex query, then querying each data to be queried in the relational database of each authorized database group corresponding to the data query request according to the identification information of the data to be queried.

[0015] As an optional implementation, the data query request type includes a comprehensive query; the identification information of the data to be queried includes the identification information of at least one item to be queried; the step of querying each data to be queried in each authorized database group corresponding to each data query request according to the identification information of the data to be queried according to the type of each data query request includes: if it is determined that the type of the data query request is a comprehensive query, then querying in the non-relational databases of each authorized database group corresponding to the data query request according to the identification information of each item to be queried, and determining the query result of each item to be queried; the query result of each item to be queried is either a result or no result; if it is determined that the result of any item to be queried is no result, then querying the item to be queried with no result in the relational databases of each authorized database group corresponding to the data query request according to the corresponding identification information.

[0016] As an optional implementation method, the data synchronization method further includes: obtaining the check codes of each relational data record in each relational database and the check codes of each non-relational data record in each non-relational database; and using each relational data to verify and correct each non-relational data.

[0017] According to a second aspect of this application, a data synchronization apparatus is provided, comprising:

[0018] The receiving module is configured to receive a data write request sent by at least one data provider, wherein the data write request includes a data provider identifier and at least one piece of data to be written;

[0019] The determination module is used to determine the target database group corresponding to each data provider based on the identifier of each data provider. Each database group includes a relational database and a non-relational database.

[0020] The write module is used to write the data to be written to the corresponding relational database and non-relational database.

[0021] According to a third aspect of this application, an electronic device is provided, comprising: a memory, a processor, and a transceiver;

[0022] The memory, the processor, and the transceiver circuit are interconnected;

[0023] The memory stores computer-executed instructions;

[0024] The transceiver is used to send and receive data;

[0025] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0026] According to a fourth aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, are used to implement the method as described in the first aspect.

[0027] The data synchronization method, apparatus, device, and storage medium provided in this application involve receiving a data write request from at least one data provider, the data write request including a data provider identifier and at least one piece of data to be written; determining the target database group corresponding to each data provider based on the data provider identifier, each database group including a relational database and a non-relational database; and writing the data to be written into the corresponding relational database and non-relational database. Since each data provider corresponds to a different target database group, after determining the target database group corresponding to each data provider, each database group can write its corresponding data to be written into its relational database and non-relational database. When each database group writes data to the database, it only consumes its own resources and does not affect other database groups. Data synchronization includes data writing, data deletion, data modification, and data querying. Therefore, the solution of this application can efficiently synchronize a large amount of concurrent data. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0029] Figure 1A This is a network architecture diagram corresponding to an application scenario provided in the embodiments of this application;

[0030] Figure 1B This is a network architecture diagram corresponding to another application scenario provided in the embodiments of this application;

[0031] Figure 2 This is a flowchart illustrating the data synchronization method provided according to Embodiment 1 of this application;

[0032] Figure 3 This is a flowchart illustrating the data synchronization method provided in Embodiment 2 of this application;

[0033] Figure 4 This is a flowchart illustrating the data synchronization method provided in Embodiment 3 of this application;

[0034] Figure 5 This is a flowchart illustrating the data synchronization method provided in Embodiment 4 of this application;

[0035] Figure 6 This is a schematic diagram of the data synchronization device provided according to Embodiment 5 of this application;

[0036] Figure 7 This is a schematic diagram of the structure of an electronic device provided according to Embodiment Six of this application.

[0037] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0038] The prior art involved in this application will be described in detail and analyzed below.

[0039] With the development of technologies such as the Internet, mobile Internet, and the Internet of Things, more and more data is being generated. Producers not only use the data they generate themselves, but also need to provide it to consumers. Therefore, producers and consumers need to exchange data through data synchronization.

[0040] Currently, there are three ways to achieve data synchronization between producers and consumers.

[0041] In the first approach, producers store data directly in their own database after generating it, while simultaneously opening the database to consumers. Consumers can directly access the producer's database to achieve data synchronization. However, in this approach, producers and consumers share the database. When consumers acquire large amounts of data or producers write large amounts of data, it consumes significant hardware resources, slowing down the database and affecting both consumers and producers, making it impossible to efficiently synchronize large amounts of concurrent data.

[0042] The second approach involves producers providing data access interfaces to consumers. However, this approach doesn't allow for pre-determined information such as consumer type and number when setting up the interface. When new consumers emerge or have new data access needs, new data access interfaces must be developed for them. These new interfaces not only consume significant resources during development but also may not be compatible with the database after completion. Furthermore, a large number of calls to these interfaces consumes substantial hardware resources, impacting database performance. Therefore, providing open data access interfaces is also inefficient for synchronizing large amounts of concurrent data.

[0043] The third approach involves data producers registering on a data exchange platform and synchronizing their data there. Simultaneously, consumers subscribe to the platform, which then handles data synchronization between producers and consumers. Because the data exchange platform isolates producers and consumers, it avoids mutual interference. However, this approach requires a very large data exchange platform, consuming significant resources, and the data synchronization mechanism, relying on registration and subscription, lacks real-time performance and flexibility, making it inefficient for synchronizing large amounts of concurrent data.

[0044] In summary, existing technologies cannot efficiently synchronize large amounts of concurrent data.

[0045] Therefore, when faced with the problems in existing technologies, the inventors, through creative research, realized that to efficiently synchronize large amounts of concurrent data, it is necessary to avoid the excessive resource consumption of databases during the synchronization of large amounts of concurrent data, while simultaneously improving the synchronization efficiency of each piece of data. Thus, the advantages of high read / write speed and high synchronization efficiency of non-relational databases, combined with the stability and error-free nature of relational databases, can be combined to form database groups that integrate relational and non-relational databases. The same data can be stored simultaneously in both relational and non-relational databases within these groups. Furthermore, the number of database groups can be expanded, and the groups do not interfere with each other. Therefore, efficient synchronization of large amounts of concurrent data can be achieved through multiple database groups containing both non-relational and relational databases.

[0046] In summary, the inventors propose the following technical solution: receiving a data write request from at least one data provider, the data write request including a data provider identifier and at least one piece of data to be written; determining the target database group corresponding to each data provider based on the data provider identifier, each database group including a relational database and a non-relational database; and writing each piece of data to be written into the corresponding relational database and non-relational database. Since each data provider corresponds to a different target database group, after determining the target database group corresponding to each data provider, each database group can write its corresponding data to be written into its relational database and non-relational database. When each database group writes data to the database, it only occupies its own resources and does not affect other database groups. Data synchronization includes data writing, data deletion, data modification, and data querying. Therefore, the solution of this application can efficiently synchronize a large amount of concurrent data.

[0047] The data synchronization method, apparatus, device, and storage medium provided in this application aim to solve the above-mentioned technical problems of the prior art. The technical solutions of this application and how they solve the aforementioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0048] The network architecture and application scenarios of the data synchronization method provided in the embodiments of this application will be described below. When the following description refers to the accompanying drawings, unless otherwise indicated, the same data in different drawings represent the same or similar elements.

[0049] Figure 1A This is a network architecture diagram corresponding to an application scenario provided in the embodiments of this application. Figure 1BThis is a network architecture diagram corresponding to another application scenario provided in the embodiments of this application.

[0050] like Figure 1A As shown in the embodiment of this application, a network architecture corresponding to an application scenario includes: an electronic device 11, at least one data provider 12, and at least one database group 13. The database group 13 includes a relational database 131 and a non-relational database 132. The electronic device 11 is communicatively connected to each database group 13.

[0051] The data provider 12 is communicatively connected to the electronic device 11. The data provider 12 can send a data write request to the electronic device 11. The data write request includes the data provider identifier and at least one piece of data to be written.

[0052] Electronic device 11 receives a data write request sent by at least one data provider 12, determines the target database group 13 corresponding to each data provider 12 according to the identifier of each data provider 12, and writes each piece of data to be written into the corresponding relational database 131 and non-relational database 132.

[0053] like Figure 1B As shown, in Figure 1A Based on the above, the network architecture diagram corresponding to another application scenario provided in this application embodiment also includes at least one data consumer terminal 14. The data consumer terminal 14 is communicatively connected to the electronic device 11, and the data consumer terminal 14 can send a data query request to the electronic device 11. The data query request includes the query interface type, query authorization information, and identification information of at least one piece of data to be queried.

[0054] Electronic device 11 receives data query requests sent by at least one data consumer terminal 14, determines the type of each data query request based on the query interface type, determines at least one authorized database group corresponding to each data query request based on the query authorization information, and queries each piece of data to be queried in the corresponding authorized database group according to the identification information of the data to be queried, thereby obtaining the query results for each data query request. Furthermore, electronic device 11 can also send the query results of each data query request to the corresponding data consumer terminal 14.

[0055] The embodiments of this application will now be described with reference to the accompanying drawings. The embodiments described below do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0056] Example 1

[0057] Figure 2This is a flowchart illustrating the data synchronization method provided in Embodiment 1 of this application. Figure 2 As shown, the executing entity of this application is a data synchronization device, which is located in an electronic device. The data synchronization method provided in this embodiment includes steps 201 to 203.

[0058] Step 201: Receive a data write request from at least one data provider. The data write request includes a data provider identifier and at least one copy of data to be written.

[0059] In this embodiment, the data provider can be a data producer device or a device that needs to store data in the database group.

[0060] Understandably, a data write request can include one or more pieces of data to be written. The data to be written can be structured or unstructured.

[0061] The data provider identifier can be the IP address of the data provider or a user ID, etc.

[0062] Step 202: Determine the target database group corresponding to each data provider based on the identifier of each data provider. Each database group includes one relational database and one non-relational database.

[0063] In this embodiment, the electronic device can pre-store the mapping relationship between each data provider identifier and each database group, and can determine the target database group corresponding to each data provider based on the mapping relationship.

[0064] The number of database groups can be one or more, and the number of database groups can be adjusted according to the needs of the data producer. For example, when the amount of data to be written on the data producer increases, an additional database group can be added. When deleting a large amount of data on the data producer, idle database groups can be shut down.

[0065] Each database group includes one relational database and one non-relational database. The relational database stores data using a two-dimensional table structure. The non-relational database stores data using key-value pairs. Database groups can be distributed or clustered, and are managed by electronic devices.

[0066] Step 203: Write each piece of data to be written into the corresponding relational database and non-relational database.

[0067] In this embodiment, the electronic device can call the data write interfaces of the relational database and the non-relational database in each database group to store a copy of the data to be written as a data record in the relational database and a data record in the non-relational database. Furthermore, after a successful write, both the relational database and the non-relational database's data write interfaces can return a write success message to the electronic device.

[0068] The identifier of the data record after the same data is written to a relational database is the same as the identifier of the data record after being written to a non-relational database.

[0069] When the data to be written is represented by a two-dimensional table structure, it can be directly written to a relational database. Alternatively, the data in each row and column of the two-dimensional table structure can be traversed, converted into key-value pairs, and then written to a non-relational database. The identifier of the data to be written can be used as the primary key of the data record in the relational database.

[0070] The data synchronization method provided in this embodiment involves receiving a data write request from at least one data provider, the data write request including a data provider identifier and at least one piece of data to be written; determining the target database group corresponding to each data provider based on the data provider identifier, each database group including a relational database and a non-relational database; and writing each piece of data to be written into the corresponding relational database and non-relational database. Since each data provider corresponds to a different target database group, after determining the target database group corresponding to each data provider, each database group can write its corresponding data to be written into its relational database and non-relational database. When each database group writes data to the database, it only occupies its own resources and does not affect other database groups. Data synchronization includes data writing, data deletion, data modification, and data querying. Therefore, the solution of this application can efficiently synchronize a large amount of concurrent data.

[0071] As an optional implementation, based on Embodiment 1, steps 1101 to 1102 are added after step 203.

[0072] Step 1101: Determine at least one corresponding data consumer to be notified based on each data write request.

[0073] In this embodiment, the electronic device may pre-store the mapping relationship between each database group and its corresponding data consumer. After new data is added to a database group, the electronic device needs to promptly notify the data consumer corresponding to that database group. The data consumer corresponding to the database group can be a data consumer with permission to access that database group.

[0074] After receiving each data write request and determining the target database group corresponding to each write request, the electronic device can determine the data consumer corresponding to the target database group as the data consumer to be notified based on the mapping relationship between each database group and its corresponding data consumer.

[0075] Step 1102: Send a data addition message to each data consumer to be notified. If it is determined that the data addition message failed to be sent, the data addition message is put into the addition message queue, and the data addition message is sent to the data consumer to be notified again after a preset time interval, until the number of times the data addition message is sent to the data consumer to be notified reaches the preset number or it is determined that the data addition message was sent successfully. The addition message queue is used to temporarily store the data addition message.

[0076] In this embodiment, the electronic device is communicatively connected to the data consumer. After the electronic device receives a write success message from the relational database, it can send a data addition message to the data consumer corresponding to each target database. The data addition message is used to notify the data consumer that a new data record has been added to its corresponding authorized database group, and to instruct the data consumer to send a notification message to the electronic device.

[0077] Within a preset time period after the electronic device sends a new data message to the data consumer, if a notification message is received from the data consumer, the new data message is considered successfully sent; otherwise, the new data message is considered to have failed to be sent.

[0078] Here, the data consumer might fail to receive the new data message due to excessive load or poor network conditions. Therefore, the electronic device can push the new data message to the data consumer multiple times until a preset number of pushes is reached or the push is confirmed to be successful. This ensures the data consumer receives the new data message promptly and can quickly access the newly added data records, achieving efficient and timely data synchronization between the data provider and consumer. The preset number of pushes can be 5.

[0079] Example 2

[0080] Figure 3 This is a flowchart illustrating the data synchronization method provided in Embodiment 2 of this application. Figure 3 As shown, this embodiment refines step 203 based on embodiment one, and step 203 refinement includes steps 301 to 302.

[0081] Step 301: Determine the data sequence number of each piece of data to be written, with the data sequence number of each piece of data to be written increasing sequentially.

[0082] In this embodiment, the electronic device can determine the data sequence number of each piece of data to be written based on the order in which the write requests are received. Furthermore, upon receiving a new write request, it can increment the previously determined data sequence number to determine the data sequence number of each piece of data to be written in the current write request.

[0083] Step 302: Using the data sequence number of each data to be written as an identifier, write each data to be written into the corresponding relational database and non-relational database.

[0084] In this embodiment, the identifier is used to distinguish data records in each database group. Since the data sequence numbers of the data to be written are sequentially increasing and each is unique, the electronic device can use the sequence number of each data to be written as the identifier to write each data record to the corresponding relational database and non-relational database. This avoids situations where the relational database and non-relational database generate their own identifiers when writing data to their respective databases, thus preventing data identifier conflicts and confusion between database groups.

[0085] The data synchronization method provided in this embodiment determines the data sequence number of each piece of data to be written, with the data sequence numbers of each piece of data incrementing sequentially. Using the data sequence number of each piece of data as an identifier, the data to be written is then written to the corresponding relational database and non-relational database. Since the data sequence numbers of each piece of data to be written increment sequentially and are all different, using the data sequence number as an identifier to write the data to the corresponding relational database and non-relational database avoids data identifier conflicts and confusion between database groups.

[0086] As an optional implementation, based on Embodiment 2, the step 301 of "determining the data sequence number of each data to be written" is further refined, which includes steps 401 to 403.

[0087] Step 401: Obtain the maximum identifier number in each data record of each database group, and determine the maximum identifier number as the cache sequence number.

[0088] In this embodiment, the electronic device includes non-volatile memory, and the cached serial number is stored in the non-volatile memory. The cached serial number stored in the non-volatile memory will not be lost after the electronic device is powered off.

[0089] In this embodiment, the maximum identifier is the maximum value among all identifiers in all datasets across all database groups. Since the identifiers of each data record are different in each relational database and each data record in each non-relational database, the identifiers of each data record in each database group can be sorted to obtain the maximum identifier. This maximum identifier is then used as the cache sequence number and stored in non-volatile memory.

[0090] Step 402, for each data write request, perform the following operations:

[0091] The data sequence number of each piece of data to be written in the data write request is determined based on the cache sequence number. The data sequence numbers of each piece of data to be written in the data write request are sequentially increased and are all greater than the cache sequence number.

[0092] Update the cache sequence number to the largest item among the data sequence numbers of each data to be written in this write request.

[0093] In this embodiment, for any data write request, the data sequence number of each data to be written can be determined by adding a preset positive number to the cache sequence number in sequence, based on the number of data to be written in the data write request.

[0094] After writing each piece of data to be written to its corresponding database group using its data sequence number as an identifier, the largest identifier among all data records in each database group changes to the largest data sequence number in that write request. Therefore, the cache sequence number can be updated to the largest item among the data sequence numbers of each piece of data to be written in that write request, making it easier to determine the data sequence number of each piece of data to be written in subsequent write requests.

[0095] Repeat step 402 until all the data to be written in each data write request received by the electronic device is written to the corresponding database group.

[0096] As an example of this embodiment, the cache sequence number is 100, with a preset positive value of 1. When the data write request includes 5 pieces of data to be written, the values ​​obtained by incrementing the cache sequence number 100 by 1 sequentially are determined as the data sequence numbers of each piece of data to be written, which are 101, 102, 103, 104, and 105, respectively. Using the data sequence numbers 101, 102, 103, 104, and 105 as identifiers, these 5 pieces of data to be written are written to the relational database and non-relational database corresponding to the data write request. The cache sequence number is then updated to the largest item, 105, among the data sequence numbers of these 5 pieces of data to be written.

[0097] As an optional implementation, in this embodiment, after determining the data sequence number of each piece of data to be written, the electronic device can call the respective data write interfaces of the relational database and non-relational database corresponding to the data write request, and store the data to be written into the corresponding relational database and non-relational database. Furthermore, after receiving a write success message from the relational database, the cache sequence number can be updated to the largest item among the data sequence numbers of each piece of data to be written.

[0098] The data synchronization method provided in this embodiment obtains the maximum identifier number in each data record of each database group and determines the maximum identifier number as the cache sequence number. For each data write request, the following operations are performed: the data sequence number of each data to be written in the data write request is determined according to the cache sequence number, wherein the data sequence numbers of each data to be written in the write request are sequentially incremented and all are greater than the cache sequence number; the cache sequence number is updated to the maximum item among the data sequence numbers of each data to be written in the write request. Since the data sequence number of each data to be written is determined based on the cache sequence number, and the cache sequence number is stored in non-volatile memory and is not lost when power is off, it can ensure that the data sequence numbers of each data to be written are different, avoiding conflicts between identifier numbers between database groups. At the same time, after the electronic device is powered off and restarted, the latest cache sequence number before the power outage can be quickly obtained.

[0099] Example 3

[0100] Figure 4 This is a flowchart illustrating the data synchronization method provided in Embodiment 3 of this application. Figure 4 As shown, the data synchronization method provided in this embodiment, based on any of the above embodiments, further includes steps 501 to 504.

[0101] Step 501: Receive at least one data query request sent by a data consumer. The data query request includes the query interface type, query authorization information, and identification information of at least one piece of data to be queried.

[0102] In this embodiment, the data consumer can be a data consumer device or a device that needs to read data from the database group. The data consumer can also be the same device as the data provider described above.

[0103] Step 502: Determine the type of each data query request based on the type of each query interface.

[0104] In this embodiment, the query interface types include Structured Query Language (SQL) and Hypertext Transfer Protocol (HTTP).

[0105] When the query interface type is HTTP, the data format can be a JavaScript object notation or a JSON document format. The HTTP protocol is flexible enough and has good network penetration, which can meet the data needs of data consumers and data producers in different scenarios.

[0106] Step 503: Determine at least one authorized database group corresponding to each data query request based on the query authorization information.

[0107] In this embodiment, the query authorization information can be a session authorization code, a data consumer identification code, etc. The electronic device can pre-store the mapping relationship between each query authorization information and an authorization database group. Each query authorization information corresponds to at least one authorization database group. Alternatively, the query authorization information can include an authorization database group identifier, allowing the electronic device to determine at least one authorization database corresponding to the data query request based on the authorization database group identifier. It is understood that the electronic device will only query each piece of data to be queried in the data query request within the authorized database group.

[0108] Step 504: Based on the type of each data query request, query each data to be queried in the corresponding authorized database group according to the identification information of the data to be queried.

[0109] In this embodiment, the identification information of the data to be queried can be fields, field values, etc. in a relational database, or keys, key values, etc. in a non-relational database.

[0110] In this embodiment, when the data query request type is HTTP, the identification information of the data to be queried is a key or key-value pair in a non-relational database. Therefore, the data to be queried can be queried in the non-relational database of each authorized data group corresponding to the data query request, using the identification information of the data to be queried as the keyword.

[0111] For example, when the data query request type is HTTP, the identifier information of the data to be queried is "key=1001". Using "key=1001" as the keyword, in the non-relational database of each authorized data group, the key value corresponding to key 1001 is found to be name. Then the query result of the data to be queried is name.

[0112] In this embodiment, when the data query request is SQL, the identification information of the data to be queried is a field or field value in a relational database. Therefore, the data to be queried can be queried in the relational database of each authorized data group corresponding to the data query request, using the identification information of the data to be queried as the key.

[0113] For example, when the data query request type is SQL, the identifier information of the data to be queried is field u.cid = 10011027 and field d.cid = 10011027. Therefore, in the relational database of each authorized data group corresponding to this data query request, we can query the data record with the field value of u.cid being 10011027 and the field value of d.cid being 10011027, and we can identify this data record as the query result of the data to be queried.

[0114] The data synchronization method provided in this embodiment receives data query requests sent by at least one data consumer. Each data query request includes a query interface type, query authorization information, and identification information for at least one piece of data to be queried. The method determines the type of each data query request based on the query interface type, determines at least one authorized database group corresponding to each data query request based on the query authorization information, and queries the data to be queried in each authorized database group according to the identification information of the data to be queried, based on the type of each data query request. Because the data to be queried is distributed to both relational and non-relational databases based on the type of the data query request, the query speed can be maximized while meeting the needs of the data consumer, thereby achieving more efficient synchronization of large amounts of concurrent data.

[0115] As an optional implementation, based on Embodiment 4, the types of data query requests include complex queries and simple queries, and step 504 is further refined, which includes steps 601 to 602.

[0116] Step 601: If it is determined that the type of data query request is a simple query, then query each data to be queried in the non-relational database of each authorized database group corresponding to the data query request according to the identification information of the data to be queried.

[0117] In this embodiment, a simple query can be to query the value corresponding to a key in the database, or it can be to query the identifier of a data record whose key value is a preset value.

[0118] Complex queries can be queries for the value of a specific field, the values ​​of other fields in a data record where a field has a preset value, or the identifier of a data record where several preset fields have preset values.

[0119] In this embodiment, the type of a data query request with an HTTP query interface can be defined as a simple query, and the type of a data query request with an SQL query interface can be defined as a complex query.

[0120] In this embodiment, the non-relational database may include a master database and at least one slave database. For simple queries, the data to be queried can be queried in multiple slave databases of the non-relational database. The number of data to be queried in each non-relational database slave database can be determined by the number of slave databases and the number of data to be queried. For example, the data to be queried can be evenly distributed among the non-relational database slave databases for querying, so as to quickly obtain query results and improve data synchronization efficiency.

[0121] Step 602: If the data query request is determined to be a complex query, then query each data to be queried in the relational database of each authorized database group corresponding to the data query request according to the identification information of the data to be queried.

[0122] In this embodiment, the relational database includes a master database and at least one slave database. For complex queries, the data to be queried can be queried in multiple slave databases. The number of data to be queried in each slave database can be determined by the number of slave databases and the number of data to be queried. For example, the data to be queried can be evenly distributed among the relational database slave databases for querying to quickly obtain query results and improve data synchronization efficiency.

[0123] The data synchronization method provided in this embodiment categorizes data query requests into complex and simple queries. If the data query request is determined to be a simple query, the method queries the data to be queried in the non-relational databases of the corresponding authorized database groups according to the identifier information of the data to be queried. If the data query request is determined to be a complex query, the method queries the data to be queried in the relational databases of the corresponding authorized database groups according to the identifier information of the data to be queried. Because simple queries are performed in non-relational databases and complex queries are performed in relational databases, query results can be obtained as quickly as possible for different types of data queries, thereby improving data synchronization efficiency.

[0124] As an optional implementation, based on any of the above embodiments, the type of data query request includes a comprehensive query, the identification information of the data to be queried includes the identification information of at least one item to be queried, and step 504 is further refined, then step 504 is further refined to include steps 701 to 702.

[0125] Step 701: If the data query request type is determined to be a comprehensive query, then query according to the identification information of each query item in the non-relational database of each authorized database group corresponding to the data query request, and determine the query result of each query item; the query result of each query item is either a result or no result.

[0126] In this embodiment, the data query request type includes comprehensive queries. Specifically, in this embodiment, a data query request with an SQL query interface type can be defined as a comprehensive query.

[0127] The representation information of the data to be queried includes at least one identifier for the queried item. The queried item can be a field, field value, primary key value, etc. Among them, the field includes at least one of the row field and column field.

[0128] In this embodiment, the identifier information of the query item can be queried first in the non-relational databases of each authorized database corresponding to the data query request, and the query result can be determined as either having results or not having results. Here, since the data in the non-relational databases is not as comprehensive as that in the relational databases, there may be cases where no query results are found.

[0129] Step 702: If it is determined that there is no result for any query item, then query the query item with no result in the relational data of each authorized database group corresponding to the data query request according to the corresponding identification information.

[0130] In this embodiment, for any piece of data to be queried, if it is determined that there is no result for any item to be queried, then in the authorized relational database corresponding to the data query request, these items with no results are queried according to their corresponding identification information to obtain the query results for each item to be queried.

[0131] The query results of each query item are combined to obtain the query result of the query data, and the query result of the query item can be sent to the corresponding data consumer.

[0132] The data synchronization method provided in this embodiment, if determined that the type of data query request is a comprehensive query, then queries are performed in the non-relational databases of each authorized database group corresponding to the data query request according to the identification information of each query item, and the query results for each query item are determined; the query results for each query item are either result-based or resultless; if it is determined that any query item has no result, then the query item with no result is queried in the relational databases of each authorized database group corresponding to the data query request according to the corresponding identification information. Because the query is first performed in the non-relational database, quickly identifying the query items that can yield results in the non-relational database, and then querying the resultless query items in the relational database, the query results for the data to be queried can be determined quickly and completely, improving data synchronization efficiency.

[0133] Example 4

[0134] Figure 5 This is a flowchart illustrating the data synchronization method provided in Embodiment 4 of this application. Figure 5 As shown, the data synchronization method provided in this embodiment, based on any of the above embodiments, further includes steps 801 to 802.

[0135] Step 801: Obtain the check codes of each relational data record in each relational database and the check codes of each non-relational data record in each non-relational database.

[0136] In this embodiment, when the electronic device writes data to a relational database and a non-relational database, it can generate a corresponding data checksum. For example, a Cyclic Redundancy Check (CRC) algorithm can be used to generate the checksum.

[0137] In this embodiment, the electronic device can generate a full verification task according to a preset time period. The full verification task is used to verify and correct each non-relational data in the non-relational data.

[0138] The electronic device performs a full verification task, sequentially retrieving the check codes of relational and non-relational data records with the same identifier from each database group. It then compares the check codes of the relational and non-relational data records.

[0139] Step 802: Verify and correct each non-relational data using each relational data.

[0140] In this embodiment, if the comparison results are the same, the verification is completed, and the verification codes of the next set of relational and non-relational data records with the same identifier are obtained and compared again. If the comparison results are different, the data records in the non-relational database are corrected using the data records in the relational database.

[0141] The data synchronization method provided in this embodiment obtains the checksums of each relational data record in each relational database and each non-relational data record in each non-relational database; and uses the relational data to verify and correct the non-relational data. Because the relational data records in the relational database are used to verify the non-relational data records in the non-relational database, real-time consistency and eventual consistency of data in each database group can be guaranteed.

[0142] As an optional implementation, the data synchronization method provided in this embodiment further includes steps 901 to 902, based on any of the above embodiments.

[0143] Step 901: Receive a data modification request sent by at least one data provider. The data modification request includes at least one data modification information. The data modification information may include at least one identifier of a data record to be modified and at least one corresponding item to be modified.

[0144] In this embodiment, after successfully writing the data to be written to the corresponding database group, the electronic device can send the identifier of each data record stored in a relational database or a non-relational database to the corresponding data provider. The items to be modified can be fields, field values, etc.

[0145] Step 902: Modify the data record to be modified in the relational database of the database group to be modified, based on the identifier of the data record to be modified and its corresponding at least one item to be modified.

[0146] In this embodiment, after receiving a data modification request sent by the data provider, the electronic device can determine the database group to be modified based on the identifier of the data record to be modified, and modify each item to be modified in each data record to be modified in the relational database of the database group to be modified.

[0147] Here, because the electronic device generates a full verification task according to a preset time period, it verifies and corrects each non-relational data in the non-relational database. Therefore, when modifying data records, only the data records in the relational database need to be modified, while the data records in the non-relational database can be modified by the full verification task.

[0148] As an optional implementation, the data synchronization method provided in this embodiment further includes steps 1001 to 1002 after step 901.

[0149] Step 1001: Determine at least one corresponding data consumer to be pushed based on the data modification information.

[0150] In this embodiment, the electronic device can determine the database group to be modified based on the identifier of the data record to be modified. The electronic device can pre-store the mapping relationship between each data consumer and the authorized database group, and can determine the data consumer corresponding to the database group to be modified based on the mapping relationship between each data consumer and the authorized database group.

[0151] Step 1002: Send a data change message to each data consumer to be pushed. If it is determined that the data change message failed to be sent, the data change message is put into the change message queue, and the data change message is sent to the data consumer to be pushed again after a preset time interval, until the number of times the data change message is sent to the data consumer to be pushed reaches the preset number or it is determined that the data change message was sent successfully. The change message queue is used to temporarily store the data change message.

[0152] Here, the message queue being changed can be the same queue as the new message queue mentioned earlier.

[0153] In this embodiment, the electronic device can send a data change message to the data consumer through a communication connection with the data consumer. The data change message is used to notify the data consumer that data records in its corresponding authorized database group have changed, and also to instruct the data consumer to send a change notification message to the electronic device.

[0154] If a change notification message is received from the data consumer within a preset time period after the electronic device sends a data change message to the data consumer, the data change message is considered to have been sent successfully; otherwise, the data change message is considered to have failed to be sent.

[0155] The data consumer may fail to receive data change messages in a timely manner due to its own reasons. Therefore, the electronic device can push data change messages to the data consumer multiple times until the number of pushes reaches a preset number or the data change message is successfully pushed, thus ensuring that the data consumer can receive the data change messages in a timely manner. The preset number of pushes can be 5.

[0156] Example 5

[0157] Figure 6 This is a schematic diagram of the data synchronization device provided according to Embodiment 5 of this application. Figure 6 As shown, the data synchronization device 60 provided in this embodiment includes: a receiving module 61, a determining module 62, and a writing module 63.

[0158] The receiving module 61 is configured to receive a data write request sent by at least one data provider, wherein the data write request includes a data provider identifier and at least one piece of data to be written.

[0159] The determination module 62 is used to determine the target database group corresponding to each data provider based on the identifier of each data provider. Each database group includes one relational database and one non-relational database.

[0160] The write module 63 is used to write each piece of data to be written into the corresponding relational database and non-relational database.

[0161] As an optional implementation, the writing module 63 is specifically used to determine the data sequence number of each piece of data to be written, with the data sequence number of each piece of data to be written increasing sequentially; and to write each piece of data to be written into the corresponding relational database and non-relational database using the data sequence number of each piece of data to be written as an identifier.

[0162] As an optional implementation, the writing module 63 is further configured to: obtain the maximum identifier number in each data record of each database group, and determine the maximum identifier number as the cache sequence number; for each data write request, perform the following operations: determine the data sequence number of each data to be written in the data write request based on the cache sequence number, wherein the data sequence number of each data to be written in the data write request is sequentially increased and is greater than the cache sequence number; update the cache sequence number to the maximum item among the data sequence numbers of each data to be written in the data write request.

[0163] As an optional implementation, the data synchronization device 60 further includes a query module, which is used to: receive data query requests sent by at least one data consumer, the data query requests including query interface type, query authorization information, and identification information of at least one piece of data to be queried; determine the type of each data query request according to each query interface type; determine at least one authorized database group corresponding to each data query request according to each query authorization information; and query each piece of data to be queried in each authorized database group corresponding to each data query request according to the identification information of the data to be queried, based on the type of each data query request.

[0164] As an optional implementation, the data query request type includes complex queries and simple queries. Specifically, the query module is used to: if the data query request type is determined to be a simple query, then query each data to be queried in the non-relational databases of each authorized database group corresponding to the data query request according to the identification information of the data to be queried; if the data query request type is determined to be a complex query, then query each data to be queried in the relational databases of each authorized database group corresponding to the data query request according to the identification information of the data to be queried.

[0165] As an optional implementation, the data query request type includes a comprehensive query. Specifically, the query module is further used to: if it is determined that the data query request type is a comprehensive query, then perform a query in the non-relational databases of each authorized database group corresponding to the data query request according to the identification information of each query item, and determine the query result of each query item; the query result of each query item is either a result or no result; if it is determined that any query item has no result, then perform a query in the relational databases of each authorized database group corresponding to the data query request for the query item with no result according to the corresponding identification information.

[0166] As an optional implementation, the data synchronization device 60 further includes a verification module, which is used to obtain the verification codes of each relational data record in each relational database and the verification codes of each non-relational data record in each non-relational database; and to verify and correct each non-relational data using each relational data.

[0167] The data synchronization device provided in this embodiment can execute the data synchronization method provided in any of the above embodiments. The specific implementation method and principle are similar, and will not be described again here.

[0168] Example 6

[0169] Figure 7 This is a schematic diagram of the structure of an electronic device according to Embodiment Six of this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes a memory 71, a processor 72, and a transceiver 73.

[0170] The memory 71, processor 72 and transceiver 73 are interconnected.

[0171] Memory 71 stores computer-executed instructions.

[0172] Transceiver 73 is used for sending and receiving data.

[0173] The processor 72 executes the computer execution instructions stored in the memory 71 to implement the data synchronization method provided in any of the above embodiments. The specific implementation method and principle are similar and will not be described again here.

[0174] The memory 71, processor 72 and transceiver 73 can be interconnected via a bus.

[0175] The memory 71 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, etc.

[0176] In an exemplary embodiment, the electronic device 70 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the data synchronization method described above.

[0177] Embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these instructions are used to implement the data synchronization method provided in any of the above embodiments. Exemplarily, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), magnetic tape, floppy disk, or optical data storage device, etc.

[0178] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the module division in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules can be combined, or integrated into another system, or some features can be ignored or not executed.

[0179] Furthermore, unless otherwise specified, the functional modules in the various embodiments of this application can be integrated into one module, or each module can exist physically separately, or two or more modules can be integrated together. The integrated modules described above can be implemented in hardware or as software program modules.

[0180] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the module division in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed.

[0181] Furthermore, unless otherwise specified, the functional modules in the various embodiments of this application can be integrated into one module, or each unit / module can exist physically separately, or two or more modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0182] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0183] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0184] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0185] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that, include: Receive at least one data provider sending a data write request, the data write request including a data provider identifier and at least one piece of data to be written; The target database group corresponding to each data provider is determined based on the identifier of each data provider. Each database group includes one relational database and one non-relational database. Obtain the maximum identifier number in each data record of each database group, and determine the maximum identifier number as the cache sequence number. Based on the cache sequence number, determine the data sequence number of each data to be written in the data write request. The data sequence number of each data to be written is sequentially incremented and is greater than the cache sequence number. Then update the cache sequence number to the maximum item among the data sequence numbers of each data to be written in the write request. Using the data sequence number of each data to be written as an identifier, each data to be written is written to the corresponding relational database and non-relational database.

2. The data synchronization method according to claim 1, characterized in that, Also includes: Receive at least one data query request sent by a data consumer, wherein the data query request includes query interface type, query authorization information and identification information of at least one piece of data to be queried; The type of each data query request is determined based on the type of each query interface. Determine at least one authorized database group corresponding to each data query request based on the query authorization information; Based on the type of each data query request, the system queries each data to be queried in the corresponding authorized database group according to the identification information of the data to be queried.

3. The method according to claim 2, characterized in that, The data query request types include complex queries and simple queries; the process of querying each data to be queried in each authorized database group corresponding to each data query request according to the identification information of the data to be queried, based on the type of each data query request, includes: If the data query request is determined to be a simple query, then each data to be queried is queried in the non-relational databases of the authorized database groups corresponding to the data query request according to the identification information of the data to be queried; If the type of data query request is determined to be a complex query, then each piece of data to be queried is queried in the relational database of each authorized database group corresponding to the data query request, according to the identification information of the data to be queried.

4. The method according to claim 2, characterized in that, The data query request type includes comprehensive query; the identification information of the data to be queried includes identification information of at least one item to be queried; the step of querying each piece of data to be queried in each authorized database group corresponding to each data query request according to the identification information of the data to be queried, based on the type of each data query request, includes: If the data query request is determined to be a comprehensive query, then the query is performed in the non-relational databases of each authorized database group corresponding to the data query request according to the identification information of each query item, and the query result of each query item is determined; the query result of each query item is either a result or no result. If it is determined that there is no result for any of the query items, then the query item with no result will be queried in the relational data of each authorized database group corresponding to the data query request according to the corresponding identification information.

5. The method according to any one of claims 1-4, characterized in that, Also includes: Retrieve the checksums of each relational data record in each relational database and each non-relational data record in each non-relational database; The relational data is used to verify and correct the non-relational data.

6. A data synchronization device, characterized in that, include: The receiving module is configured to receive a data write request sent by at least one data provider, wherein the data write request includes a data provider identifier and at least one piece of data to be written; The determination module is used to determine the target database group corresponding to each data provider based on the identifier of each data provider. Each database group includes a relational database and a non-relational database. The write module is used to obtain the maximum identifier number in each data record of each database group, and determine the maximum identifier number as the cache sequence number. Based on the cache sequence number, it determines the data sequence number of each data to be written in the data write request, wherein the data sequence numbers of each data to be written are sequentially incremented and are all greater than the cache sequence number. The cache sequence number is then updated to the maximum item among the data sequence numbers of each data to be written in the write request. Using the data sequence number of each data to be written as the identifier, each data to be written is written to the corresponding relational database and non-relational database.

7. An electronic device, characterized in that, include: Memory, processor, and transceiver; The memory, the processor, and the transceiver circuitry are interconnected; The memory stores computer-executed instructions; The transceiver is used to send and receive data; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

Images