User data synchronization method and apparatus, storage medium, and electronic device

Abstract

The application discloses a kind of user data synchronization method, device, storage medium and electronic equipment.It relates to financial technology or other related fields, which comprises: obtaining N routing table, wherein each routing table contains user identification and data storage information, the content recorded in routing table is determined according to user identification, data storage information represents the expected storage path of user identification matched user data in the first storage platform, and N is a positive integer;Obtain the user data to be synchronized sent by the second storage platform, according to the user identification in the user data to be synchronized, divide the user data to be synchronized into N files;Determine the routing table matched in each file of N files, according to the data storage information in the routing table matched by each file, synchronously synchronize the user data in each file to the first storage platform.The application solves the technical problem of low synchronization efficiency when synchronizing data between platforms in related technologies.

Description

Technical Field

[0001] This invention relates to financial technology or other related fields, and more specifically, to a method, apparatus, storage medium, and electronic device for synchronizing user data. Background Technology

[0002] With economic development and technological advancements, many organizations have prioritized transforming their mainframes into distributed architectures due to the high coupling and low concurrency of traditional mainframes. Data synchronization has been a central element throughout this transformation process. From the initial migration of mainframe data to a distributed platform, to the parallel operation of mainframe and distributed systems, and finally to situations where production issues require repairs based on mainframe data, data synchronization plays a crucial role in all these stages.

[0003] In related technologies, due to the different data sharding rules of host and distributed platforms, data synchronization methods are often simple and direct. Figure 1 This is a schematic diagram of a data synchronization method in related technologies, such as... Figure 1 As shown, the process involves first generating the data on the host machine, merging and decoding it before sending it to the distributed platform. Then, the data is split according to the distributed platform's routing rules before finally being imported into the distributed platform's database. While this method is easy to understand and operate, it suffers from low synchronization efficiency when migrating large amounts of data.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, storage medium, and electronic device for synchronizing user data, to at least solve the technical problem of low synchronization efficiency in synchronizing data between platforms in related technologies.

[0006] According to one aspect of the present invention, a method for synchronizing user data is provided, comprising: acquiring N routing tables, wherein each routing table contains a user identifier and data storage information, the content recorded in the routing table is determined according to the user identifier, and the data storage information represents the expected storage path of the user data matched by the user identifier in a first storage platform, where N is a positive integer; acquiring user data to be synchronized sent by a second storage platform, and dividing the user data to be synchronized into N files according to the user identifier in the user data to be synchronized; determining the routing table matched by each of the N files, and synchronizing the user data in each file to the first storage platform in parallel according to the data storage information in the routing table matched by each file.

[0007] Furthermore, the user data synchronization method also includes: obtaining a master routing table, wherein the master routing table contains all user identifiers recorded in the second storage platform and data storage information matching all user identifiers; for each user identifier recorded in the master routing table, calculating the remainder when the user identifier is divided by N; and based on the remainder when each user identifier recorded in the routing table is divided by N, dividing each user identifier and data storage information recorded in the routing table into N routing tables.

[0008] Furthermore, the user data synchronization method also includes: for each user identifier recorded in the total routing table, if the remainder when the user identifier is divided by N is M, then the first target data is allocated to the Mth routing table among the N routing tables, where the first target data refers to the user identifier in the total routing table and the data storage information matched by that user identifier; if the remainder when the user identifier is divided by N is 0, then the first target data is allocated to the Nth routing table among the N routing tables.

[0009] Furthermore, the user data synchronization method also includes: obtaining a preset first value and determining the first value as N; obtaining a preset threshold, and determining N based on the number of user identifiers in the total routing table if the number of user identifiers in the total routing table is greater than or equal to the preset threshold, and determining a preset second value as N if the number of user identifiers in the total routing table is less than the preset threshold.

[0010] Furthermore, the user data synchronization method also includes: for each user identifier recorded in the general routing table, determining a general user identifier that matches the user identifier, wherein the user identifier is an identifier used to represent the user identity in the target business scenario, and the general user identifier is an identifier used to represent the user identity in all business scenarios; performing a hash calculation on the general user identifier to obtain a hash value, and determining the data storage information that matches the user identifier based on the hash value.

[0011] Furthermore, the user data synchronization method also includes: for each user identifier in the user data to be synchronized, calculating the remainder when the user identifier is divided by N; if the remainder when the user identifier is divided by N is M, then the second target data is divided into the Mth file of the N files, where the second target data refers to the user data whose user identifier matches the user data in the user data to be synchronized; if the remainder when the user identifier is divided by N is 0, then the second target data is divided into the Nth file of the N files.

[0012] Furthermore, the user data synchronization method also includes: for the Pth file in N files, determining the Pth routing table in the N routing tables as the routing table that matches the Pth file, where P is less than or equal to N.

[0013] According to another aspect of the present invention, a user data synchronization apparatus is also provided, comprising: a first acquisition module, configured to acquire N routing tables, wherein each routing table contains a user identifier and data storage information, the content recorded in the routing table is determined according to the user identifier, and the data storage information represents the expected storage path of the user data matched by the user identifier in a first storage platform, where N is a positive integer; a second acquisition module, configured to acquire user data to be synchronized sent by a second storage platform, and divide the user data to be synchronized into N files according to the user identifier in the user data to be synchronized; and a synchronization module, configured to determine the routing table matched by each of the N files, and synchronize the user data in each file to the first storage platform in parallel according to the data storage information in the routing table matched by each file.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described user data synchronization method at runtime.

[0015] According to another aspect of the present invention, an electronic device is also provided, the electronic device including one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are configured to run the programs, wherein the programs are configured to execute the above-described user data synchronization method during runtime.

[0016] In this embodiment of the invention, the routing information and the user data to be synchronized are divided into multiple files, and then the multiple files are processed in parallel to achieve data synchronization. By obtaining N routing tables, the user data to be synchronized sent by the second storage platform is obtained. Based on the user identifier in the user data to be synchronized, the user data to be synchronized is divided into N files, thereby determining the routing table matching each file in the N files. Based on the data storage information in the routing table matching each file, the user data in each file is synchronized to the first storage platform in parallel.

[0017] It is worth noting that when all user data records in the user data to be synchronized are in the same table, the first storage platform can only read the user data in the table sequentially and match it with the routing table. Therefore, when there is too much user data to be synchronized, the first storage platform needs to consume a lot of time for information reading and matching. In this implementation, by dividing the data to be synchronized into N files, the first storage platform can use multiple threads to read the user data in each file in parallel and match it with the routing table, thereby quickly determining the data storage information corresponding to the user data and improving the data synchronization efficiency.

[0018] Therefore, the solution provided in this application achieves the goal of dividing routing information and user data to be synchronized into multiple files, and then processing multiple files in parallel to achieve data synchronization, thereby improving the technical effect of data synchronization efficiency and solving the technical problem of synchronization efficiency in related technologies when synchronizing data between platforms. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a schematic diagram of a data synchronization method in related technologies;

[0021] Figure 2 This is a flowchart of an optional user data synchronization method according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of an optional user data synchronization method according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of an optional user data synchronization device according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It should be noted that the user data synchronization method, device, storage medium and electronic device disclosed herein can be used in the fintech field, or in any field other than fintech. The application fields of the user data synchronization method, device, storage medium and electronic device disclosed herein are not limited.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0029] Example 1

[0030] According to an embodiment of the present invention, an embodiment of a user data synchronization method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] Figure 2 This is a flowchart of an optional user data synchronization method according to an embodiment of the present invention, such as... Figure 2 As shown, this method is applied to a first storage platform and includes the following steps:

[0032] Step S201: Obtain N routing tables, where each routing table contains user identifiers and data storage information. The content recorded in the routing table is determined based on the user identifier, and the data storage information represents the expected storage path of the user data matched by the user identifier in the first storage platform. N is a positive integer.

[0033] In this embodiment, the first storage platform is used to synchronize data from the second storage platform to the first storage platform. The first storage platform can be a distributed platform, and the second storage platform can be a traditional host. In other embodiments, the first storage platform and the second storage platform can also be other types of platforms.

[0034] Optionally, the routing table can be considered to store multiple key-value pairs. The key in the key-value pair is the user identifier, and the value is the data storage information. The key in the routing table can be determined in advance from the second storage platform, that is, the user identifier in the routing table is the user identifier that exists in the second storage platform. The value in the routing table can be set by relevant personnel, or it can be automatically determined by the first storage platform according to a preset method.

[0035] The content recorded in the routing table is determined based on the user identifier. For example, if the user identifier is a number, the user identifier ending in 1 and its corresponding data storage information are recorded in the first routing table of N routing tables, the user identifier ending in 2 and its corresponding data storage information are recorded in the second routing table of N routing tables, and so on, to determine the content recorded in the routing table. As another example, if the user identifier is an English letter, the user identifier ending in 'ac' and its corresponding data storage information are recorded in the first routing table of N routing tables, the user identifier ending in 'df' and its corresponding data storage information are recorded in the second routing table of N routing tables, and so on. As yet another example, if the user identifier is a Chinese character, the content recorded in each routing table can be determined according to the number of strokes in the Chinese characters.

[0036] Step S202: Obtain the user data to be synchronized sent by the second storage platform, and divide the user data to be synchronized into N files according to the user identifier in the user data to be synchronized.

[0037] In this embodiment, the method used by the first storage platform to divide the user data to be synchronized into N files is the same as the method used to divide the user identifiers into the routing tables. Therefore, in this embodiment, it is always guaranteed that if a certain user identifier is recorded in the Mth routing table of the N routing tables, then that user identifier and the corresponding user data in the user data to be synchronized will also be divided into the Mth file of the N files. It should also be noted that since there may be more than one record of user data for the same user, the same user identifier may match one or more user records in the data to be synchronized.

[0038] Step S203: Determine the routing table that matches each file in the N files, and synchronize the user data in each file to the first storage platform in parallel based on the data storage information in the routing table that matches each file.

[0039] Once the routing tables matching each file in the N files are determined, for each file, the first storage platform can find the data storage information corresponding to the user data in that file from the routing table that matches that file, and thus store the user data in the file into the storage path indicated in the data storage information, thereby completing data synchronization.

[0040] Based on the scheme defined in steps S201 to S203 above, it can be understood that in this embodiment of the invention, the routing information and the user data to be synchronized are divided into multiple files, and then multiple files are processed in parallel to achieve data synchronization. By obtaining N routing tables, and then obtaining the user data to be synchronized sent by the second storage platform, the user data to be synchronized is divided into N files according to the user identifier in the user data to be synchronized, thereby determining the routing table matching each file in the N files. According to the data storage information in the routing table matching each file, the user data in each file is synchronized to the first storage platform in parallel, that is, synchronized to each database in the first storage platform.

[0041] It is worth noting that when all user data records in the user data to be synchronized are in the same table, the first storage platform can only read the user data in the table sequentially and match it with the routing table. Therefore, when there is too much user data to be synchronized, the first storage platform needs to consume a lot of time for information reading and matching. In this implementation, by dividing the data to be synchronized into N files, the first storage platform can use multiple threads to read the user data in each file in parallel and match it with the routing table, thereby quickly determining the data storage information corresponding to the user data and improving the data synchronization efficiency.

[0042] Therefore, the solution provided in this application achieves the goal of dividing routing information and user data to be synchronized into multiple files, and then processing multiple files in parallel to achieve data synchronization, thereby improving the technical effect of data synchronization efficiency and solving the technical problem of synchronization efficiency in related technologies when synchronizing data between platforms.

[0043] In one optional embodiment, the user identifier consists of numbers, wherein the first storage platform can obtain N routing tables by: obtaining a total routing table, wherein the total routing table contains all user identifiers recorded in the second storage platform and contains data storage information matching all user identifiers; for each user identifier recorded in the total routing table, calculating the remainder when the user identifier is divided by N; and based on the remainder when each user identifier recorded in the routing table is divided by N, dividing each user identifier recorded in the routing table and the data storage information into N routing tables.

[0044] Optionally, the user identifier in the master routing table is pre-determined from the second storage platform. The data storage information can be set by relevant personnel or automatically determined by the first storage platform according to a preset method. In the master routing table, the user identifier and data storage information are also stored in key-value pairs.

[0045] In this embodiment, the user identifier is composed of numbers. Therefore, during the partitioning process, the first storage platform sequentially reads each user identifier recorded in the total routing table and calculates the remainder when the user identifier is divided by N for each user identifier recorded in the total routing table. For example, if the user identifier is 11 and N is 3, the remainder is 2. Or, for example, if the user identifier is 11 and N is 5, the remainder is 1.

[0046] Furthermore, the first storage platform can allocate user identifiers with a remainder of 1 and their corresponding data storage information to the first routing table, user identifiers with a remainder of 2 and their corresponding data storage information to the second routing table, and so on. Optionally, the first storage platform can also allocate user identifiers with remainders of 1-3 and their corresponding data storage information to the first routing table, user identifiers with remainders of 4-6 and their corresponding data storage information to the second routing table, and so on.

[0047] It should be noted that the above process can effectively divide the user identifier and data storage information in the master routing table.

[0048] In one optional embodiment, during the process of dividing each user identifier recorded in the routing table and the data storage information into N routing tables based on the remainder when each user identifier is divided by N, the first storage platform can, for each user identifier recorded in the total routing table, divide the first target data into the Mth routing table of the N routing tables if the remainder when the user identifier is divided by N is M, and divide the first target data into the Nth routing table if the remainder when the user identifier is divided by N is 0. Here, the first target data refers to the user identifier in the total routing table and the data storage information matching that user identifier.

[0049] For example, user identifiers with a remainder of 1 and their corresponding data storage information are assigned to the first routing table, user identifiers with a remainder of 2 and their corresponding data storage information are assigned to the second routing table, and so on. User identifiers with a remainder of 0 and their corresponding data storage information are assigned to the Nth routing table. Optionally, in this embodiment, the initial content of the N routing tables is empty. Assuming the total routing table records X key-value pairs, in this embodiment, the total routing table is traversed only once, and X table write operations are performed; the time taken can be denoted as T2.

[0050] It should be noted that the above process improves the efficiency of routing table partitioning.

[0051] In one optional embodiment, the first storage platform may determine N in one of the following ways: obtain a preset first value and determine the first value as N; obtain a preset threshold, and determine N based on the number of user identifiers in the total routing table if the number of user identifiers in the total routing table is greater than or equal to the preset threshold, and determine a preset second value as N if the number of user identifiers in the total routing table is less than the preset threshold.

[0052] The preset first value can be pre-set and may be related to the business scenario of the user data. The first storage platform can determine the preset first value based on the business scenario of the data to be received and synchronized. For example, the second storage platform first sends a data synchronization request to the first storage platform, and the data synchronization request includes the business scenario of the data to be synchronized. If the first storage platform allows data synchronization, the first storage platform receives the data to be synchronized. Optionally, the preset first value can also be a fixed value, i.e., unrelated to the business scenario.

[0053] Optionally, the first storage platform can also obtain a preset threshold, and then compare the number of user identifiers in the total routing table with the preset threshold. If the number of user identifiers in the total routing table is greater than or equal to the preset threshold, the platform can take the square root of the number of user identifiers in the total routing table and then round it down. The rounded number is then determined as N. That is, if the number of user identifiers in the total routing table is X, then... The integer obtained is determined as N. Optionally, the first storage platform can also use other calculation methods to process the number of user identifiers in the total routing table to determine N. Conversely, if the number of user identifiers in the total routing table is less than a preset threshold, a preset second value is determined as N. The time taken to determine N can be denoted as T1. Compared to the massive data storage, this process is merely a simple numerical calculation, and the time used is negligible.

[0054] It should be noted that the above process enables the effective determination of N, thereby facilitating the improvement of the efficiency of matching from the routing table.

[0055] In an optional embodiment, the first storage platform can determine the data storage information in the following way: for each user identifier recorded in the general routing table, determine the general user identifier that matches the user identifier, wherein the user identifier is an identifier used to represent the user identity in the target business scenario, and the general user identifier is an identifier used to represent the user identity in all business scenarios; perform a hash calculation on the general user identifier to obtain a hash value, and determine the data storage information that matches the user identifier based on the hash value.

[0056] For example, a user identifier could be a user's fund account or trading account, while a general user identifier could be a user's mobile phone number or ID number. The matching relationship between user identifiers and general user identifiers can be pre-determined by the first storage platform from the second storage platform.

[0057] Once the general user identifier matching each user identifier recorded in the master routing table is determined, the first storage platform can perform a hash calculation on the general user identifier to obtain a hash value, and then obtain the correspondence between the hash value and the storage path in the first storage platform. Based on this correspondence and the hash value, the data storage information matching the corresponding user identifier can be determined.

[0058] It should be noted that the above process enables the automatic determination of data storage information, thereby reducing labor costs and improving data synchronization efficiency.

[0059] In one optional embodiment, during the process of dividing the user data to be synchronized into N files based on the user identifiers in the user data to be synchronized, the first storage platform can calculate the remainder when the user identifier is divided by N for each user identifier in the user data to be synchronized. If the remainder when the user identifier is divided by N is M, then the second target data is divided into the Mth file of the N files; if the remainder when the user identifier is divided by N is 0, then the second target data is divided into the Nth file of the N files. Here, the second target data refers to the user data whose user identifier matches the user data in the user data to be synchronized.

[0060] During the partitioning process, the first storage platform sequentially reads each user identifier recorded in the user data to be synchronized, and calculates the remainder when the user identifier is divided by N for each user identifier recorded in the user data to be synchronized.

[0061] Furthermore, the first storage platform can allocate user data matching user identifiers with a remainder of 1 to the first file, user data matching user identifiers with a remainder of 2 to the second file, and so on. User data matching user identifiers with a remainder of 0 is allocated to the Nth file. Optionally, in this embodiment, the initial content of the N files is empty. Compared to reading and writing routing tables, file reading and writing is much faster. At the same data level, the time T3 for partitioning into N files is less than the time T2 for partitioning into N routing tables. Moreover, the process of partitioning into N files can be processed in parallel with the process of partitioning into N routing tables.

[0062] It should be noted that the above process achieves an effective division of user data to be synchronized.

[0063] In an optional embodiment, during the process of determining the routing table that matches each file in N files, the first storage platform can determine the Pth routing table in the N routing tables as the routing table that matches the Pth file, where P is less than or equal to N.

[0064] Since the N files and N routing tables are partitioned in the same way, and both are partitioned based on the remainder when the user identifier is divided by N, the Pth routing table in the N routing tables has the same remainder when the user identifier in the Pth file in the N files is divided by N. Therefore, the Pth routing table matches the Pth file. This enables the rapid determination of the matching relationship between files and routing tables.

[0065] It's important to note that database processing efficiency is related to the amount of data in a table, the creation of indexes, and other factors. When a single table contains more than 20 million rows, the performance of SQL operations will drop drastically. Assuming the amount of user data to be synchronized is Y, and matching is performed using batched files and a routing table, the amount of data that a single thread needs to match is Y / N, and the required matching time is... Where, ΔT s This represents the time taken to match a user identifier in a file with a smaller data volume against a user identifier in a routing table with a smaller data volume. This matching time T4 is much shorter than the matching time T5 = Y·ΔT in related technologies. G , where ΔT G This represents the time taken to match the user identifier in the file with the user identifier in the routing table, which has a larger data volume. In other words, it represents the matching time without splitting the routing table and the file.

[0066] Figure 3This is a schematic diagram of an optional user data synchronization method according to an embodiment of the present invention. In this embodiment, during the data synchronization process, since the user identifiers in the routing table are relatively stable in most cases, i.e., a large number of user identifiers will not suddenly increase, therefore, Figure 3 As shown, the process of determining the number of batches N and the process of dividing the data into N routing tables can be performed in advance by the first storage platform. Furthermore, when data synchronization requires multiple iterations, these processes can be reused in subsequent data synchronization processes. Therefore, the aforementioned T1 and T2 can be disregarded in the data synchronization time. During the data synchronization process, as... Figure 3 As shown, the first storage platform performs the following operations each time: retrieves the user data to be synchronized, divides the user data to be synchronized into N files, matches them in parallel, and then stores the data based on the matching results. Therefore, the data synchronization time T 耗时 It can be represented as follows:

[0067]

[0068] Since file reading and writing are much faster than reading and writing routing tables and data matching, the time T3 is much smaller than the difference between T4 and T5, and the difference between T4 and T5 increases with the amount of data. Therefore, compared with solutions in related technologies, the method provided in this application can effectively improve data synchronization efficiency.

[0069] The following example illustrates the difference between the data synchronization method in related technologies and the user data synchronization method in this application. In related technologies, the data synchronization process is as follows:

[0070] Step 1: Merge and download the host data to the server of the distributed platform. The host data contains 100 million data entries.

[0071] Step 2: The distributed platform reads the data in the file sent by the host, and queries the routing table containing 25 million data entries one by one to obtain the corresponding routing label (i.e., data storage information). This is equivalent to performing 100 million slow queries in sequence, and finally obtaining 16 files.

[0072] Step 3: Import the 16 files into the distributed platform's database.

[0073] In this embodiment, the data synchronization process is as follows:

[0074] Step 1: Know in advance the total amount of data in the routing table (i.e., 25 million as mentioned above), and assess that the number of routing tables to be allocated is 5,000;

[0075] Step 2: Divide the total routing table into batches to obtain 5000 routing tables, with each routing table containing 5000 data entries;

[0076] Step 3: Merge and download the host data to the server of the distributed platform. The host data contains 100 million data entries.

[0077] Step 4: Divide the host into batches, resulting in 5000 files, each with 20000 data entries;

[0078] Step 5: Start 5000 threads to match files and routing tables in parallel;

[0079] Step 6: Distribute the matching results file to the database of the distributed platform.

[0080] It is obvious that since data in the same table can only be read sequentially and cannot be processed in parallel, the time required to match 100 million and 25 million data entries in related technologies is far greater than the time required to match 20,000 and 5,000 data entries in parallel in this embodiment. Specifically, by controlling the amount of data in the routing table to the millions, query efficiency can be effectively improved. For example, if the number of batches in the above example is fixed at 20, each routing table will only contain 1.25 million data entries, thus greatly improving query efficiency while reducing the number of threads to a real-time controllable value.

[0081] Therefore, the solution provided in this application achieves the goal of dividing routing information and user data to be synchronized into multiple files, and then processing multiple files in parallel to achieve data synchronization, thereby improving the technical effect of data synchronization efficiency and solving the technical problem of synchronization efficiency in related technologies when synchronizing data between platforms.

[0082] Example 2

[0083] According to an embodiment of the present invention, an embodiment of a user data synchronization device is provided, wherein, Figure 4 This is a schematic diagram of an optional user data synchronization device according to an embodiment of the present invention, such as... Figure 4 As shown, the device includes:

[0084] The first acquisition module 401 is used to acquire N routing tables, wherein each routing table contains user identifiers and data storage information. The content recorded in the routing table is determined based on the user identifier, and the data storage information represents the expected storage path of the user data matched by the user identifier in the first storage platform, where N is a positive integer.

[0085] The second acquisition module 402 is used to acquire the user data to be synchronized sent by the second storage platform, and divide the user data to be synchronized into N files according to the user identifier in the user data to be synchronized;

[0086] The synchronization module 403 is used to determine the routing table that matches each of the N files, and based on the data storage information in the routing table that matches each file, synchronize the user data in each file to the first storage platform in parallel.

[0087] It should be noted that the first acquisition module 401, the second acquisition module 402, and the synchronization module 403 mentioned above correspond to steps S201 to S203 in the above embodiments. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment 1.

[0088] Optionally, the user data synchronization device further includes: a third acquisition module for acquiring a total routing table, wherein the total routing table contains all user identifiers recorded in the second storage platform and data storage information matching all user identifiers; a calculation module for calculating the remainder of each user identifier recorded in the total routing table when divided by N; and a partitioning module for partitioning each user identifier recorded in the routing table and the data storage information into N routing tables based on the remainder of each user identifier recorded in the routing table when divided by N.

[0089] Optionally, the partitioning module further includes: a first partitioning submodule, used to partition the first target data to the Mth routing table of the N routing tables if the remainder when the user identifier is divided by N is M for each user identifier recorded in the total routing table, wherein the first target data refers to the user identifier in the total routing table and the data storage information matched by the user identifier; and a second partitioning submodule, used to partition the first target data to the Nth routing table of the N routing tables if the remainder when the user identifier is divided by N is 0.

[0090] Optionally, the user data synchronization device further includes: a first determining module, used to obtain a preset first value and determine the first value as N; a second determining module, used to obtain a preset threshold, and determine N based on the number of user identifiers in the total routing table when the number of user identifiers in the total routing table is greater than or equal to the preset threshold, and determine the preset second value as N when the number of user identifiers in the total routing table is less than the preset threshold.

[0091] Optionally, the user data synchronization device further includes: a third determining module, used to determine a general user identifier that matches the user identifier for each user identifier recorded in the general routing table, wherein the user identifier is an identifier used to represent the user identity in the target business scenario, and the general user identifier is an identifier used to represent the user identity in all business scenarios; and a fourth determining module, used to perform a hash calculation on the general user identifier to obtain a hash value, and determine the data storage information that matches the user identifier based on the hash value.

[0092] Optionally, the second acquisition module further includes: a calculation submodule, used to calculate the remainder of each user identifier in the user data to be synchronized when divided by N; a third partitioning submodule, used to partition the second target data into the Mth file of the N files if the remainder of the user identifier divided by N is M, wherein the second target data refers to the user data in the user data to be synchronized whose user identifier matches; and a fourth partitioning submodule, used to partition the second target data into the Nth file of the N files if the remainder of the user identifier divided by N is 0.

[0093] Optionally, the synchronization module further includes: a determination submodule, used to determine the Pth routing table in the N routing tables as the routing table that matches the Pth file for the Pth file, where P is less than or equal to N.

[0094] Example 3

[0095] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described user data synchronization method at runtime.

[0096] Example 4

[0097] According to another aspect of the present invention, an electronic device is also provided, wherein, Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of the present invention, such as... Figure 5 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when executed by one or more processors, enable the processors to run the programs, wherein the programs are configured to execute the aforementioned user data synchronization method during runtime. When the processor executes the program, it performs the following steps: obtaining N routing tables, each containing a user identifier and data storage information. The content recorded in the routing tables is determined based on the user identifier, and the data storage information represents the expected storage path of the user data matching the user identifier in the first storage platform, where N is a positive integer; obtaining user data to be synchronized sent by the second storage platform, and dividing the user data to be synchronized into N files based on the user identifier in the user data; determining the routing table matching each of the N files, and synchronizing the user data in each file to the first storage platform in parallel based on the data storage information in the routing table matching each file.

[0098] Furthermore, when the processor executes the program, it also performs the following steps: obtaining the total routing table, wherein the total routing table contains all user identifiers recorded in the second storage platform and contains data storage information matching all user identifiers; for each user identifier recorded in the total routing table, calculating the remainder when the user identifier is divided by N; and based on the remainder when each user identifier recorded in the routing table is divided by N, dividing each user identifier recorded in the routing table and the data storage information into N routing tables.

[0099] Furthermore, when the processor executes the program, it also implements the following steps: for each user identifier recorded in the total routing table, if the remainder when the user identifier is divided by N is M, then the first target data is assigned to the Mth routing table among the N routing tables, where the first target data refers to the user identifier in the total routing table and the data storage information matched by that user identifier; if the remainder when the user identifier is divided by N is 0, then the first target data is assigned to the Nth routing table among the N routing tables.

[0100] Furthermore, when the processor executes the program, it also performs the following steps: obtaining a preset first value and determining the first value as N; obtaining a preset threshold, and determining N based on the number of user identifiers in the total routing table if the number of user identifiers in the total routing table is greater than or equal to the preset threshold, and determining a preset second value as N if the number of user identifiers in the total routing table is less than the preset threshold.

[0101] Furthermore, when the processor executes the program, it also performs the following steps: for each user identifier recorded in the overall routing table, it determines the general user identifier that matches the user identifier, where the user identifier is the identifier used to represent the user identity in the target business scenario, and the general user identifier is the identifier used to represent the user identity in all business scenarios; it performs a hash calculation on the general user identifier to obtain a hash value, and determines the data storage information that matches the user identifier based on the hash value.

[0102] Furthermore, when the processor executes the program, it also performs the following steps: for each user identifier in the user data to be synchronized, calculate the remainder when the user identifier is divided by N; if the remainder when the user identifier is divided by N is M, then divide the second target data into the Mth file of the N files, where the second target data refers to the user data whose user identifier matches the user data in the user data to be synchronized; if the remainder when the user identifier is divided by N is 0, then divide the second target data into the Nth file of the N files.

[0103] Furthermore, when the processor executes the program, it also performs the following steps: for the Pth file in N files, the Pth routing table in the N routing tables is determined as the routing table that matches the Pth file, where P is less than or equal to N.

[0104] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0105] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0110] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for synchronizing user data, characterized in that, The user data synchronization method is applied to a first storage platform, and the method includes: Obtain N routing tables, wherein the N routing tables are obtained by dividing the total routing table according to the user identifier. Each routing table contains the user identifier and data storage information. The content recorded in the routing table is determined according to the user identifier. The data storage information represents the expected storage path of the user data matched by the user identifier in the first storage platform. N is a positive integer. Obtain the user data to be synchronized sent by the second storage platform, and divide the user data to be synchronized into N files according to the user identifier in the user data to be synchronized. The method used to divide the user data to be synchronized into N files is the same as the method used to divide the user identifier into N routing tables. Determine the routing table that matches each of the N files, and based on the data storage information in the routing table that matches each file, synchronize the user data in each file to the first storage platform in parallel. Determining the routing table matching each file in the N files includes: For the Pth file among the N files, the Pth routing table among the N routing tables is determined as the routing table that matches the Pth file, where P is less than or equal to N.

2. The method according to claim 1, characterized in that, The user identifier consists of numbers, and the N routing tables are obtained in the following way: Obtain the total routing table, wherein the total routing table contains all user identifiers recorded in the second storage platform, and contains the data storage information that matches all user identifiers; For each user identifier recorded in the total routing table, calculate the remainder when the user identifier is divided by N; Based on the remainder of each user identifier recorded in the routing table when divided by N, the user identifiers and data storage information recorded in the routing table are divided into the N routing tables.

3. The method according to claim 2, characterized in that, Based on the remainder when each user identifier recorded in the routing table is divided by N, the user identifiers and data storage information recorded in the routing table are divided into the N routing tables, including: For each user identifier recorded in the total routing table, if the remainder when the user identifier is divided by N is M, then the first target data is assigned to the Mth routing table among the N routing tables, wherein the first target data refers to the user identifier in the total routing table and the data storage information matching the user identifier; If the remainder when the user identifier is divided by N is 0, then the first target data is assigned to the Nth routing table among the N routing tables.

4. The method according to claim 2, characterized in that, The N is determined in one of the following ways: Obtain a preset first value, and determine the first value as N; A preset threshold is obtained. If the number of user identifiers in the total routing table is greater than or equal to the preset threshold, N is determined based on the number of user identifiers in the total routing table. If the number of user identifiers in the total routing table is less than the preset threshold, a preset second value is determined as N.

5. The method according to claim 2, characterized in that, The data storage information is determined in the following way: For each user identifier recorded in the total routing table, a general user identifier matching the user identifier is determined, wherein the user identifier is an identifier used to represent the user identity in the target business scenario, and the general user identifier is an identifier used to represent the user identity in all business scenarios; The general user identifier is hashed to obtain a hash value, and the data storage information matching the user identifier is determined based on the hash value.

6. The method according to claim 3, characterized in that, Based on the user identifier in the user data to be synchronized, the user data to be synchronized is divided into N files, including: For each user identifier in the user data to be synchronized, calculate the remainder when the user identifier is divided by N; If the remainder when the user identifier is divided by N is M, then the second target data is divided into the Mth file of the N files, wherein the second target data refers to the user data in the user data to be synchronized that matches the user identifier; If the remainder when the user identifier is divided by N is 0, then the second target data is divided into the Nth file among the N files.

7. A user data synchronization device, characterized in that, include: The first acquisition module is used to acquire N routing tables, wherein the N routing tables are obtained by dividing the total routing table according to the user identifier. Each routing table contains the user identifier and data storage information. The content recorded in the routing table is determined according to the user identifier. The data storage information represents the expected storage path of the user data matched by the user identifier in the first storage platform. N is a positive integer. The second acquisition module is used to acquire the user data to be synchronized sent by the second storage platform, and divide the user data to be synchronized into N files according to the user identifier in the user data to be synchronized. The method used to divide the user data to be synchronized into N files is the same as the method used to divide the user identifier into N routing tables. The synchronization module is used to determine the routing table that matches each of the N files, and synchronize the user data in each file to the first storage platform in parallel based on the data storage information in the routing table that matches each file. The synchronization module further includes a determination submodule, used to determine, for the Pth file in the N files, the Pth routing table in the N routing tables as the routing table that matches the Pth file, where P is less than or equal to N.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the user data synchronization method according to any one of claims 1 to 6 when it is run.

9. An electronic device, characterized in that, The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform operations on the programs, wherein the programs are configured to execute the user data synchronization method as described in any one of claims 1 to 6 at runtime.

Images