A data processing method, device and electronic equipment

Abstract

The application provides a data processing method and device and electronic equipment, wherein the method obtains a first data feature identifier and a target number of source data, constructs a data filtering condition based on the data feature identifier and the target number of target objects, and moves source data in a source object that meets a same data filtering condition to a same target object. Since the data feature identifier of the source data is a partition where the data is located in a database, and the number of target objects is determined in advance according to the data volume of the source data, the filtering condition constructed based on the data feature identifier and the target number can divide the data stored in the source object again, and move the source data meeting the same filtering condition to the same target object. In this way, the source data in the source object can be moved to different target objects after being scattered, which effectively ensures that the source data can be smoothly moved to the corresponding target objects for storage, and effectively improves the data migration success rate.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data processing method, apparatus and electronic device. Background Technology

[0002] In the field of database technology, data from a single database is typically split and stored across multiple databases based on actual needs. This distributes the load on individual databases and improves data reliability. In some application scenarios, there is a need to migrate data from the current database to a target database to expand the capacity of the original database. However, existing data transfer services only support complete database migration. This can lead to migration failures if the target database's capacity is insufficient to store the data from the migrated database. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method, apparatus, and electronic device to improve the success rate of data migration.

[0004] In a first aspect, embodiments of this application provide a data processing method, wherein the method includes:

[0005] Obtain the first data feature identifier and target number of the source data, wherein the data feature identifier is used to characterize the data partition in the source object where the source data is located, and the target number is used to characterize the number of target objects to which the source data is migrated, and the target number is predetermined based on the data volume of the source data;

[0006] Based on the first data feature identifier as the target data feature identifier and the number of targets, data filtering conditions are constructed;

[0007] Migrate source data that meet the same data filtering criteria from the source object to the same target object.

[0008] In conjunction with the first aspect, in a second possible embodiment, the source object includes a database or a data table, and the target object includes a database or a data table.

[0009] In conjunction with the first aspect, in a third possible embodiment, the step of constructing data filtering conditions based on the first data feature identifier as the target data feature identifier and the number of targets includes:

[0010] The data filtering conditions are obtained by taking the modulo of the target number using the target data feature identifier;

[0011] The step of migrating source data that meets the same data filtering conditions from the source object to the same target object includes:

[0012] Source data with the same modulo result are migrated to the same target object.

[0013] In conjunction with the first aspect, in a fourth possible embodiment, the target number is determined in advance based on the amount of source data and the expected amount of data.

[0014] In conjunction with the second to fourth possible embodiments of the first aspect, in the fifth possible embodiment, if there is a target object among the target objects whose access volume is greater than a preset access volume threshold, the second data feature identifier of the source data is determined, wherein the second data feature identifier is not a unique identifier of the source data;

[0015] Based on the second data feature identifier as the target data feature identifier and the number of targets, data filtering conditions are constructed;

[0016] Migrate source data that meet the same data filtering criteria from the source object to the same target object.

[0017] Secondly, embodiments of this application provide a data processing apparatus, wherein the apparatus includes:

[0018] The acquisition module is used to acquire the first data feature identifier and the target number of the source data, wherein the data feature identifier is used to characterize the data partition in the source object where the source data is located, and the target number is used to characterize the number of target objects to which the source data is migrated, and the target number is predetermined based on the amount of data in the source data;

[0019] The construction module is used to construct data filtering conditions based on the first data feature identifier being the target data feature identifier and the number of targets;

[0020] The data migration module is used to migrate source data that meets the same data filtering conditions from the source object to the same target object.

[0021] In conjunction with the second aspect, in a second possible embodiment, the source object includes a database or a data table, the target object includes a database or a data table, and the target number is determined in advance based on the amount of data in the source data and the expected amount of data.

[0022] In conjunction with the second aspect, in a third possible embodiment, the construction module is specifically used to take the modulo of the target number using the target data feature identifier to obtain the data filtering conditions; the data migration module is specifically used to migrate source data with the same modulo result to the same target object.

[0023] In conjunction with the second to third possible embodiments of the second aspect, in a fourth possible embodiment, the apparatus further includes:

[0024] The determination module is used to determine the second data feature identifier of the source data after the source data that meets the same data filtering conditions is migrated to the same target object. If there is a target object in each target object with an access volume greater than a preset access volume threshold, the second data feature identifier is not a unique identifier of the source data.

[0025] The construction module is also used to construct data filtering conditions based on the second data feature identifier as the target data feature identifier and the number of targets.

[0026] Thirdly, embodiments of this application provide an electronic device, comprising:

[0027] Processor; and

[0028] Stored program memory,

[0029] The program includes instructions that, when executed by the processor, cause the processor to perform the data processing method described in the first aspect.

[0030] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the data processing method described in the first aspect.

[0031] The beneficial technical effects of this application are:

[0032] This application provides a data processing method, apparatus, and electronic device. By obtaining a first data feature identifier and a target number of source data, and based on the first data feature identifier as the target data feature identifier and the target number of target objects, data filtering conditions are constructed, and source data that meet the same data filtering conditions in the source objects are migrated to the same target object.

[0033] In this embodiment, the source data's data feature identifier is determined by the partition it resides in the database. Since the number of target objects is predetermined based on the volume of the source data, the filtering criteria constructed based on the data feature identifier and the target number allow for further partitioning of the data stored in the source objects. Source data meeting the same filtering criteria are then migrated to the same target object. This ensures that even if the capacity of a single target object cannot support storing all the source data within it, the source data can be dispersed and migrated to different target objects, effectively guaranteeing that all source data can be successfully migrated to their corresponding target objects for storage, thus significantly improving the data migration success rate. Attached Figure Description

[0034] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 This is a schematic diagram of the data transmission process in the existing DTS technology;

[0036] Figure 2 This application provides a possible flowchart of a data processing method.

[0037] Figure 3 A schematic diagram of another possible data processing method provided in an embodiment of this application;

[0038] Figure 4 A possible logic diagram of a data processing apparatus provided in an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of a possible logical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0041] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0042] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0043] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0044] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0045] In the field of database technology, by splitting data from a single database and distributing it across multiple databases, the load on each database device can be balanced. This way, even if a single device fails, only a portion of the data will be affected, not all of it, thus ensuring uninterrupted data service. For example, suppose a real-world application requires storing 100 million order records in an order database. An empty database can only support 25 million records, which is insufficient for this requirement. Related technologies propose splitting these 100 million records into four empty databases, ensuring successful data storage. This way, even if one database fails, the remaining three can still provide data services, effectively guaranteeing data service quality.

[0046] In the database field, a data table is the basic unit for organizing data. Data in a database is organized into tables according to rows and columns. Each row (or column) in a table has a name, and each column (or row) has a specified data type and size. Therefore, splitting data essentially involves splitting data tables, which can be divided into two types of splitting:

[0047] Vertical partitioning: Isolating and storing data tables from the same database into different databases.

[0048] Horizontal partitioning: Data belonging to the same data table is split into several sub-data tables according to set partitioning rules, and then each sub-data table is stored in a different database. Each sub-data table is a partition.

[0049] For example, suppose a database is used to store student information, which includes a student personal information table and a student grade information table. The student personal information includes five different attributes: student name, student ID, student gender, student grade, and student class. The student personal information is organized such that each row has a name, and each column has a specified data type and size. The student personal information table can be shown in Table 1 below:

[0050] Table 1 Student Personal Information Form

[0051]

[0052]

[0053] In Table 1, each row describes a data instance. For example, the first row describes student Zhang San, whose student ID is 123456. This student is a male student in Class 2 of Grade 3. One column is the data feature column. Data in the same column have the same data attributes and data features. For example, the data attributes of "Zhang San", "Li Si", and "Zhao Wu" are all names, and the data feature is all character type.

[0054] The student grade information includes five attributes: student name, Chinese language score, math score, English score, and physical education score. The student grade information is organized with each row having a name and each column having a specified data type and size. The student grade information table is shown in Table 2 below.

[0055] Table 2 Student Academic Information

[0056] Name Chinese language score Math score English scores Sports performance Zhang San 90 98 95 80 Li Si 85 78 90 85 Zhao Wu 88 89 85 90

[0057] Vertical partitioning can be understood as storing Table 1 and Table 2 from a unified database into different databases. Horizontal partitioning can be understood as storing data from Table 1 and Table 2 that meet certain set conditions into the same database. For example, storing all information belonging to student Zhang San in the same database.

[0058] In related technologies, DTS (Data Transmission Service) tools are commonly used to move data in databases. For specific implementation logic, please refer to [reference needed]. Figure 1 The flowchart shown is as follows. Figure 1 As shown, the DTS tool migrates the entire database (including the hierarchical structure between databases) in the first shard to the second and third shards to achieve data migration. The relationship between the master table and the slave table in the moved database does not change.

[0059] As can be seen, the existing method of data migration using DTS tools is to migrate the entire data in the database without splitting the data tables in the database again. This migration method requires that the capacity of the target database or target shard to be migrated is greater than or equal to the size of the source data to be migrated. If the capacity of the target database or target shard is smaller than the size of the source data, data migration is likely to fail, which will ultimately affect the expansion effect of the database.

[0060] In view of this, firstly, embodiments of this application provide a data processing method for improving the success rate of data migration in a database. The data processing method provided in embodiments of this application can be applied to any electronic device with data splitting or data migration capabilities, including but not limited to personal mobile terminals, personal computers, and servers.

[0061] like Figure 2 As shown, the data processing method provided in this application includes the following steps:

[0062] S11. Obtain the first data feature identifier and target number of the source data;

[0063] Among them, the data feature identifier is used to characterize the data partition in the source object where the source data is located, and the target number is used to characterize the number of target objects to which the source data is migrated. The target number is determined in advance based on the amount of source data.

[0064] S12. Based on the first data feature identifier as the target data feature identifier and the number of targets, construct data filtering conditions;

[0065] S13. Migrate source data that meet the same data filtering conditions from the source object to the same target object.

[0066] In this embodiment of the application, since the data feature identifier is the partition where the data is located in the database and / or data table, and the number of target objects is predetermined based on the amount of source data, the filtering conditions constructed based on the data feature identifier and the number of targets can further divide the data stored in the source objects, and migrate the source data that meets the same filtering conditions to the same target object. In this way, even if the capacity of a single target object cannot support storing all the source data in the entire source object, the source data in the source object can be broken down and migrated to different target objects, effectively ensuring that all source data can be successfully migrated to the corresponding target object for storage, thus significantly improving the data migration success rate.

[0067] The following will provide a detailed explanation of steps S11-S13:

[0068] To distinguish between data before and after migration, this application embodiment refers to all data that has not yet been migrated as source data. Based on this, in step S11, the source object refers to the current storage location of the data, and the target object is the location of the data after migration. Since in the database field, each database consists of several data tables, data is stored macroscopically in the database and microscopically in the data tables within the database. Therefore, in one possible embodiment, the source object (or target object) can be the database where the data currently resides, or it can refer to the specific data table where the data currently resides within the database. Thus, by using the embodiments of this application, not only is database data migration supported, but also data migration of a specific data table within the database.

[0069] As described above regarding data tables, a data table is the product of organizing data by rows and columns. It can be split into different sub-data tables based on different data characteristics, each sub-data table corresponding to one or more partitions of the original data table. Therefore, in this embodiment, the data feature identifier indicates the specific partition in the source object where the data resides. This can refer to the partition of the source data in the source database, specifically the partition of the source data in the source data table. In practical applications, the data feature identifier can be the partition key of the data in the data table.

[0070] For example, taking Table 1 above as an example, the data feature identifier for the data "Zhang San" is "Name", indicating that the data "Zhang San" is located in the "Name" partition of the student information database. Specifically, this means that the data "Zhang San" is located in the "Name" partition of the student personal information table and the "Name" partition of the student grade information table in the student information database. Similarly, "Name" is a partition key for student information, and data with the same partition key are located in the same partition of the database.

[0071] In executing step S11, in one possible embodiment, the first data feature identifier and target number of the source data input by the user can be obtained through a user interface. In this case, the target number is the number of target objects required, calculated in advance by the user based on the amount of source data. In another possible embodiment, the target number may be the number of target objects required, calculated in advance by the user based on the expected amount of data. In another possible real-time scenario, the target number is determined in advance based on the amount of source data and the expected amount of data. For example, if the current amount of source data A is 10 million data entries, and it is expected that source data A will grow to 50 million data entries, then the number of target objects that can support the storage of 60 million data entries and the number of target objects can be determined based on the amount of source data and the expected amount of data.

[0072] In another possible embodiment, databases or tables with remaining available capacity less than a preset available capacity threshold can be identified as source objects based on database log information, and the data stored in these source objects can be identified as source data. In this case, the required number of target objects can be automatically calculated based on the total amount of source data. If the capacity of the target objects is greater than or equal to the amount of source data, it can be ensured that the source data can be completely migrated to the target objects.

[0073] In step S12, data filtering conditions are constructed based on the data feature identifiers of the source data and the number of target objects. This can be achieved by dividing the data feature identifiers of the source data into a target number of value intervals based on the number of target objects, thus obtaining the data filtering conditions. Based on this, in step S13, data whose data feature identifiers fall within the same value interval in the source objects can be migrated to the same target object.

[0074] In another possible embodiment, step S12 can involve using the data feature identifier of the source data to perform a modulo calculation on the target number to obtain the corresponding data filtering conditions. Based on this, in step S13, source data with the same modulo result can be migrated to the same target object. When the data feature identifier is a character type, the modulo calculation is based on the binary value of the data feature identifier. In this embodiment, since the data filtering conditions are obtained by performing a modulo calculation on the number of target objects based on the data feature identifier, the amount of source data belonging to each target object is relatively balanced, effectively ensuring the load balance of the database.

[0075] In one possible application scenario, the source object, target object, and data filtering conditions can be determined by obtaining the following content added by the user or operator in the database configuration file.

[0076] src":"source object",

[0077] "dst":"target object",

[0078] "type":"object type",

[0079] "where": "Data filtering criteria"

[0080] Specifically, the object type refers to whether the source object and the target object are of the type of database or data table.

[0081] If the data filtering criteria are obtained by taking the modulo of the target number using data feature identifiers, then the data filtering criteria can be:

[0082] Data feature identifier > 0 and data feature identifier % target number

[0083] If the source object is a database, the target object is also a database, the data feature identifier is the customer's identity ID, and the target number is 4, then the above content should be modified accordingly:

[0084] {

[0085] "src":"database1",

[0086] "dst":"database2",

[0087] "type":"database",

[0088] "where":"customer_id>0and customer_id%4=0"

[0089] }

[0090] In this way, all data in the source database database1 that has a modulo result of 0 can be moved to the corresponding database database2.

[0091] If the source object is a data table, the target object is also a data table, the data feature identifier is still the customer's ID, and the target number is 4, then the above content should be modified accordingly:

[0092] {

[0093] "src":"table1",

[0094] "dst":"table2

[0095] "type":"table

[0096] "where":"customer_id>0and customer_id%4=0"

[0097] }

[0098] In this way, all data in the source data table table1 that has a modulo result of 0 can be moved to the corresponding data table table2.

[0099] Alternatively, in one possible embodiment, the amount of source data in the source object can be automatically determined based on the source object and target object added by the user, and data filtering conditions can be automatically set according to the amount of source data and the number of target objects.

[0100] If, after all tables in a database have been horizontally partitioned and stored in different databases, the total amount of data in each of the horizontally partitioned tables still exceeds a preset data volume threshold, then steps S11-S13 can be performed to perform a second partitioning of each horizontally partitioned table to achieve multi-table splitting. For example, table A in the source database is horizontally partitioned into tables B, C, D, and E. At this point, the total amount of data in tables B, C, D, and E still exceeds the preset data volume threshold, resulting in a large amount of concurrent data processing during data access, which is detrimental to data retrieval. In this case, tables B, C, D, and E can be further partitioned according to steps S11-S13. Table B can be further partitioned into N tables (i.e., table B1, table B2, table B3, ..., table BN, where N = 2). n (where n is an integer greater than 0).

[0101] In one possible embodiment, the configuration file in the database can be automatically set according to the determined number N of target data tables. The data tables of the source objects in the configuration file are modified to the corresponding source data tables, and the data tables of the target objects in the configuration file are modified to the corresponding target data tables. This configures the task list of the data transmission service, enabling the data transmission service to migrate data from the source data tables to the target data tables. Thus, the data transmission service provided in this embodiment can adapt to different scenarios of data table / database splitting, enabling the splitting of a single data table into multiple data tables, the splitting of multiple data tables within the same database into multiple data tables, and the splitting of data belonging to the same data feature identifier in different databases into multiple data tables.

[0102] After source data meeting the same data filtering criteria are migrated to the same target object, it means that all operations on the data in the source object are also migrated to the target object. That is, subsequent reads and writes of the source data all occur in the target object. Furthermore, in the field of database technology, data reading and writing are performed based on the data's partition key as the index. That is, data is written to the database according to the data's partition key, or the data is read from the database according to the data's partition key. In this embodiment, the data feature identifier represents the partition where the data is located in the source / target object; its essence is also the data's partition key, thus the data feature identifier serves as the index for data reading and writing.

[0103] In this embodiment, the data filtering conditions are constructed using data feature identifiers and target numbers. This can be achieved by taking the modulo of the target number with the data feature identifiers, or by dividing the data feature identifiers into numerical ranges based on the target number. This means that if the numerical distribution of the first data feature identifier of the source data is relatively concentrated, the data satisfying the same data filtering conditions will still be relatively concentrated after dividing or taking the modulo of the numerical range. This can easily lead to a large data access volume for target objects storing the same data filtering conditions during subsequent data read / write operations, which is not conducive to responding to user data read / write requests.

[0104] Based on this, in one possible embodiment, such as Figure 3 As shown, after source data that meets the same data filtering criteria is migrated to the same target object, the data processing method provided in this application further includes:

[0105] S14. Determine if there is a target object among the target objects whose access volume exceeds the preset access volume threshold. If yes, proceed to step S15; otherwise, end the process.

[0106] S15. Determine the second data feature identifier of the source data;

[0107] S16. Based on the second data feature identifier as the target data feature identifier and the target number, construct data filtering conditions; and return to the execution step S13.

[0108] Among them, the first data feature identifier and the second data feature identifier are not unique identifiers of the source data.

[0109] In one possible embodiment, when performing step S14, after the source data is migrated to the target object, a data access request test can be conducted to test the time consumed by each target object in responding to the access request. If there is a target object whose time consumption is greater than the preset response time, it indicates that there is a target object whose access volume is greater than the preset access volume threshold.

[0110] In another possible embodiment, after the source data is migrated to the target objects, a work status query request can be sent to each target object. Upon receiving the work status query request, each target object will report the number of currently pending data read / write requests. If the number of currently pending data read / write requests for any target object exceeds a preset read / write request threshold, then the access volume of that target object is determined to be greater than the preset access volume threshold. The specific method for determining the target objects whose access volume exceeds the preset access volume threshold can be flexibly selected based on actual experience or actual operation, and this application does not impose strict limitations.

[0111] In this embodiment, the unique identifier of the source data is used to characterize the uniqueness of the source data and can be used to guarantee the uniqueness of the source data. For example, if the unique identifier of the source data is a unique id, then querying the database for the unique id of source data A will yield only one query result: source data A. In contrast to the unique identifier, the data feature identifier of the source data is a subfield of the unique identifier of the source data. It can characterize the data feature attributes possessed by the source data but cannot guarantee the uniqueness of the source data.

[0112] After migrating the source data to the target objects, it's equivalent to splitting the original single data table into multiple target objects based on data filtering conditions constructed using the first data feature identifier. At this point, if the data filtering conditions constructed using the first data feature identifier result in a target object with high access volume among all target objects, it indicates that the current data table splitting method is placing significant access pressure on the high-access target objects. Therefore, it's necessary to split the original data table according to a new splitting method. Since data migration essentially involves copying source data from the source object to the target object for backup, if both the first and second data feature identifiers are unique identifiers for the source data, it will result in two data entries having the same unique identifier in the entire database. This contradicts the principle that unique identifiers guarantee data uniqueness, potentially leading to data migration failure.

[0113] Based on this, when performing step S15, in one possible embodiment, the unique identifier of the source data is obtained. Since the data feature identifier of the source data is a subfield of the unique identifier of the source data, other subfields in the subfield of the unique identifier of the source data other than the first data feature identifier can be determined as the second data feature identifier of the source data.

[0114] In another possible embodiment, the partition keys of the source data table can be obtained, wherein the partition key corresponding to the first data feature identifier is the first partition key, and any other partition key in the source data table besides the first partition key is determined as the second data feature identifier. Throughout the process, a filter whitelist can be set to automatically add the unique identifier of the source data and partition keys that have been used and do not meet the access volume requirements to the filter whitelist, so that the second data feature identifier can be determined from the unused partition keys later.

[0115] In this embodiment of the application, if there are target objects whose access volume exceeds a preset access volume threshold, it means that the data filtering conditions constructed according to the first data feature identifier of the source data result in one or more target objects experiencing a large access volume. This indicates that the value distribution of the first data feature identifier of each source data is relatively concentrated. In this case, using the first data feature identifier to construct the data filtering conditions cannot meet the normal read / write needs of the target objects. Therefore, a second data feature identifier can be redefined for constructing the data filtering conditions. The second data feature identifier is different from the first data feature identifier. If the second data feature identifier is used to construct the data filtering conditions, and its value distribution is more uniform than that of the first data feature identifier, then the source data can be migrated more evenly to each target object. This allows each target object to respond to user access needs in a balanced manner, providing a smoother data service.

[0116] Since data migration essentially involves copying source data from a source object to a target object for backup, in this embodiment, after source data meeting the same data filtering criteria is migrated to the target object, existing data transmission service tools can be used to compare the data in the source and target objects to verify their consistency. If they are inconsistent, an alarm message can be sent to notify the user or operator of the data migration error. If they are consistent, the data access port or interface can be changed from the source object's access interface to the target object's access port or interface.

[0117] For example, taking a scenario where both the source and target objects are databases, if the data in the target database being migrated to is identical to the data in the source database being migrated out, then the access ports (or interfaces) for each data item are changed from the source database's access port (or interface) to the target database's access port (or interface). Specifically, in one possible embodiment, the access ports (or interfaces) for each data item in the database's configuration file can be manually changed according to the access ports (or interfaces) of the target database to which the data is being migrated.

[0118] In another possible embodiment, domain name resolution can be used. By sending a data access request, the data characteristic identifier corresponding to the partition is obtained. Based on the received data packet information, domain name resolution is performed to determine the domain name of the database where each data resides. The domain name that is distinct from the source database is determined as the domain name of the target database. The access information of the target object is then modified according to the determined target database domain name. Using this embodiment, the database corresponding to the data can be automatically changed, simplifying the database port change process and improving the efficiency of database data migration.

[0119] Secondly, embodiments of this application provide a data processing apparatus, such as... Figure 4 As shown, the data processing device 40 includes:

[0120] The acquisition module 401 is used to acquire the first data feature identifier and the target number of the source data. The data feature identifier is used to characterize the data partition in the source object where the source data is located, and the target number is used to characterize the number of target objects to which the source data is migrated. The target number is predetermined based on the amount of source data.

[0121] Module 402 is used to construct data filtering conditions based on the first data feature identifier as the target data feature identifier and the number of targets;

[0122] The data migration module 403 is used to migrate source data that meets the same data filtering conditions from the source object to the same target object.

[0123] In one possible embodiment, the source object includes a database or a data table, and the target object includes a database or a data table.

[0124] In one possible embodiment, the construction module 402 is specifically used to take the modulo of the target number using the target data feature identifier to obtain the data filtering conditions; the data migration module 403 is specifically used to migrate source data with the same modulo result to the same target object based on the data filtering conditions.

[0125] In one possible embodiment, the determining module 404 is used to determine the second data feature identifier of the source data after the source data that meets the same data filtering conditions is migrated to the same target object, if there is a target object in each target object with an access volume greater than a preset access volume threshold, wherein the second data feature identifier is not a unique identifier of the source data.

[0126] The execution module 405 is used to return the steps of constructing data filtering conditions based on the target data feature identifier and the number of targets.

[0127] An exemplary embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this application.

[0128] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0129] An exemplary embodiment of this application also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of this application.

[0130] refer to Figure 5 The present invention describes a structural block diagram of an electronic device 500 that can serve as a server or client of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0131] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the device 50. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0132] Multiple components in electronic device 50 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 50. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 504 may include, but is not limited to, disks and optical discs. Communication unit 509 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0133] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the aforementioned data processing methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 50 via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the data processing methods described in the first aspect by any other suitable means (e.g., by means of firmware).

[0134] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the first data feature identifier and target number of the source data, wherein the data feature identifier is used to characterize the data partition in the source object where the source data is located, and the target number is used to characterize the number of target objects to which the source data is migrated, and the target number is predetermined based on the data volume of the source data; the source object includes: a data table, and the target object includes: a data table; Based on the first data feature identifier as the target data feature identifier and the number of targets, data filtering conditions are constructed, including: The data filtering conditions are obtained by taking the modulo of the binary value of the target data feature identifier with the target number; Migrating source data that meets the same data filtering criteria from the source object to the same target object includes: Migrate source data with the same modulo result to the same target object; After the source data that meets the same data filtering conditions is migrated to the same target object, if there is a target object in each target object with an access volume greater than a preset access volume threshold, the second data feature identifier of the source data is determined, wherein the second data feature identifier is not a unique identifier of the source data; Based on the second data feature identifier as the target data feature identifier and the number of targets, data filtering conditions are constructed; Migrate source data that meet the same data filtering criteria from the source object to the same target object.

2. The method according to claim 1, characterized in that, The target number is determined in advance based on the amount of source data and the expected amount of data.

3. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first data feature identifier and the target number of the source data. The data feature identifier is used to characterize the data partition in the source object where the source data is located, and the target number is used to characterize the number of target objects to which the source data is migrated. The target number is predetermined based on the amount of data in the source data. The source object includes a database or a data table, and the target object includes a database or a data table. The construction module is used to construct data filtering conditions based on the first data feature identifier as the target data feature identifier and the number of targets, including: The data filtering conditions are obtained by taking the modulo of the binary value of the target data feature identifier with the target number; The data migration module is used to migrate source data with the same modulo result to the same target object; The determination module is used to determine the second data feature identifier of the source data after source data that meets the same data filtering conditions is migrated to the same target object. If there is a target object in each target object with an access volume greater than a preset access volume threshold, the second data feature identifier is not a unique identifier of the source data. The construction module is also used to construct data filtering conditions based on the second data feature identifier as the target data feature identifier and the number of targets.

4. The apparatus according to claim 3, characterized in that, The target number is determined in advance based on the amount of source data and the expected amount of data.

5. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-2.

6. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-2.

Images