An electronic device and a data processing method

Abstract

The present application discloses an electronic device and a data processing method, and relates to the field of big data computing technology. The electronic device of this embodiment includes a processor and a memory; the processor is used to: determine the target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit; select the target database key based on the data field value of each database key in the target data unit; divide the data field value of the target database key into a target number of subfields, and determine the sub-identifier of each subfield, and process the subfield corresponding to each sub-identifier through different tasks. This embodiment reduces the occurrence of data tilt by reasonably allocating computing resources without increasing computing resources, and ensures the efficiency and stability of the computing process.

Description

Technical Field

[0001] The present application relates to the field of big data computing technology, and in particular to an electronic device and a data processing method. Background Art

[0002] During big data computing, a large amount of data is concentrated on one or several machines for computing. The computing speed of these data is much lower than the average computing speed, which causes the entire computing process to be too slow. This situation is called data skew.

[0003] In related technologies, the occurrence of data skew during big data computing can be reduced by increasing computing resources.

[0004] However, the above method will cause waste of resources. Summary of the invention

[0005] The present application provides an electronic device and a data processing method for reducing the occurrence of data skew during big data calculation while reducing resource waste.

[0006] In a first aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0007] The memory stores a program code, and when the program code is executed by the processor, the processor performs the following process:

[0008] Determine the target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit;

[0009] Selecting a target database key based on the data field values ​​of each database key in the target data unit;

[0010] The data field value of the target database key is divided into a target number of subfields, and a sub-identifier of each subfield is determined, and the subfield corresponding to each sub-identifier is processed through different tasks.

[0011] In the above scheme, the number of database keys and data field values ​​in the data unit characterizes the overall distribution of the data field values ​​in the corresponding data unit, based on which the target data unit that may contain skewed data can be efficiently determined; then, based on the data field values ​​of each database key in the target data unit, the target database key corresponding to the skewed data can be accurately selected from the target data unit; by dividing the data field values ​​of these target database keys into multiple sub-fields and determining the sub-identifier of each sub-field, the sub-field corresponding to each sub-identifier is processed through different tasks. Without increasing computing resources, the computing resources are reasonably allocated to reduce the occurrence of data skew and ensure the efficiency and stability of the computing process.

[0012] In some optional implementation manners, the processor is specifically configured to:

[0013] A data unit whose number of database keys is less than a preset number and whose data field value is greater than a first preset data field value is determined as the target data unit.

[0014] In the above scheme, since in the data units where the number of database keys is less than the preset number and the data field value is greater than the first preset data field value, a large number of data field values ​​are concentrated in a small number of database keys and are likely to contain skewed data, these data units can be determined as target data units.

[0015] In some optional implementation manners, the processor is specifically configured to:

[0016] For any database key in the target data unit, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than a second preset data field value, the database key is determined as the target database key; wherein the target queue is the queue with the largest remaining data field value in the queue group; the remaining data field value of any queue is the difference between the upper limit of the data field value addition of the queue and the data field value of the database key added to the queue; the queue group includes at least one queue for adding database keys.

[0017] In the above scheme, since the target queue is the queue with the strongest adding capability in the queue group, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than the second preset data field value, it means that the sum of the two is too large and the database key cannot be added to the target queue, which indirectly indicates that the data field value of the database key is large. Therefore, the database key is determined as the target database key.

[0018] In some optional implementations, after determining the database key as the target database key, the processor is further configured to:

[0019] A new queue is created in the queue group, and the target database key is added to the new queue.

[0020] In the above solution, since the target database key cannot be added to the existing queue, a new queue is created in the queue group to add the target database key to the queue.

[0021] In some optional implementations, before determining the database key as the target database key, the processor is further configured to:

[0022] Determine that the remaining data field value in the target queue is greater than a third preset data field value; wherein the third preset data field value is less than the second preset data field value.

[0023] The above scheme ensures that the data field value of the added database key in the target queue will not be too large by first determining that the remaining data field value in the target queue is greater than the third preset data field value. In this way, when the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than the second preset data field value, it can be more accurately determined as the target database key.

[0024] In some optional implementation manners, the processor is specifically configured to:

[0025] If there is an associated database key in any data unit other than any target data unit, the associated database key is expanded in the other data unit to obtain the target number of expanded database keys; wherein the associated database key is a database key having the same initial identifier as the target database key in the target data unit;

[0026] After merging the subfields of the target database key into the expanded database keys respectively, the subfields corresponding to the sub-identifiers are processed through different tasks;

[0027] The multiple processing results corresponding to the same initial identifier are processed collectively to obtain the target processing result.

[0028] In the above scheme, when there are associated database keys in other data units other than the target data unit, the associated database keys are expanded, and the subfields of the target database keys are respectively merged into the expanded database keys, and the subfields corresponding to each sub-identifier are processed through different tasks; and then the processing results of multiple associations corresponding to the same initial identifier obtained by processing are collectively processed, so that when there are associated database keys in other data units, the final target processing result can be accurately obtained.

[0029] In a second aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0030] Determine the target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit;

[0031] Selecting a target database key based on the data field values ​​of each database key in the target data unit;

[0032] The data field value of the target database key is divided into a target number of subfields, and a sub-identifier of each subfield is determined, and the subfield corresponding to each sub-identifier is processed through different tasks.

[0033] In some optional implementations, determining the target data unit based on the number of database keys and data field values ​​in each data unit includes:

[0034] A data unit whose number of database keys is less than a preset number and whose data field value is greater than a first preset data field value is determined as the target data unit.

[0035] In some optional implementations, selecting a target database key based on a data field value of each database key in the target data unit includes:

[0036] For any database key in the target data unit, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than a second preset data field value, the database key is determined as the target database key; wherein the target queue is the queue with the largest remaining data field value in the queue group; the remaining data field value of any queue is the difference between the upper limit of the data field value addition of the queue and the data field value of the database key added to the queue; the queue group includes at least one queue for adding database keys.

[0037] In some optional implementation manners, after determining the database key as the target database key, the method further includes:

[0038] A new queue is created in the queue group, and the target database key is added to the new queue.

[0039] In some optional implementation manners, before determining the database key as the target database key, the method further includes:

[0040] Determine that the remaining data field value in the target queue is greater than a third preset data field value; wherein the third preset data field value is less than the second preset data field value.

[0041] In some optional implementations, the subfields corresponding to each sub-identifier are processed by different tasks, including:

[0042] If there is an associated database key in any data unit other than any target data unit, the associated database key is expanded in the other data unit to obtain the target number of expanded database keys; wherein the associated database key is a database key having the same initial identifier as the target database key in the target data unit;

[0043] After merging the subfields of the target database key into the expanded database keys respectively, the subfields corresponding to the sub-identifiers are processed through different tasks;

[0044] The multiple processing results corresponding to the same initial identifier are processed collectively to obtain the target processing result.

[0045] In a third aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0046] A data unit determination module, configured to determine a target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit;

[0047] A database key selection module, configured to select a target database key based on the data field value of each database key in the target data unit;

[0048] The field processing module is used to divide the data field value of the target database key into a target number of subfields, determine the sub-identifier of each subfield, and process the subfield corresponding to each sub-identifier through different tasks.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any one of the second aspects above.

[0050] In addition, the technical effects brought about by any implementation method in the second to fourth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A database key schematic diagram provided for an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a flow chart of a first data processing method provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a second data processing method provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a first queue group provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of a second queue group provided in an embodiment of the present application;

[0057] Figure 6 A flowchart of a method for processing a subfield provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the merger between the subfield and the expanded database key provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of a data processing device provided in an embodiment of the present application;

[0060] Fig. 9 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0062] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0063] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two devices. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0064] During big data computing (such as big data computing of white goods), a large amount of data is concentrated on one or several machines for computing. The computing speed of these data is much lower than the average computing speed, which makes the entire computing process too slow. This situation is called data skew.

[0065] See also Figure 1 As shown in the figure, there are n database keys in the big data platform (hive); among them, the data field value of database key key1 is relatively large, and the data field value of database keys key2 to key n The data field value of key1 is small, and the machine corresponding to key1 has to bear huge processing pressure. After other machines finish calculating, they have to wait for the machine corresponding to key1 to finish calculating. Therefore, when data skew occurs, the computing efficiency is low.

[0066] Some embodiments reduce the occurrence of data skew by increasing computing resources, such as increasing the number of machines corresponding to key1 so that more machines can compute the data of key1.

[0067] However, the above key2~key n The corresponding machines are relatively idle, and computing resources are not well utilized, resulting in a waste of resources.

[0068] In view of this, an embodiment of the present application proposes an electronic device and a data processing method, which determines a target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit; based on the data field values ​​of each database key in the target data unit, a target database key is selected; the data field value of the target database key is divided into a target number of subfields, and a sub-identifier of each subfield is determined, and the subfield corresponding to each sub-identifier is processed through different tasks.

[0069] In the above scheme, the number of database keys and data field values ​​in the data unit characterizes the overall distribution of the data field values ​​in the corresponding data unit, based on which the target data unit that may contain skewed data can be efficiently determined; then, based on the data field values ​​of each database key in the target data unit, the target database key corresponding to the skewed data can be accurately selected from the target data unit; by dividing the data field values ​​of these target database keys into multiple sub-fields and determining the sub-identifier of each sub-field, the sub-field corresponding to each sub-identifier is processed through different tasks. Without increasing computing resources, the computing resources are reasonably allocated to reduce the occurrence of data skew and ensure the efficiency and stability of the computing process.

[0070] The following will be combined with the accompanying drawings and specific embodiments to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0071] The present application embodiment provides a first data processing method, such as Figure 2 As shown, the following steps are included:

[0072] Step S201: Determine the target data unit based on the number of database keys and data field values ​​in each data unit.

[0073] The data field value of the data unit is the sum of the data field values ​​of the database key in the data unit.

[0074] The above database key includes an initial identifier and a data field value, wherein the initial identifier is information representing the database key (key ID), and the data field value is the content of the database key (key value). In implementation, different data units may include database keys with the same initial identifier and different data field values.

[0075] The above data unit may be a resilient distributed dataset (RDD).

[0076] In practice, all data units contain a large number of database keys, and directly determining whether each database key corresponds to skewed data will take a lot of time. Since the number of data units is small, and the number of database keys and data field values ​​in each data unit characterize the overall distribution of data field values ​​in the corresponding data unit, by determining the target data unit that may contain skewed data, the efficiency of determining the target database key corresponding to the skewed data can be improved.

[0077] In this embodiment, if the data field values ​​of a data unit are concentrated in a small number of database keys, the data unit is likely to contain skewed data; conversely, if the data field values ​​of a data unit are dispersed in a large number of database keys, the data unit is likely not to contain skewed data.

[0078] Step S202: Select a target database key based on the data field value of each database key in the target data unit.

[0079] In this embodiment, the target data unit may contain database keys corresponding to some non-skewed data. Therefore, it is necessary to determine which database keys are the target database keys corresponding to the skewed data based on the data field values ​​of each database key in the target data unit.

[0080] Step S203: Divide the data field value of the target database key into a target number of subfields, determine the sub-identifier of each subfield, and process the subfield corresponding to each sub-identifier through different tasks.

[0081] The data field value of the target database key is relatively large, and the data field value of the target database key needs to be split. In this embodiment, the data field value of the target database key is divided into a target number of sub-fields, so that the sub-fields corresponding to each sub-identifier are processed by different tasks, that is, one target database key is processed by multiple tasks, which reduces the processing pressure of the machine corresponding to the target database key, thereby reducing the occurrence of data skew.

[0082] Exemplarily, the sub-identifier of the above-mentioned subfield can be obtained based on the initial identifier of the target database key and the serial number of the subfield in the data field value, such as: the sub-identifier of the subfield is the initial identifier_the serial number of the subfield in the data field value, or the sub-identifier of the subfield is the serial number of the subfield in the data field value_the initial identifier.

[0083] The above-mentioned method for determining the sub-identifier of the sub-field is only an exemplary description and is not specifically limited in this embodiment.

[0084] In some optional implementations, the target quantity is determined based on the data field value of the target database key, that is, different target database keys correspond to different target quantities. For example: the larger the data field value of the target database key, the larger the target quantity, and the target database key is processed by more tasks; the smaller the data field value of the target database key, the smaller the target quantity, and the target database key is processed by fewer tasks, thereby making the allocation of computing resources more reasonable.

[0085] In the above scheme, the number of database keys and data field values ​​in the data unit characterizes the overall distribution of the data field values ​​in the corresponding data unit, based on which the target data unit that may contain skewed data can be efficiently determined; then, based on the data field values ​​of each database key in the target data unit, the target database key corresponding to the skewed data can be accurately selected from the target data unit; by dividing the data field values ​​of these target database keys into multiple sub-fields and determining the sub-identifier of each sub-field, the sub-field corresponding to each sub-identifier is processed through different tasks. Without increasing computing resources, the computing resources are reasonably allocated to reduce the occurrence of data skew and ensure the efficiency and stability of the computing process.

[0086] In some optional implementation manners, the above step S201 may be implemented by but not limited to the following methods:

[0087] A data unit whose number of database keys is less than a preset number and whose data field value is greater than a first preset data field value is determined as the target data unit.

[0088] Exemplarily, if the number of database keys in a data unit is less than a preset number, it means that the number of database keys in the data unit is small. If the data field value of a data unit is greater than a first preset data field value, it means that the sum of the data field values ​​of the database keys of the data unit is large. These data units not only contain fewer database keys, but also correspond to larger data field values, that is, a large number of data field values ​​are concentrated in a small number of database keys. The data unit is likely to contain skewed data. Based on this, these data units can be determined as target data units.

[0089] In implementation, the above-mentioned preset number and the first preset data field value can be set according to the actual application scenario, and this embodiment does not make any specific limitation on this.

[0090] In the above scheme, since in the data units where the number of database keys is less than the preset number and the data field value is greater than the first preset data field value, a large number of data field values ​​are concentrated in a small number of database keys, it is very likely to contain skewed data. Therefore, these data units can be determined as target data units.

[0091] The present application embodiment provides a second data processing method, such as Figure 3 As shown, the following steps are included:

[0092] Step S301: Determine the target data unit based on the number of database keys and data field values ​​in each data unit.

[0093] The data field value of the data unit is the sum of the data field values ​​of the database key in the data unit.

[0094] The step S301 may refer to the above step S201 and will not be described in detail here.

[0095] Step S302: For any database key in the target data unit, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than a second preset data field value, the database key is determined as the target database key.

[0096] Among them, the target queue is the queue with the largest remaining data field value in the queue group; the remaining data field value of any queue is the difference between the upper limit of adding the data field value of the queue and the data field value of the database key added to the queue; the queue group includes at least one queue for adding database keys.

[0097] In this embodiment, a queue group is provided, and the queue group includes at least one queue for adding database keys; the target database key is determined by sequentially adding each database key in the target data unit to the queue. Exemplary:

[0098] Before adding a database key to a queue, first determine the difference between the upper limit of the data field value addition of each queue in the queue group and the data field value of the database key already added to the queue (i.e., the remaining data field value of each queue). The remaining data field value of each queue represents its adding capability. The more remaining data field values ​​a queue has, the stronger its adding capability is. Therefore, it is only necessary to determine whether the database key can be added to the target queue based on the data field value of the database key and the data field value of the added database key in the queue with the largest remaining data field value (i.e., the target queue). If the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than the second preset data field value, it means that the sum of the two is too large and the database key cannot be added to the target queue, which indirectly indicates that the data field value of the database key is large, so the database key is determined as the target database key.

[0099] This embodiment does not specifically limit the second preset data field value. In some specific implementations, the second preset data field value is less than an upper limit added to the data field values ​​of all queues.

[0100] See also Figure 4 As shown, when adding database key 1 to the queue, there are queue 1, queue 2 and queue 3 in the queue group. Among these three queues, the remaining data field value of queue 1 is the largest, so queue 1 is the target queue;

[0101] The sum of the data field value of database key 1 and the data field value of the added database key in queue 1 is greater than the second preset data field value, database key 1 is the target database key, and database key 1 cannot be added to the current queue of the queue group.

[0102] The above example is only to more clearly illustrate the process of determining the target database key, but the present embodiment is not limited thereto.

[0103] It can be understood that if the sum of the data field value of the database key and the data field value of the added database key in the target queue is not greater than the second preset data field value, the database key is not the target database key and is added to the target database.

[0104] Step S303: Divide the data field value of the target database key into a target number of subfields, determine the sub-identifier of each subfield, and process the subfield corresponding to each sub-identifier through different tasks.

[0105] The step S303 may refer to the above step S203 and will not be described in detail here.

[0106] In the above scheme, since the target queue is the queue with the strongest adding capability in the queue group, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than the second preset data field value, it means that the sum of the two is too large and the database key cannot be added to the target queue, which indirectly indicates that the data field value of the database key is large. Therefore, the database key is determined as the target database key.

[0107] As described above, the target database key cannot be added to the current queue of the queue group. Based on this, in some optional implementations, after the above step S302, the following steps are further included:

[0108] A new queue is created in the queue group, and the target database key is added to the new queue.

[0109] Or the above Figure 4 For example, after setting database key 1 as the target database key, create queue 4 in the queue group and add database key 1 to queue 4;

[0110] See also Figure 5 As shown, when other database keys are added to the queue later, there will be four queues in the queue group: queue 1, queue 2, queue 3, and queue 4.

[0111] The above example is only to more clearly illustrate how to add the target database key to the queue, but this embodiment is not limited thereto.

[0112] In the above solution, since the target database key cannot be added to the existing queue, a new queue is created in the queue group to add the target database key to the queue.

[0113] During implementation, it may happen that the data field values ​​of the added database keys in all queues of the queue group are relatively large. At this time, the remaining data field value in the target queue itself is not large, and the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than the second preset data field value. This may not be caused by the larger data field value of the database key, but by the larger data field value of the added database key in the target queue.

[0114] Based on this, in some optional implementation manners, before the above step S302, the following steps are also included:

[0115] Determine that the remaining data field value in the target queue is greater than a third preset data field value; wherein the third preset data field value is less than the second preset data field value.

[0116] The above scheme ensures that the data field value of the added database key in the target queue will not be too large by first determining that the remaining data field value in the target queue is greater than the third preset data field value. In this way, when the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than the second preset data field value, it can be more accurately determined as the target database key.

[0117] As described above, different data units may include database keys with the same initial identifier but different data field values. Therefore, in some embodiments, other data units other than the target data unit may have associated database keys (the associated database keys are database keys with the same initial identifier as the target database key in the target data unit).

[0118] In view of this scenario, the embodiment of the present application provides a method for processing subfields, such as Figure 6 As shown, the following steps are included:

[0119] Step S601: Expand the associated database keys in the other data units to obtain the target number of expanded database keys.

[0120] In implementation, there are associations between database keys with the same initial identifier, and these database keys need to be associated. By expanding the associated database key in the data unit containing the associated database key, the expanded database keys with the same number of subfields as the corresponding target database key are obtained.

[0121] Step S602: After merging the subfields of the target database key into the expanded database keys respectively, the subfields corresponding to the sub-identifiers are processed through different tasks.

[0122] As described above, the number of expanded database keys is the same as the number of subfields of the corresponding target database key, and each subfield can be merged into the corresponding expanded database key.

[0123] In some optional implementations, the identifier of the expanded database key may correspond to the sub-identifier of the sub-field, thereby enabling efficient and accurate merging.

[0124] Step S603: performing collective processing on multiple processing results corresponding to the same initial identifier to obtain a target processing result.

[0125] In implementation, the corresponding processing result is obtained by processing the associated database key, and the corresponding processing result is also obtained by processing the subfield, that is, there are multiple processing results corresponding to the same initial identifier. There is an association between database keys with the same initial identifier, and the target processing result is obtained by collectively processing multiple processing results corresponding to the same initial identifier.

[0126] See also Figure 7 As shown, a specific example is given below:

[0127] Data unit 1 is the target data unit. The initial identifier of the target database key 1 in data unit 1 is A1. The data field value is C0, which is divided into three subfields, subfield C1, subfield C2 and subfield C3. The subidentifier corresponding to subfield C1 is B1_A1; the subidentifier corresponding to subfield C2 is B2_A1; and the subidentifier corresponding to subfield C3 is B3_A1.

[0128] In data unit 2, there is an associated database key with an initial identifier of A1 and a data field value of C4. The associated database key is expanded to obtain three expanded database keys: expanded database key 1 with an identifier of B1_A1 and a data field value of C4; expanded database key 2 with an identifier of B2_A1 and a data field value of C4; and expanded database key 3 with an identifier of B3_A1 and a data field value of C4.

[0129] Subfields C1 to C3 are respectively joined to the corresponding expanded database key, that is, subfield C1 is joined to the expanded database key 1 of the same identifier, subfield C2 is joined to the expanded database key 2 of the same identifier, and subfield C3 is joined to the expanded database key 3 of the same identifier;

[0130] Then, the subfield C1 is processed through different tasks to obtain the processing result D1; the subfield C2 is processed to obtain the processing result D2; the subfield C3 is processed to obtain the processing result D3; the data field value C4 is processed to obtain the processing result D4.

[0131] The four processing results D1 to D4 corresponding to the same initial identifier A1 are subjected to union processing to obtain the target processing result.

[0132] It can be understood that the above examples are only intended to more clearly illustrate how to obtain the final target processing result, and are not intended to limit the present application.

[0133] In the above scheme, when there are associated database keys in other data units other than the target data unit, the associated database keys are expanded, and the subfields of the target database keys are respectively merged into the expanded database keys, and the subfields corresponding to each sub-identifier are processed through different tasks; and then the processing results of multiple associations corresponding to the same initial identifier obtained by processing are collectively processed, so that when there are associated database keys in other data units, the final target processing result can be accurately obtained.

[0134] like Figure 8 As shown, based on the same inventive concept, an embodiment of the present application provides a data processing device 800, including: a data unit determination module 801, a database key selection module 802 and a field processing module 803.

[0135] The data unit determination module 801 is used to determine the target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit;

[0136] A database key selection module 802, configured to select a target database key based on the data field value of each database key in the target data unit;

[0137] The field processing module 803 is used to divide the data field value of the target database key into a target number of subfields, determine the sub-identifier of each subfield, and process the subfield corresponding to each sub-identifier through different tasks.

[0138] In some optional implementation manners, the data unit determination module 801 is specifically configured to:

[0139] A data unit whose number of database keys is less than a preset number and whose data field value is greater than a first preset data field value is determined as the target data unit.

[0140] In some optional implementations, the database key selection module 802 is specifically used to:

[0141] For any database key in the target data unit, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than a second preset data field value, the database key is determined as the target database key; wherein the target queue is the queue with the largest remaining data field value in the queue group; the remaining data field value of any queue is the difference between the upper limit of the data field value addition of the queue and the data field value of the database key added to the queue; the queue group includes at least one queue for adding database keys.

[0142] In some optional implementations, after determining the database key as the target database key, the database key selection module 802 is further configured to:

[0143] A new queue is created in the queue group, and the target database key is added to the new queue.

[0144] In some optional implementations, before determining the database key as the target database key, the database key selection module 802 is further configured to:

[0145] Determine that the remaining data field value in the target queue is greater than a third preset data field value; wherein the third preset data field value is less than the second preset data field value.

[0146] In some optional implementation manners, the field processing module 803 is specifically used for:

[0147] If there is an associated database key in any data unit other than any target data unit, the associated database key is expanded in the other data unit to obtain the target number of expanded database keys; wherein the associated database key is a database key having the same initial identifier as the target database key in the target data unit;

[0148] After merging the subfields of the target database key into the expanded database keys respectively, the subfields corresponding to the sub-identifiers are processed through different tasks;

[0149] The multiple processing results corresponding to the same initial identifier are processed collectively to obtain the target processing result.

[0150] Since the device is the device in the method in the embodiment of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0151] like Fig. 9 As shown, based on the same inventive concept, an embodiment of the present application provides an electronic device 900 including: a processor 901 and a memory 902 .

[0152] The memory 902 may be a volatile memory, such as a random-access memory (RAM); the memory 902 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 902 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 902 may be a combination of the above memories.

[0153] The processor 901 may include one or more central processing units (CPU), graphics processing units (GPU) or digital processing units, etc.

[0154] The specific connection medium between the memory 902 and the processor 901 is not limited in the embodiment of the present application. Fig. 9 In the embodiment, the memory 902 and the processor 901 are connected via a bus 903. The bus 903 is connected to the processor 901 via a bus 903. Fig. 9 In the figure, the bus 903 is shown by a bold line, and can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0155] The memory 902 stores program codes. When the program codes are executed by the processor 901, the processor 901 performs the following process:

[0156] Determine the target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit;

[0157] Selecting a target database key based on the data field values ​​of each database key in the target data unit;

[0158] The data field value of the target database key is divided into a target number of subfields, and a sub-identifier of each subfield is determined, and the subfield corresponding to each sub-identifier is processed through different tasks.

[0159] In some optional implementation manners, the processor is specifically configured to:

[0160] A data unit whose number of database keys is less than a preset number and whose data field value is greater than a first preset data field value is determined as the target data unit.

[0161] In some optional implementation manners, the processor is specifically configured to:

[0162] For any database key in the target data unit, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than a second preset data field value, the database key is determined as the target database key; wherein the target queue is the queue with the largest remaining data field value in the queue group; the remaining data field value of any queue is the difference between the upper limit of the data field value addition of the queue and the data field value of the database key added to the queue; the queue group includes at least one queue for adding database keys.

[0163] In some optional implementations, after determining the database key as the target database key, the processor is further configured to:

[0164] A new queue is created in the queue group, and the target database key is added to the new queue.

[0165] In some optional implementations, before determining the database key as the target database key, the processor is further configured to:

[0166] Determine that the remaining data field value in the target queue is greater than a third preset data field value; wherein the third preset data field value is less than the second preset data field value.

[0167] In some optional implementation manners, the processor is specifically configured to:

[0168] If there is an associated database key in any data unit other than any target data unit, the associated database key is expanded in the other data unit to obtain the target number of expanded database keys; wherein the associated database key is a database key having the same initial identifier as the target database key in the target data unit;

[0169] After merging the subfields of the target database key into the expanded database keys respectively, the subfields corresponding to the sub-identifiers are processed by different tasks;

[0170] The multiple processing results corresponding to the same initial identifier are processed collectively to obtain the target processing result.

[0171] Since the electronic device is the electronic device in the method in the embodiment of the present application, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0172] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above-mentioned data processing method are implemented. The readable storage medium may be a non-volatile readable storage medium.

[0173] The present application is described above with reference to the block diagrams and / or flow charts showing the methods, devices (systems) and / or computer program products according to the embodiments of the present application. It should be understood that a block of the block diagram and / or flow chart diagram and a combination of blocks of the block diagram and / or flow chart diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer and / or other programmable data processing device to produce a machine, so that the instructions executed by the computer processor and / or other programmable data processing device create a method for implementing the functions / actions specified in the block diagram and / or flow chart block.

[0174] Accordingly, the present application may also be implemented with hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, the present application may take the form of a computer program product on a computer-usable or computer-readable storage medium, which has a computer-usable or computer-readable program code implemented in the medium, for use by an instruction execution system or in conjunction with an instruction execution system. In the context of the present application, a computer-usable or computer-readable medium may be any medium that may contain, store, communicate, transmit, or convey a program for use by an instruction execution system, device, or apparatus, or in conjunction with an instruction execution system, device, or apparatus.

[0175] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An electronic device, characterized in that: include: Processor and memory; The memory stores a program code, and when the program code is executed by the processor, the processor performs the following process: Determine the target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit; Selecting a target database key based on the data field values ​​of each database key in the target data unit; Divide the data field value of the target database key into a target number of subfields, determine the sub-identifier of each subfield, and process the subfield corresponding to each sub-identifier through different tasks; The processor is specifically used for: Determine a data unit whose number of database keys is less than a preset number and whose data field value is greater than a first preset data field value as the target data unit; The processor is specifically used for: For any database key in the target data unit, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than a second preset data field value, the database key is determined as the target database key; wherein the target queue is the queue with the largest remaining data field value in the queue group; the remaining data field value of any queue is the difference between the upper limit of the data field value addition of the queue and the data field value of the database key added to the queue; the queue group includes at least one queue for adding database keys.

2. The electronic device according to claim 1, characterized in that: After determining the database key as the target database key, the processor is further configured to: A new queue is created in the queue group, and the target database key is added to the new queue.

3. The electronic device according to claim 1, characterized in that: Before determining the database key as the target database key, the processor is further configured to: Determine that the remaining data field value in the target queue is greater than a third preset data field value; wherein the third preset data field value is less than the second preset data field value.

4. The electronic device according to claim 1, characterized in that: The processor is specifically used for: If there is an associated database key in any data unit other than any target data unit, the associated database key is replicated in the other data unit to obtain the target number of replicated database keys; wherein the associated database key is a database key having the same initial identifier as the target database key in the target data unit; After merging the subfields of the target database key into the replicated database keys respectively, the subfields corresponding to the sub-identifiers are processed through different tasks; The multiple processing results corresponding to the same initial identifier are processed collectively to obtain the target processing result.

5. A data processing method, characterized in that: The method comprises: Determine the target data unit based on the number of database keys and data field values ​​in each data unit; wherein the data field value of the data unit is the sum of the data field values ​​of the database keys in the data unit; Selecting a target database key based on the data field values ​​of each database key in the target data unit; Divide the data field value of the target database key into a target number of subfields, determine the sub-identifier of each subfield, and process the subfield corresponding to each sub-identifier through different tasks; Determining a target data unit based on the number of database keys and data field values ​​in each data unit includes: Determine a data unit whose number of database keys is less than a preset number and whose data field value is greater than a first preset data field value as the target data unit; Selecting a target database key based on the data field value of each database key in the target data unit includes: For any database key in the target data unit, if the sum of the data field value of the database key and the data field value of the added database key in the target queue is greater than a second preset data field value, the database key is determined as the target database key; wherein the target queue is the queue with the largest remaining data field value in the queue group; the remaining data field value of any queue is the difference between the upper limit of the data field value addition of the queue and the data field value of the database key added to the queue; the queue group includes at least one queue for adding database keys.

6. The method according to claim 5, characterized in that After determining the database key as the target database key, the method further includes: A new queue is created in the queue group, and the target database key is added to the new queue.

Images