Data processing method, device, electronic device, storage medium and program product
By dynamically adjusting the reduction number according to the number of target fields and target key-value data, the problems of low utilization rate and high resource overhead caused by the different sizes of the to-process data are solved, and more efficient load balancing and resource utilization are achieved.
Patent Information
- Application Number
- CN202111062546.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-10
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-09-10
AI Technical Summary
In the prior art, when the size of the data to be processed is different, the same number of reduces is allocated for the data to be processed of various sizes, resulting in a reduction in the utilization rate of the reduce node and an increase in the system's resource overhead, which is not conducive to load balancing.
By determining the target field and target key value, the reduction number is flexibly adjusted according to the number of target key value data, ensuring the effective utilization of reduce nodes, reducing system resource overhead, and promoting load balancing. The specific method includes dynamically adjusting the reduction number based on the comparison of the target key-value data with the number of reduced nodes to ensure the efficiency of reduction processing and resource utilization.
It ensures the utilization rate of reduce nodes, reduces system resource overhead, and facilitates load balancing, and improves data processing efficiency.
Smart Images

Figure CN113722071B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a data processing method, device, electronic device, storage medium, and program product. Background Art
[0002] Reduce refers to a computing method that copies data from the relevant map end to the reduce node for reduction processing at runtime. In the prior art, the number of reducers is usually immutable except that it can be set to 0, which means that some data does not need to be reduced. However, in reality, the size of the data to be processed varies. If the same number of reducers is assigned to data of various sizes, it may lead to a decrease in the utilization rate of the reduce nodes, increase the resource overhead of the system, and is not conducive to load balancing. Summary of the invention
[0003] Embodiments of the present disclosure provide a data processing method, device, electronic device, storage medium, and program product.
[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method.
[0005] Specifically, the data processing method includes:
[0006] Determine the target field and target key value;
[0007] Determine the number of reductions according to the target field and the target key value;
[0008] The target field, the target key value and the reduction quantity are used as grouping parameters, and a grouping calculation task is performed on the grouping calculation input data.
[0009] In combination with the first aspect, in a first implementation of the first aspect of the embodiment of the present disclosure, determining the number of reductions according to the target field and the target key value includes:
[0010] Retrieving the group calculation input data using the target field and the target key value as index values to obtain the quantity of the target key value data;
[0011] The number of reductions is determined according to the number of target key-value data.
[0012] In combination with the first aspect and the first implementation of the first aspect, in a second implementation of the first aspect of the embodiment of the present disclosure, determining the number of reductions according to the number of target key-value data includes:
[0013] Get the number of reduction nodes;
[0014] The number of the target key-value data is compared with the number of reduction nodes to determine the reduction number.
[0015] In combination with the first aspect, the first implementation manner of the first aspect, and the second implementation manner of the first aspect, in a third implementation manner of the first aspect of the present disclosure, comparing the number of the target key-value data with the number of reduction nodes to determine the number of reductions includes:
[0016] When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data;
[0017] When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes;
[0018] When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder;
[0019] When the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes evenly, and the remainder is less than or equal to half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
[0020] In combination with the first aspect, the first implementation of the first aspect, the second implementation of the first aspect, and the third implementation of the first aspect, the fourth implementation of the first aspect of the present disclosure further includes:
[0021] Performing preset processing on the data obtained after executing the group calculation task to obtain preset processed data;
[0022] The preset processing data and the reduction quantity are sent to the reduction component.
[0023] In a second aspect, a data processing method is provided in an embodiment of the present disclosure.
[0024] Specifically, the data processing method includes:
[0025] receiving preset processing data and reduction quantity sent by the group computing component;
[0026] Determining a reduction node according to the reduction quantity;
[0027] The reduction processing is performed on the preset processing data using the determined reduction node.
[0028] In a third aspect, a data processing method is provided in an embodiment of the present disclosure.
[0029] Specifically, the data processing method includes:
[0030] The group calculation component determines a target field and a target key value, determines a reduction quantity according to the target field and the target key value, takes the target field, the target key value and the reduction quantity as group parameters, performs a group calculation task on the group calculation input data, performs preset processing on the data obtained after performing the group calculation task, obtains preset processing data, and sends the preset processing data and the reduction quantity to the reduction component;
[0031] The reduction component receives the preset processing data and the reduction quantity sent by the group calculation component, determines the reduction node according to the reduction quantity, and performs reduction processing on the preset processing data using the determined reduction node.
[0032] In a fourth aspect, a data processing device is provided in an embodiment of the present disclosure.
[0033] Specifically, the data processing device includes:
[0034] A first determination module is configured to determine a target field and a target key value;
[0035] A second determination module is configured to determine the reduction quantity according to the target field and the target key value;
[0036] The execution module is configured to perform a group calculation task on the group calculation input data with the target field, the target key value and the reduction quantity as group parameters.
[0037] In combination with the fourth aspect, in a first implementation of the fourth aspect of the embodiment of the present disclosure, the second determining module is configured as follows:
[0038] Retrieving the group calculation input data using the target field and the target key value as index values to obtain the quantity of the target key value data;
[0039] The number of reductions is determined according to the number of target key-value data.
[0040] In combination with the fourth aspect and the first implementation manner of the fourth aspect, in the second implementation manner of the fourth aspect of the embodiment of the present disclosure, the part of determining the number of reductions according to the number of the target key-value data is configured as follows:
[0041] Get the number of reduction nodes;
[0042] The number of the target key-value data is compared with the number of reduction nodes to determine the reduction number.
[0043] In combination with the fourth aspect, the first implementation manner of the fourth aspect, and the second implementation manner of the fourth aspect, in a third implementation manner of the fourth aspect of the present disclosure, the part of comparing the number of the target key-value data with the number of reduction nodes to determine the number of reductions is configured as follows:
[0044] When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data;
[0045] When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes;
[0046] When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder;
[0047] When the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes evenly, and the remainder is less than or equal to half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
[0048] In combination with the fourth aspect, the first implementation of the fourth aspect, the second implementation of the fourth aspect, and the third implementation of the fourth aspect, the present disclosure, in a fourth implementation of the fourth aspect, further includes:
[0049] The sending module is configured to perform preset processing on the data obtained after executing the group computing task, obtain preset processing data, and send the preset processing data and the reduction quantity to the reduction component.
[0050] In a fifth aspect, a data processing device is provided in an embodiment of the present disclosure.
[0051] Specifically, the data processing device includes:
[0052] A receiving module, configured to receive preset processing data and reduction quantity sent by the group computing component;
[0053] A third determination module is configured to determine a reduction node according to the reduction quantity;
[0054] The processing module is configured to use the determined reduction node to perform reduction processing on the preset processing data.
[0055] In a sixth aspect, a data processing device is provided in an embodiment of the present disclosure.
[0056] Specifically, the data processing device includes:
[0057] a group calculation component configured to determine a target field and a target key value, determine a reduction quantity according to the target field and the target key value, perform a group calculation task on the group calculation input data with the target field, the target key value and the reduction quantity as group parameters, perform preset processing on the data obtained after performing the group calculation task, obtain preset processing data, and send the preset processing data and the reduction quantity to the reduction component;
[0058] The reduction component is configured to receive the preset processing data and the reduction quantity sent by the group computing component, determine the reduction node according to the reduction quantity, and perform reduction processing on the preset processing data using the determined reduction node.
[0059] In a seventh aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions that support a data processing device to execute the above-mentioned data processing method, and the processor is configured to execute the computer instructions stored in the memory. The data processing device may also include a communication interface for the data processing device to communicate with other devices or a communication network.
[0060] In an eighth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by a data processing device, which includes computer instructions involved in the data processing device for executing the above-mentioned data processing method.
[0061] In a ninth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data processing method when executed by a processor.
[0062] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0063] The above technical solution adjusts the number of reductions by setting the number of reduction parameters for group calculation. The technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0064] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Other features, purposes and advantages of the embodiments of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:
[0066] Figure 1A flowchart showing a data processing method according to an embodiment of the present disclosure;
[0067] Figure 2 A flowchart showing a data processing method according to another embodiment of the present disclosure;
[0068] Figure 3 A flowchart showing a data processing method according to another embodiment of the present disclosure;
[0069] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown;
[0070] Figure 5 A structural block diagram showing a data processing device according to another embodiment of the present disclosure;
[0071] Figure 6 A structural block diagram showing a data processing device according to another embodiment of the present disclosure;
[0072] Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;
[0073] Figure 8 It is a structural diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0074] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.
[0075] In the embodiments of the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in this specification, and are not intended to exclude the possibility of one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof existing or being added.
[0076] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0077] The technical solution provided by the embodiment of the present disclosure adjusts the number of reductions by setting a reduction number parameter for group computing. The technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0078] Figure 1 A flow chart showing a data processing method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the data processing method includes the following steps S101-S103:
[0079] In step S101, a target field and a target key value are determined;
[0080] In step S102, the reduction quantity is determined according to the target field and the target key value;
[0081] In step S103, a group calculation task is performed on the group calculation input data with the target field, target key value and reduction quantity as group parameters.
[0082] As mentioned above, Reduce refers to a computing method that copies data from the relevant map end to the reduce node for reduction processing at runtime. In the prior art, the number of reducers is usually immutable except that it can be set to 0, which means that some data does not need to be reduced. In fact, the size of the data to be processed varies. If the same number of reducers is assigned to data of various sizes, it may lead to a decrease in the utilization rate of the reduce nodes, increase the resource overhead of the system, and is not conducive to load balancing.
[0083] In view of the above problems, in this embodiment, a data processing method is proposed, which adjusts the number of reductions by setting a reduction number parameter for group calculation. The technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0084] In one embodiment of the present disclosure, the data processing method may be applicable to a group computing component that performs data processing.
[0085] In one embodiment of the present disclosure, the target field refers to the field information based on which the group calculation is performed, and the target key value refers to the key value information based on which the group calculation is performed. For example, if the target field is A and the target key value is B, the group calculation performed with the target field and the target key value as group calculation parameters refers to the group calculation performed on all data with the key value B in field A.
[0086] In one embodiment of the present disclosure, the reduction number refers to the number of reduction nodes required for subsequent reduction processing, wherein the reduction number is determined based on the target field and the target key value, and the reduction number is less than or equal to the total number of reduction nodes.
[0087] In one embodiment of the present disclosure, the group calculation input data refers to data waiting for group calculation, and the group calculation input data may be, for example, mapping output data and the like.
[0088] In the above implementation, the target field and the target key value are first determined, and then the number of reductions required for subsequent reduction processing is determined based on the target field and the target key value. Then, the target field, the target key value and the number of reductions are used as common grouping parameters to perform the group calculation task on the group calculation input data.
[0089] In one embodiment of the present disclosure, the step S102, i.e., the step of determining the number of reductions according to the target field and the target key value, may include the following steps:
[0090] Retrieving the group calculation input data using the target field and the target key value as index values to obtain the number of target key value data;
[0091] The number of reductions is determined according to the number of target key-value data.
[0092] In this implementation, when determining the number of reductions, the target field and target key value can be used as index values to retrieve the group calculation input data to obtain the number of data corresponding to the target key value. The data corresponding to the target key value is the data that needs to be reduced subsequently. Therefore, the number of reductions can be determined based on the number of target key value data.
[0093] In one embodiment of the present disclosure, the step S102, i.e., the step of determining the number of reductions according to the number of the target key-value data, may include the following steps:
[0094] Get the number of reduction nodes;
[0095] The number of the target key-value data is compared with the number of reduction nodes to determine the reduction number.
[0096] In this implementation, when determining the number of reductions based on the number of target key-value data, the total number of reduction nodes is first obtained, and then the number of target key-value data is compared with the number of reduction nodes to determine the most appropriate number of reductions that can fully utilize the reduction nodes and reduce the system's resource overhead to facilitate load balancing.
[0097] Specifically, if the number of target key-value data is less than or equal to the number of reduction nodes, in order to make full use of the reduction nodes without wasting the reduction node resources, the reduction number can be set to the smaller number of target key-value data. For example, if the number of target key-value data is 15 and the number of reduction nodes is 18, the reduction number can be set to 15, so that all data can be processed after one round of reduction processing.
[0098] If the number of target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number can be set to the number of reduction nodes. For example, if the number of target key-value data is 15, the number of reduction nodes is 5, and 15 can divide 5, the reduction number can be set to 5, so that all data can be processed after three rounds of reduction processing.
[0099] If the number of target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder obtained is greater than half of the number of reduction nodes, the number of reductions can be set to the number of reduction nodes or the remainder. For example, if the number of target key-value data is 19, the number of reduction nodes is 5, and the remainder of 19 divided by 5 is 4, which is greater than half of 5, the number of reductions can be set to the number of reduction nodes 5, so that all data can be processed after four rounds of reduction processing. The number of reductions can also be set to the remainder 4, so that all data can be processed after five rounds of reduction processing, and the last round of reduction processing will not waste too many reduction node resources.
[0100] If the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes, and the remainder is less than or equal to half of the number of reduction nodes, the reduction number can be set to the number of reduction nodes or the remainder. For example, if the number of the target key-value data is 17, the number of reduction nodes is 5, and the remainder of 17 divided by 5 is 2, which is less than half of 5. At this time, if the reduction number is set to the number of reduction nodes 5, the last round of reduction processing will only use less than half of the reduction node resources, so it will cause a waste of reduction node resources. However, if the reduction number is set to the remainder 2, although the last round of reduction processing will not cause too much waste of reduction node resources, due to the large number of rounds required for reduction processing, it will lengthen the reduction processing process and increase the time consumption of reduction processing. Therefore, when the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes, and the remainder is less than or equal to half of the number of reduction nodes, it can be selected according to the needs of the actual application whether to set the reduction number to the number of reduction nodes or the remainder.
[0101] That is, in one embodiment of the present disclosure, the step of comparing the number of the target key-value data with the number of reduction nodes to determine the number of reductions may include the following steps:
[0102] When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data;
[0103] When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes;
[0104] When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder;
[0105] When the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes evenly, and the remainder is less than or equal to half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
[0106] In one embodiment of the present disclosure, the method may further include the following steps:
[0107] Performing preset processing on the data obtained after executing the group calculation task to obtain preset processed data;
[0108] The preset processing data and the reduction quantity are sent to the reduction component.
[0109] After the group calculation task is completed for the group calculation input data, before the data obtained after the group calculation task is sent to the reduction component, it is possible to merge, partition, clean, etc. the data obtained after the group calculation task is executed. In this implementation, in order to enable the reduction component to clearly know the number of nodes that need to participate in the reduction processing, the preset processing data and the reduction quantity obtained after the preset processing are also sent to the reduction component.
[0110] Figure 2 A flow chart showing a data processing method according to another embodiment of the present disclosure is shown as follows: Figure 2 As shown, the data processing method includes the following steps S201-S203:
[0111] In step S201, receiving preset processing data and reduction quantity sent by the group computing component;
[0112] In step S202, a reduction node is determined according to the reduction quantity;
[0113] In step S203, reduction processing is performed on the preset processing data using the determined reduction node.
[0114] As mentioned above, Reduce refers to a computing method that copies data from the relevant map end to the reduce node for reduction processing at runtime. In the prior art, the number of reducers is usually immutable except that it can be set to 0, which means that some data does not need to be reduced. In fact, the size of the data to be processed varies. If the same number of reducers is assigned to data of various sizes, it may lead to a decrease in the utilization rate of the reduce nodes, increase the resource overhead of the system, and is not conducive to load balancing.
[0115] In view of the above problems, in this embodiment, a data processing method is proposed, which adjusts the number of reductions by setting a reduction number parameter for group calculation. The technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0116] In one embodiment of the present disclosure, the data processing method may be applicable to a reduction component that performs reduction processing.
[0117] In one embodiment of the present disclosure, when the reduction nodes are determined according to the reduction number, when the reduction number is less than the reduction nodes, or the number of reduction nodes required in a round is less than the number of available reduction nodes, the required reduction nodes can be randomly selected from the available reduction nodes, or appropriate reduction nodes can be selected from the available reduction nodes according to the needs of actual applications.
[0118] In the above implementation, the preset processing data and the reduction number sent by the group computing component are first received, and then the reduction node is determined according to the reduction number. More specifically, when the reduction number is less than the number of reduction nodes, the reduction nodes participating in the reduction processing can be determined according to the reduction number. When the reduction number is greater than the number of reduction nodes, the reduction processing round can be first determined according to the reduction number, and then the reduction nodes participating in the reduction processing in the last reduction processing round are determined; finally, the reduction processing is performed on the preset processing data using the determined reduction node.
[0119] Figure 2 The technical terms and technical features involved in the embodiments shown and related Figure 1 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 2 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 1The explanations of the illustrated and related embodiments will not be repeated here.
[0120] Figure 3 A flow chart showing a data processing method according to another embodiment of the present disclosure is shown as follows: Figure 3 As shown, the data processing method includes the following steps S301-S302:
[0121] In step S301, the group calculation component determines a target field and a target key value, determines a reduction quantity according to the target field and the target key value, takes the target field, the target key value and the reduction quantity as group parameters, performs a group calculation task on the group calculation input data, performs preset processing on the data obtained after performing the group calculation task, obtains preset processing data, and sends the preset processing data and the reduction quantity to the reduction component;
[0122] In step S302, the reduction component receives the preset processing data and the reduction quantity sent by the group computing component, determines the reduction node according to the reduction quantity, and performs reduction processing on the preset processing data using the determined reduction node.
[0123] As mentioned above, Reduce refers to a computing method that copies data from the relevant map end to the reduce node for reduction processing at runtime. In the prior art, the number of reducers is usually immutable except that it can be set to 0, which means that some data does not need to be reduced. In fact, the size of the data to be processed varies. If the same number of reducers is assigned to data of various sizes, it may lead to a decrease in the utilization rate of the reduce nodes, increase the resource overhead of the system, and is not conducive to load balancing.
[0124] In view of the above problems, in this embodiment, a data processing method is proposed, which adjusts the number of reductions by setting a reduction number parameter for group calculation. The technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0125] In one embodiment of the present disclosure, the data processing method may be applicable to a data processing system that performs data processing and includes a grouping calculation component and a reduction component.
[0126] Figure 3 The technical terms and technical features involved in the embodiments shown and related Figure 1-2 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 3 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 1-2 The explanations of the illustrated and related embodiments will not be repeated here.
[0127] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.
[0128] Figure 4 FIG. 1 is a block diagram showing a data processing device according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 4 As shown, the data processing device includes:
[0129] A first determination module 401 is configured to determine a target field and a target key value;
[0130] A second determination module 402 is configured to determine the reduction quantity according to the target field and the target key value;
[0131] The execution module 403 is configured to perform a group calculation task on the group calculation input data with the target field, target key value and reduction quantity as group parameters.
[0132] As mentioned above, Reduce refers to a computing method that copies data from the relevant map end to the reduce node for reduction processing at runtime. In the prior art, the number of reducers is usually immutable except that it can be set to 0, which means that some data does not need to be reduced. In fact, the size of the data to be processed varies. If the same number of reducers is assigned to data of various sizes, it may lead to a decrease in the utilization rate of the reduce nodes, increase the resource overhead of the system, and is not conducive to load balancing.
[0133] In view of the above problems, in this embodiment, a data processing device is proposed, which adjusts the number of reductions by setting a reduction number parameter for group calculation. This technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0134] In one embodiment of the present disclosure, the data processing device may be implemented as a group computing component for performing data processing.
[0135] In one embodiment of the present disclosure, the target field refers to the field information based on which the group calculation is performed, and the target key value refers to the key value information based on which the group calculation is performed. For example, if the target field is A and the target key value is B, the group calculation performed with the target field and the target key value as group calculation parameters refers to the group calculation performed on all data with the key value B in field A.
[0136] In one embodiment of the present disclosure, the reduction number refers to the number of reduction nodes required for subsequent reduction processing, wherein the reduction number is determined based on the target field and the target key value, and the reduction number is less than or equal to the total number of reduction nodes.
[0137] In one embodiment of the present disclosure, the group calculation input data refers to data waiting for group calculation, and the group calculation input data may be, for example, mapping output data and the like.
[0138] In the above implementation, the target field and the target key value are first determined, and then the number of reductions required for subsequent reduction processing is determined based on the target field and the target key value. Then, the target field, the target key value and the number of reductions are used as common grouping parameters to perform the group calculation task on the group calculation input data.
[0139] In an embodiment of the present disclosure, the second determining module 402 may be configured as follows:
[0140] Retrieving the group calculation input data using the target field and the target key value as index values to obtain the number of target key value data;
[0141] The number of reductions is determined according to the number of target key-value data.
[0142] In this implementation, when determining the number of reductions, the target field and target key value can be used as index values to retrieve the group calculation input data to obtain the number of data corresponding to the target key value. The data corresponding to the target key value is the data that needs to be reduced subsequently. Therefore, the number of reductions can be determined based on the number of target key value data.
[0143] In one embodiment of the present disclosure, the part of determining the number of reductions according to the number of target key-value data may be configured as follows:
[0144] Get the number of reduction nodes;
[0145] The number of the target key-value data is compared with the number of reduction nodes to determine the reduction number.
[0146] In this implementation, when determining the number of reductions based on the number of target key-value data, the total number of reduction nodes is first obtained, and then the number of target key-value data is compared with the number of reduction nodes to determine the most appropriate number of reductions that can fully utilize the reduction nodes and reduce the system's resource overhead to facilitate load balancing.
[0147] Specifically, if the number of target key-value data is less than or equal to the number of reduction nodes, in order to make full use of the reduction nodes without wasting the reduction node resources, the reduction number can be set to the smaller number of target key-value data. For example, if the number of target key-value data is 15 and the number of reduction nodes is 18, the reduction number can be set to 15, so that all data can be processed after one round of reduction processing.
[0148] If the number of target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number can be set to the number of reduction nodes. For example, if the number of target key-value data is 15, the number of reduction nodes is 5, and 15 can divide 5, the reduction number can be set to 5, so that all data can be processed after three rounds of reduction processing.
[0149] If the number of target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder obtained is greater than half of the number of reduction nodes, the number of reductions can be set to the number of reduction nodes or the remainder. For example, if the number of target key-value data is 19, the number of reduction nodes is 5, and the remainder of 19 divided by 5 is 4, which is greater than half of 5, the number of reductions can be set to the number of reduction nodes 5, so that all data can be processed after four rounds of reduction processing. The number of reductions can also be set to the remainder 4, so that all data can be processed after five rounds of reduction processing, and the last round of reduction processing will not waste too many reduction node resources.
[0150] If the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes, and the remainder is less than or equal to half of the number of reduction nodes, the reduction number can be set to the number of reduction nodes or the remainder. For example, if the number of the target key-value data is 17, the number of reduction nodes is 5, and the remainder of 17 divided by 5 is 2, which is less than half of 5. At this time, if the reduction number is set to the number of reduction nodes 5, the last round of reduction processing will only use less than half of the reduction node resources, so it will cause a waste of reduction node resources. However, if the reduction number is set to the remainder 2, although the last round of reduction processing will not cause too much waste of reduction node resources, due to the large number of rounds required for reduction processing, it will lengthen the reduction processing process and increase the time consumption of reduction processing. Therefore, when the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes, and the remainder is less than or equal to half of the number of reduction nodes, it can be selected according to the needs of the actual application whether to set the reduction number to the number of reduction nodes or the remainder.
[0151] That is, in one embodiment of the present disclosure, the part of comparing the number of the target key-value data with the number of reduction nodes to determine the number of reductions may be configured as follows:
[0152] When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data;
[0153] When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes;
[0154] When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder;
[0155] When the number of the target key-value data is greater than the number of reduction nodes, cannot divide the number of reduction nodes evenly, and the remainder is less than or equal to half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
[0156] In one embodiment of the present disclosure, the device may further include:
[0157] The sending module is configured to perform preset processing on the data obtained after executing the group computing task, obtain preset processing data, and send the preset processing data and the reduction quantity to the reduction component.
[0158] After the group calculation task is completed for the group calculation input data, before the data obtained after the group calculation task is sent to the reduction component, it is possible to merge, partition, clean, etc. the data obtained after the group calculation task is executed. In this implementation, in order to enable the reduction component to clearly know the number of nodes that need to participate in the reduction processing, the preset processing data and the reduction quantity obtained after the preset processing are also sent to the reduction component.
[0159] Figure 5 FIG. 2 shows a structural block diagram of a data processing device according to another embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 5 As shown, the data processing device includes:
[0160] The receiving module 501 is configured to receive the preset processing data and reduction quantity sent by the group computing component;
[0161] A third determination module 502 is configured to determine a reduction node according to the reduction quantity;
[0162] The processing module 503 is configured to use the determined reduction node to perform reduction processing on the preset processing data.
[0163] As mentioned above, Reduce refers to a computing method that copies data from the relevant map end to the reduce node for reduction processing at runtime. In the prior art, the number of reducers is usually immutable except that it can be set to 0, which means that some data does not need to be reduced. In fact, the size of the data to be processed varies. If the same number of reducers is assigned to data of various sizes, it may lead to a decrease in the utilization rate of the reduce nodes, increase the resource overhead of the system, and is not conducive to load balancing.
[0164] In view of the above problems, in this embodiment, a data processing device is proposed, which adjusts the number of reductions by setting a reduction number parameter for group calculation. This technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0165] In one embodiment of the present disclosure, the data processing device may be implemented as a reduction component that performs reduction processing.
[0166] In one embodiment of the present disclosure, when the reduction nodes are determined according to the reduction number, when the reduction number is less than the reduction nodes, or the number of reduction nodes required in a round is less than the number of available reduction nodes, the required reduction nodes can be randomly selected from the available reduction nodes, or appropriate reduction nodes can be selected from the available reduction nodes according to the needs of actual applications.
[0167] In the above implementation, the preset processing data and the reduction number sent by the group computing component are first received, and then the reduction node is determined according to the reduction number. More specifically, when the reduction number is less than the number of reduction nodes, the reduction nodes participating in the reduction processing can be determined according to the reduction number. When the reduction number is greater than the number of reduction nodes, the reduction processing round can be first determined according to the reduction number, and then the reduction nodes participating in the reduction processing in the last reduction processing round are determined; finally, the reduction processing is performed on the preset processing data using the determined reduction node.
[0168] Figure 5 The technical terms and technical features involved in the embodiments shown and related Figure 4 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 5 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 4 The explanations of the illustrated and related embodiments will not be repeated here.
[0169] Figure 6 FIG. 2 shows a structural block diagram of a data processing device according to another embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 6 As shown, the data processing device includes:
[0170] The group calculation component 601 is configured to determine a target field and a target key value, determine a reduction quantity according to the target field and the target key value, perform a group calculation task on the group calculation input data with the target field, the target key value and the reduction quantity as group parameters, perform a preset processing on the data obtained after performing the group calculation task, obtain preset processing data, and send the preset processing data and the reduction quantity to the reduction component;
[0171] The reduction component 602 is configured to receive the preset processing data and the reduction quantity sent by the group computing component, determine the reduction node according to the reduction quantity, and perform reduction processing on the preset processing data using the determined reduction node.
[0172] As mentioned above, Reduce refers to a computing method that copies data from the relevant map end to the reduce node for reduction processing at runtime. In the prior art, the number of reducers is usually immutable except that it can be set to 0, which means that some data does not need to be reduced. In fact, the size of the data to be processed varies. If the same number of reducers is assigned to data of various sizes, it may lead to a decrease in the utilization rate of the reduce nodes, increase the resource overhead of the system, and is not conducive to load balancing.
[0173] In view of the above problems, in this embodiment, a data processing device is proposed, which adjusts the number of reductions by setting a reduction number parameter for group calculation. This technical solution can flexibly adjust the number of reductions, thereby ensuring the utilization rate of reduce nodes, reducing the resource overhead of the system, and also facilitating load balancing.
[0174] In one embodiment of the present disclosure, the data processing device may be implemented as a data processing system that performs data processing and includes a grouping calculation component and a reduction component.
[0175] Figure 6 The technical terms and technical features involved in the embodiments shown and related Figure 4-5 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 6The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 4-5 The explanations of the illustrated and related embodiments will not be repeated here.
[0176] The present disclosure also discloses an electronic device, Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 7 As shown, the electronic device 700 includes a memory 701 and a processor 702; wherein,
[0177] The memory 701 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 702 to implement the above method steps.
[0178] Figure 8 It is a structural diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure.
[0179] like Figure 8 As shown, the computer system 800 includes a processing unit 801, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The processing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0180] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. The drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage section 808 as needed. Among them, the processing unit 801 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.
[0181] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium, and the computer program includes a program code for executing the route planning method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 809, and / or installed from a removable medium 811.
[0182] The embodiment of the present disclosure also discloses a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, any of the above method steps is implemented.
[0183] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0184] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.
[0185] As another aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the embodiments of the present disclosure.
[0186] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.
Claims
1. A data processing method, comprising: Determine a target field and a target key value, wherein the target field refers to the field information based on which the group calculation is performed, and the target key value refers to the key value information based on which the group calculation is performed; Determine the number of reductions according to the target field and the target key value; Taking the target field, target key value and reduction quantity as grouping parameters, performing a grouping calculation task on the grouping calculation input data; Wherein, determining the reduction quantity according to the target field and the target key value includes: Retrieving the group calculation input data using the target field and the target key value as index values to obtain the quantity of the target key value data; Determining the number of reductions according to the number of target key-value data; Wherein, determining the reduction quantity according to the quantity of the target key-value data includes: Get the number of reduction nodes; Compare the number of the target key-value data with the number of reduction nodes to determine the reduction number; The step of comparing the number of the target key-value data with the number of reduction nodes to determine the reduction number includes: When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data; When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes; When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
2. The method according to claim 1, further comprising: Performing preset processing on the data obtained after executing the group calculation task to obtain preset processed data; The preset processing data and the reduction quantity are sent to the reduction component.
3. A data processing method, comprising: Receive preset processing data and reduction quantity sent by the group computing component, wherein the reduction quantity is determined according to the quantity of target key value data, the target key value data is retrieved from the group computing input data using a target field and a target key value as index values, the target field refers to field information based on which the group computing is performed, and the target key value refers to key value information based on which the group computing is performed; Determining a reduction node according to the reduction quantity; Using the determined reduction node to perform reduction processing on the preset processing data; The reduction quantity is determined according to the quantity of target key-value data, specifically: When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data; When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes; When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
4. A data processing method, comprising: The group calculation component determines a target field and a target key value, retrieves the group calculation input data using the target field and the target key value as index values, obtains the number of target key value data, determines the reduction number according to the number of target key value data, performs a group calculation task on the group calculation input data using the target field, the target key value and the reduction number as group parameters, performs preset processing on the data obtained after executing the group calculation task, obtains preset processed data, and sends the preset processed data and the reduction number to the reduction component; The reduction component receives the preset processing data and the reduction quantity sent by the grouping computing component, determines the reduction node according to the reduction quantity, and performs reduction processing on the preset processing data using the determined reduction node; The step of determining the number of reductions according to the number of target key-value data is as follows: When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data; When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes; When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
5. A data processing device, comprising: A first determination module is configured to determine a target field and a target key value, wherein the target field refers to field information based on which the group calculation is performed, and the target key value refers to key value information based on which the group calculation is performed; A second determination module is configured to determine the reduction quantity according to the target field and the target key value; An execution module is configured to perform a group calculation task on the group calculation input data with the target field, the target key value and the reduction quantity as group parameters; Wherein, the second determining module is configured as follows: Retrieving the group calculation input data using the target field and the target key value as index values to obtain the quantity of the target key value data; Determining the number of reductions according to the number of target key-value data; The part of determining the number of reductions according to the number of target key-value data is configured as follows: Get the number of reduction nodes; Compare the number of the target key-value data with the number of reduction nodes to determine the reduction number; The part of comparing the number of the target key-value data with the number of reduction nodes to determine the number of reduction nodes is configured as follows: When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data; When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes; When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
6. The apparatus according to claim 5, further comprising: The sending module is configured to perform preset processing on the data obtained after executing the group computing task, obtain preset processing data, and send the preset processing data and the reduction quantity to the reduction component.
7. A data processing device, comprising: A receiving module is configured to receive preset processing data and reduction quantity sent by the group computing component, wherein the reduction quantity is determined according to the quantity of target key value data, the target key value data is retrieved from the group computing input data using a target field and a target key value as index values, the target field refers to field information based on the group computing, and the target key value refers to key value information based on the group computing; A third determination module is configured to determine a reduction node according to the reduction quantity; A processing module, configured to perform reduction processing on the preset processing data using the determined reduction node; The part where the reduction quantity is determined according to the quantity of target key-value data is configured as follows: When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data; When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes; When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
8. A data processing device, comprising: The group computing component is configured to determine a target field and a target key value, retrieve the group computing input data using the target field and the target key value as index values, obtain the number of target key value data, determine the number of reductions according to the number of target key value data, perform a group computing task on the group computing input data using the target field, the target key value and the number of reductions as grouping parameters, perform preset processing on the data obtained after performing the group computing task, obtain preset processed data, and send the preset processed data and the number of reductions to the reduction component; A reduction component is configured to receive the preset processing data and the reduction quantity sent by the group computing component, determine the reduction node according to the reduction quantity, and perform reduction processing on the preset processing data using the determined reduction node; The part of determining the number of reductions according to the number of target key-value data is configured as follows: When the number of the target key-value data is less than or equal to the number of reduction nodes, setting the reduction number to the number of the target key-value data; When the number of the target key-value data is greater than the number of reduction nodes and can divide the number of reduction nodes, the reduction number is set to the number of reduction nodes; When the number of the target key-value data is greater than the number of reduction nodes, the number of reduction nodes cannot be divided evenly, and the remainder is greater than half of the number of reduction nodes, the reduction number is set to the number of reduction nodes or the remainder.
9. An electronic device comprising a memory and a processor; wherein: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 4.
10. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instruction is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
