Data processing methods and apparatus

Abstract

This application discloses a data processing method and apparatus, relating to distributed data processing technology. The specific implementation scheme is as follows: a first distributed node determines the priority of each data partition corresponding to the data processing task; based on the processing requirements of the sub-data included in the source data and the priority of each data partition, it determines a target data partition that matches the processing requirements of the sub-data; and sends the sub-data, the identifier of the target data partition, and the priority information of the target data partition to a second distributed node; the second distributed node writes the sub-data and the priority information of the target data partition into the target data partition, and processes the sub-data in the target data partition according to the priority of the target data partition. This application can ensure that data with high processing requirements can be processed during peak network access periods or when the network is congested.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a distributed data processing technology. Background Technology

[0002] In a distributed computing system, the source data input to the system may include various types and properties of data. The processing requirements for different types or properties of data may differ.

[0003] In existing distributed systems, a shuffle-like mechanism is used to divide the source data into multiple partitions to be processed. The distributed system processes these partitions indiscriminately, which may result in some partitions with high processing requirements (such as high timeliness and / or accuracy requirements) not being processed during peak network access periods or network congestion. Summary of the Invention

[0004] This application provides a data processing method and apparatus that can ensure that data with high processing requirements can be processed during peak network access periods or when the network is congested.

[0005] In a first aspect, this application provides a data processing method, comprising: a first distributed node determining the priority of each data partition corresponding to a data processing task, the data processing task including: processing operations to be performed on source data; the first distributed node determining a target data partition matching the processing requirements of the sub-data in the source data and the priority of each data partition; and the first distributed node sending the sub-data, the identifier of the target data partition, and the priority information of the target data partition to a second distributed node.

[0006] In this scheme, the node receiving the source data determines the priority of the data partition, and determines the data partition whose priority matches the processing requirements of the sub-data of the source data and the priority of each data partition, so that the sub-data can be written into the data partition whose priority matches the processing requirements of the sub-data. In this way, when the node processing the data processes the data according to the priority of the data partition, it can process the data differently according to the processing requirements of the data. During peak network access or network congestion, it can ensure that data with high processing requirements can be processed.

[0007] In one possible design, the first distributed node determines the priority of each data partition corresponding to the current data processing task, including: the first distributed node determines the priority of each data partition according to priority setting information. Optionally, the priority setting information includes: first information, which indicates a preset ratio between the number of data partitions of various priorities; or, second information, which indicates a preset proportion of the total number of data partitions for each priority.

[0008] In this scheme, the first distributed node determines the priority of each data partition based on the priority setting information of the data partition, which can improve the efficiency of determining the priority of the data partition and make the ratio between data partitions of various priorities more reasonable.

[0009] In one possible design, the method further includes: the first distributed node determining the packet loss attributes of each data partition corresponding to the current data processing task; the first distributed node sending the packet loss attribute information of the target data partition to the second distributed node; correspondingly, the first distributed node determining the target data partition matching the processing requirements of the sub-data in the source data according to the processing requirements of the sub-data and the priority of each data partition, including: the first distributed node determining the target data partition matching the processing requirements of the sub-data according to the processing requirements of the sub-data included in the source data, the priority of each data partition and the packet loss attributes.

[0010] This solution can further ensure that data with high processing requirements can be processed by setting the packet loss attribute of data partitions.

[0011] In one possible design, the first distributed node determines the packet loss attributes of each data partition corresponding to the current data processing task, including: the first distributed node determines the packet loss attributes of each data partition based on packet loss attribute setting information, the priority of each data partition, and preset conditions; wherein, the preset conditions include: if there is a data partition with a packet loss attribute of "packet lossable" in the data partition of the first priority, then all data partitions of the second priority have a packet loss attribute of "packet lossable", and the first priority is higher than the second priority. Optionally, the packet loss attribute setting information includes: third information, which indicates a preset ratio between the number of data partitions with various packet loss attributes; or, fourth information, which indicates a preset proportion of the number of data partitions with various packet loss attributes to the total number of data partitions.

[0012] In this scheme, the first distributed node determines the packet loss attribute of each data partition based on the packet loss attribute setting information, the priority of each data partition, and preset conditions. This can improve the efficiency of determining the packet loss attribute of the data partition and make the ratio between data partitions with various packet loss attributes more reasonable.

[0013] Secondly, this application provides a data processing method, comprising: a second distributed node receiving sub-data, an identifier of a target data partition, and priority information of the target data partition sent by a first distributed node, wherein the target data partition is a data partition that matches the processing requirements of the sub-data; and the second distributed node writing the sub-data and the priority information of the target data partition into the target data partition.

[0014] In this scheme, the second distributed node receives the priority information of the data partition from the first distributed node, which enables the second distributed node to process the sub-data in the data partition according to the priority of the data partition. During peak network access periods or when the network is congested, it can ensure that data with high processing requirements can be processed.

[0015] In one possible design, the second distributed node further includes processing sub-data in the target data partition according to the priority of the target data partition as indicated by the priority information of the target data partition.

[0016] In this scheme, the second distributed node processes the sub-data in the data partition according to the priority of the data partition. During peak network access periods or when the network is congested, it can ensure that data with high processing requirements can be processed.

[0017] In one possible design, the method further includes: the second distributed node receiving packet loss attribute information of the target data partition sent by the first distributed node; and the second distributed node writing the packet loss attribute information of the target data partition into the target data partition.

[0018] This solution can further ensure that high-demand sub-data can be processed during peak network access periods or when the network is congested.

[0019] In one possible design, the second distributed node processes the sub-data in the target data partition according to the priority of the target data partition indicated by the priority information of the target data partition, including: the second distributed node processes the sub-data in the target data partition according to the priority of the target data partition indicated by the priority information of the target data partition and the packet loss attribute of the target data partition indicated by the packet loss attribute information of the target data partition.

[0020] In this scheme, the second distributed node processes the sub-data in the data partition according to the priority of the data partition and the packet loss attribute. During peak network access periods or when the network is congested, it can further ensure that data with high processing requirements can be processed.

[0021] Thirdly, this application provides a data processing apparatus, comprising: a processing module, configured to determine the priority of each data partition corresponding to a data processing task, the data processing task including: processing operations to be performed on source data; the processing module is further configured to a first distributed node determine a target data partition matching the processing requirements of the sub-data in the source data and the priority of each data partition; and a transceiver module, configured to send the sub-data, the identifier of the target data partition, and the priority information of the target data partition to a second distributed node.

[0022] In one possible design, the processing module is specifically used to: determine the priority of each data partition based on priority setting information.

[0023] In one possible design, the priority setting information includes: first information, which indicates a preset ratio between the number of data partitions of various priorities; or, second information, which indicates a preset ratio of the number of data partitions of various priorities to the total number of data partitions.

[0024] In one possible design, the processing module is further configured to: determine the packet loss attributes of each data partition corresponding to the current data processing task; the transceiver module is further configured to send the packet loss attribute information of the target data partition to the second distributed node; correspondingly, the processing module is specifically configured to: determine the target data partition that matches the processing requirements of the sub-data in the source data, the priority of each data partition and the packet loss attributes.

[0025] In one possible design, the processing module is specifically used to: determine the packet loss attribute of each data partition based on packet loss attribute setting information, the priority of each data partition, and preset conditions; wherein, the preset conditions include: if there is a data partition with a packet loss attribute of packet loss capability in the data partition with the first priority, then all data partitions with the second priority have the packet loss attribute of packet loss capability, and the first priority is higher than the second priority.

[0026] In one possible design, the packet loss attribute setting information includes: third information, which indicates a preset ratio between the number of data partitions for various packet loss attributes; or, fourth information, which indicates a preset ratio of the number of data partitions for each of various packet loss attributes to the total number of data partitions.

[0027] Fourthly, this application provides a data processing apparatus, comprising: a transceiver module, configured to receive sub-data, an identifier of a target data partition, and priority information of the target data partition sent by a first distributed node, wherein the target data partition is a data partition that matches the processing requirements of the sub-data; and a processing module, configured to write the sub-data and the priority information of the target data partition into the target data partition.

[0028] In one possible design, the processing module is further configured to process sub-data in the target data partition according to the priority of the target data partition indicated by the priority information of the target data partition.

[0029] In one possible design, the transceiver module is further configured to receive packet loss attribute information of the target data partition sent by the first distributed node; the processing module is further configured to write the packet loss attribute information of the target data partition into the target data partition.

[0030] In one possible design, the processing module is specifically used to: process the sub-data in the target data partition according to the priority of the target data partition indicated by the priority information of the target data partition and the packet loss attribute of the target data partition indicated by the packet loss attribute information of the target data partition.

[0031] Fifthly, this application provides a data processing method, comprising: a first distributed node determining the priority of each data partition corresponding to a data processing task, wherein the data processing task includes: processing operations to be performed on source data; the first distributed node determining a target data partition matching the processing requirements of the sub-data in the source data and the priority of each data partition.

[0032] In a sixth aspect, this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect and any possible design of the first aspect or the method described in the second aspect and any possible design of the second aspect.

[0033] In a seventh aspect, this application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described in the first aspect and any possible design of the first aspect or the method described in the second aspect and any possible design of the second aspect.

[0034] One embodiment of the above application has the following advantages or beneficial effects: It ensures that data with high processing requirements can be processed during peak network access times or network congestion. This is because the node receiving the source data determines the priority of the data partitions, and based on the processing requirements of the sub-data in the source data and the priority of each data partition, it determines data partitions whose priorities match the processing requirements of the sub-data. This allows sub-data to be written into data partitions whose priorities match the processing requirements of the sub-data. The node processing the data then processes the sub-data in the data partitions according to their priorities. Therefore, this overcomes the problem in the prior art where data is processed indiscriminately, and during peak network access times or network congestion, it cannot guarantee that data with high processing requirements can be processed. Thus, it achieves the technical effect of ensuring that data with high processing requirements can be processed during peak network access times or network congestion.

[0035] Other effects of the above-mentioned alternative methods will be described below in conjunction with specific embodiments. Attached Figure Description

[0036] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0037] Figure 1 This is a possible system architecture diagram related to an embodiment of this application;

[0038] Figure 2 This is an interaction of the data processing method provided in the embodiments of this application. Figure 1 ;

[0039] Figure 3 This is a schematic diagram of the priority of data partitions provided in the embodiments of this application. Figure 1 ;

[0040] Figure 4 This is a schematic diagram of the priority of data partitions provided in the embodiments of this application. Figure 2 ;

[0041] Figure 5 This is an interaction of the data processing method provided in the embodiments of this application. Figure 2 ;

[0042] Figure 6 This is a schematic diagram of the priority and packet loss attributes of the data partition provided in the embodiments of this application. Figure 1 ;

[0043] Figure 7 This is a schematic diagram of the priority and packet loss attributes of the data partition provided in the embodiments of this application. Figure 2 ;

[0044] Figure 8 This is the flow chart of the data processing method provided in the embodiments of this application. Figure 1 ;

[0045] Figure 9 This is the flow chart of the data processing method provided in the embodiments of this application. Figure 2 ;

[0046] Figure 10 This is a block diagram of a data processing apparatus used to implement embodiments of this application.

[0047] Figure 11 This is a block diagram of an electronic device used to implement the data processing method of the embodiments of this application. Detailed Implementation

[0048] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0049] In a distributed system, the node that receives source data input, the node that processes source data, and the node that outputs processed source data can be the same node or different nodes. That is, in a distributed system, there can be nodes that receive source data input, nodes that process source data, and nodes that output processed source data, or there can be nodes that simultaneously have the functions of receiving source data input, processing source data, and outputting processed source data.

[0050] Figure 1 This is a possible system architecture diagram related to an embodiment of this application. See also: Figure 1 The system architecture includes one or more first distributed nodes and one or more second distributed nodes.

[0051] In one approach, the first distributed node can be a node in the distributed system that receives source data input. Sub-data of the source data is written into the data partition of the second distributed node in the distributed system, and the second distributed node processes the sub-data in the data partition.

[0052] The data processing method of this application will be described below using specific embodiments.

[0053] Figure 2 This is an interaction of the data processing method provided in the embodiments of this application. Figure 1 See Figure 2 The method in this embodiment includes:

[0054] Step S201: The first distributed node determines the priority of each data partition corresponding to the data processing task. The data processing task package includes the processing operations that need to be performed on the source data.

[0055] In this embodiment, the first distributed node can be a node that receives source data input, and correspondingly, the source data can be the data input to the first distributed node.

[0056] After the data processing task is determined, a certain number of data partitions will be pre-set according to the data processing task. These certain number of data partitions are the data partitions corresponding to the data processing task.

[0057] The first distributed node adds a key value to each sub-data item included in the source data. Sub-data items with the same key value are written to the same data partition. Before the first distributed node sends the sub-data of the source data to the second distributed node so that the second distributed node can write the sub-data to the corresponding data partition, the first distributed node determines the priority of each data partition corresponding to the data processing task.

[0058] In one approach, the first distributed node determines the priority of each data partition corresponding to the data processing task, including: the first distributed node determines the priority of each data partition corresponding to the data processing task based on priority setting information. Further, the first distributed node determines the priority of each data partition corresponding to the data processing task based on priority setting information, including: "the first distributed node determines the priority of each data partition corresponding to the data processing task based on priority setting information and the distribution of data partitions."

[0059] The priority setting information includes: first information, which indicates a preset ratio between the number of data partitions of various priorities; or, the priority setting information includes: second information, which indicates a preset ratio of the number of data partitions of various priorities to the total number of data partitions.

[0060] The first information can be used to indicate not only the preset ratio between the number of data partitions of various priorities, but also the number of types of data partition priorities.

[0061] For example, if the first information is 3:2, it indicates that there are 2 types of data partition priorities. In a data processing task corresponding to 10 data partitions, 6 data partitions are set as the first priority, and 4 data partitions are set as the second priority, with the first priority being higher than the second priority. If the 10 data partitions are distributed across two nodes, and each node includes 5 data partitions, then for each of the 5 data partitions in each of the two nodes, the priority of 3 data partitions is the first priority, and the priority of 2 data partitions is the second priority. For example... Figure 3 As shown, data partitions 31-33 have the highest priority, data partitions 34-35 have the second highest priority, data partitions 36-38 have the highest priority, and data partitions 39-40 have the second highest priority.

[0062] The second information implicitly or explicitly indicates the preset proportion of the number of data partitions of various priorities to the total number of data partitions, and may also indicate the number of types of data partition priorities.

[0063] For example, the second information is 60%, indicating that there are two data partition priorities. It implicitly indicates that the number of data partitions with the first priority accounts for 60% of the total number of data partitions, and the number of data partitions with the second priority accounts for 40% of the total number of data partitions. The first priority is higher than the second priority.

[0064] For example, the second information is 40%, 40%, and 20%, indicating that there are three data partition priorities. It explicitly indicates that the number of data partitions with the first priority accounts for a preset percentage of the total number of data partitions (40%), the number of data partitions with the second priority accounts for a preset percentage of the total number of data partitions (40%), and the number of data partitions with the third priority accounts for a preset percentage of the total number of data partitions (20%). The first priority is higher than the second priority, and the second priority is higher than the third priority. In the case of a data processing task corresponding to 10 data partitions, 4 data partitions are set as the first priority, 4 data partitions as the second priority, and 2 data partitions as the third priority. If the 10 data partitions are distributed across two nodes, and each node includes 5 data partitions, then for each of the 5 data partitions in each of the two nodes, the priority of 2 data partitions is the first priority, the priority of 2 data partitions is the second priority, and the priority of 1 data partition is the third priority. Figure 4As shown, data partitions 41-42 have the highest priority, data partitions 43-44 have the second highest priority, and data partition 45 has the third highest priority; data partitions 46-47 have the highest priority, partitions 48-49 have the second highest priority, and partition 50 has the third highest priority.

[0065] The first distributed node determines the priority of each data partition based on the priority setting information of the data partition, which can improve the efficiency of determining the priority of data partitions and make the ratio between data partitions of various priorities more reasonable.

[0066] In another approach, the first distributed node determines the priority of each data partition corresponding to the data processing task, including: the first distributed node randomly determines the priority of each data partition corresponding to the data processing task.

[0067] Step S202: The first distributed node determines the target data partition that matches the processing requirements of the sub-data included in the source data and the priority of each data partition.

[0068] It is understood that the target data partition in this embodiment is a target data partition whose priority matches the processing requirements of the sub-data. These processing requirements include accuracy requirements and / or timeliness requirements, and may also include other requirements, which are not limited in this embodiment.

[0069] For each sub-data item included in the source data: the first distributed node determines the target data partition that matches the processing requirements of that sub-data item based on the processing requirements of the sub-data item included in the source data and the priority of each data partition. It is understood that the principle of sub-data items with the same key value corresponding to the same data partition is still followed here.

[0070] For example, the data processing task includes data partitions with first priority and second priority. If the timeliness requirement for processing sub-data 1 includes being the sub-data with the highest timeliness requirement among all sub-data, then the first priority data partition matches the processing requirement of this sub-data, and a first priority data partition can be determined as the target data partition corresponding to sub-data 1. If the timeliness requirement for processing sub-data 2 includes being second only to the highest timeliness requirement, then the second priority data partition, or the first priority data partition, or, if there are no remaining first priority data partitions, the second priority data partition matches the processing requirement of sub-data 2, and a second priority data partition can be determined as the target data partition corresponding to sub-data 2. If the timeliness requirement for processing sub-data 3 includes being the sub-data with the highest accuracy requirement among all sub-data, then the first priority data partition matches the processing requirement of this sub-data, and a first priority data partition can be determined as the target data partition corresponding to sub-data 3.

[0071] Step S203: The first distributed node sends the sub-data, the identifier of the target data partition corresponding to the sub-data, and the priority information of the target data partition to the second distributed node.

[0072] After the first distributed node determines the sub-data and the target data partition corresponding to the sub-data, it sends the identifier of the sub-data, the target data partition corresponding to the sub-data, and the priority information of the target data partition to the second distributed node.

[0073] Step S204: The second distributed node writes the priority information of the sub-data and target data partitions into the target data partition.

[0074] After receiving the sub-data, the identifier of the target data partition corresponding to the sub-data, and the priority information of the target data partition, the second distributed node writes the sub-data and the priority information of the target data partition into the target data partition indicated by the identifier of the target data partition.

[0075] Step S205: The second distributed node processes the sub-data in the target data partition according to the priority of the target data partition as indicated by the priority information of the target data partition.

[0076] For example, if the priority information of the target data partition indicates that the target data partition has the highest priority—first priority—then the second distributed node will process the sub-data in the target data partition first. Alternatively, if the priority information of the target data partition indicates that the target data partition has the highest priority—second priority, and second priority is lower than first priority, then the second distributed node will process the data in the first priority data partition before processing the sub-data in the target data partition.

[0077] In this embodiment, the node receiving the source data determines the priority of the data partition, and determines the data partition whose priority matches the processing requirements of the sub-data of the source data and the priority of each data partition, so that the sub-data can be written into the data partition whose priority matches the processing requirements of the sub-data. In this way, when the node processing the data processes the data according to the priority of the data partition, it can process the data according to the different processing requirements of the data. During peak network access or network congestion, it can ensure that data with high processing requirements can be processed.

[0078] To further ensure that high-demand sub-data can be processed during peak network access periods or when the network is congested, another data processing method is proposed below.

[0079] Figure 5 This is an interaction of the data processing method provided in the embodiments of this application. Figure 2 See Figure 5 The method in this embodiment includes:

[0080] Step S501: The first distributed node determines the priority and packet loss attributes of each data partition corresponding to the data processing task. The data processing task package is the source data that needs to be processed.

[0081] In this embodiment, the meanings of the first distributed node and the source data are the same as those of other nodes. Figure 2 The implementation shown has the same meaning. The method by which the first distributed node determines the priority of each data partition corresponding to the data processing task is the same as... Figure 2 The implementation details shown are not repeated here.

[0082] The following section explains the packet loss attributes of each data partition corresponding to the data processing task determined by the first distributed node.

[0083] In one approach, the first distributed node determines the packet loss attribute of each data partition, including: the first distributed node determines the packet loss attribute of each data partition based on packet loss attribute settings, the priority of each data partition, and preset conditions. The preset condition is: if a data partition with a first-priority data partition has a packet loss attribute of "packet lossable," then the packet loss attribute of a data partition with a second-priority data partition is also "packet lossable," with the first priority being higher than the second priority. The packet loss attribute can be either "non-packet lossable" or "packet lossable."

[0084] The packet loss attribute setting information includes: third information, which indicates the preset ratio between the number of data partitions for various packet loss attributes; or, the packet loss attribute setting information includes fourth information, which indicates the preset ratio of the number of data partitions for various packet loss attributes to the total number of data partitions.

[0085] The third piece of information can indicate the preset ratio between the number of data partitions for various packet loss attributes. The third piece of information can be the same as or different from the first piece of information.

[0086] For example, if the third information is 3:2, it indicates that the ratio of data partitions with a packet loss attribute of "non-droppable" to data partitions with a packet loss attribute of "droppable" is 3:2. A data processing task corresponds to 10 data partitions, distributed across two nodes. Each node contains 5 data partitions, and within each node, 3 partitions have first priority and 2 partitions have second priority, with first priority being higher than second priority. Therefore, the packet loss attribute of the 3 first-priority data partitions in each node is set to "non-droppable," and the packet loss attribute of the 2 second-priority data partitions is set to "droppable."

[0087] For example, if the third information is 2:3, it indicates that the ratio of data partitions with a non-lossable packet loss attribute to data partitions with a lossable packet loss attribute is 2:3. A data processing task corresponds to 10 data partitions, distributed across two nodes. Each node contains 5 data partitions, with 3 being first-priority partitions and 2 being second-priority partitions, where first priority is higher than second priority. Therefore, the packet loss attribute of the 2 first-priority partitions on each node is set to non-lossable, and the packet loss attribute of the 1 first-priority partition and the 2 second-priority partitions is set to lossable. For example... Figure 6 As shown, data partitions 61-62 have the highest priority and are non-lossable; data partition 63 has the highest priority and is lossable; data partitions 64-65 have the second highest priority and are lossable; data partitions 66-67 have the highest priority and are non-lossable; data partition 68 has the highest priority and is lossable; and data partitions 69-70 have the second highest priority and are lossable.

[0088] For example, if the third information is 2:3, it indicates that the ratio of data partitions with a non-lossable packet loss attribute to data partitions with a lossable packet loss attribute is 2:3. A data processing task corresponds to 10 data partitions, distributed across two nodes. Each node contains 5 data partitions. Within each node, there are 2 first-priority partitions, 2 second-priority partitions, and 1 third-priority partition. Therefore, the packet loss attribute of the 2 first-priority partitions on each node is set to non-lossable, and the packet loss attribute of the 2 second-priority partitions and the 1 third-priority partition is set to lossable. For example... Figure 7 As shown, data partitions 71-72 have the highest priority and are non-lossable; data partitions 73-74 have the second highest priority and are lossable; data partition 75 has the second highest priority and is lossable; data partitions 76-77 have the highest priority and are non-lossable; data partitions 78-79 have the second highest priority and are lossable; and data partition 80 has the third highest priority and is lossable.

[0089] For the fourth information, the fourth information implicitly or explicitly indicates the preset proportion of the total number of data partitions of each priority level.

[0090] For example, the fourth information is 60%, which implicitly indicates that the number of data partitions with the attribute of non-packet loss accounts for a preset proportion of 60% of the total number of data partitions, and the number of data partitions with the attribute of packet loss accounts for a preset proportion of 40% of the total number of data partitions.

[0091] For example, the second information is 40% and 60%, indicating that the number of data partitions with the attribute of non-packet loss accounts for a preset percentage of the total number of data partitions, which is 40%, and the number of data partitions with the attribute of packet loss accounts for a preset percentage of the total number of data partitions, which is 60%.

[0092] This method can improve the efficiency of determining the packet loss attributes of data partitions and make the ratio between data partitions with various packet loss attributes more reasonable.

[0093] In another approach, the first distributed node determines the priority and preset conditions of each data partition, including: the first distributed node determines the packet loss attributes of each data partition based on the priority and preset conditions of the data partition.

[0094] Step S502: The first distributed node determines the target data partition that matches the processing requirements of the sub-data included in the source data, the priority of each data partition, and the packet loss attributes.

[0095] Understandably, the target data partition in this embodiment is a target data partition whose priority and packet loss attributes match the processing requirements of the sub-data. These processing requirements include accuracy requirements and / or timeliness requirements, and may also include other requirements, which are not limited in this embodiment.

[0096] For each sub-data item included in the source data: the first distributed node determines the target data partition that matches the processing requirements of the sub-data item based on the processing requirements of the sub-data item included in the source data, the priority of each data partition, and the packet loss attributes. It is understood that the principle of sub-data items with the same key value corresponding to the same data partition is still followed here.

[0097] For example, the data processing task includes data partitions with first priority and data partitions with second priority. If the processing requirement for sub-data 1 has the highest timeliness requirement among all sub-data, and the accuracy requirement is also the highest accuracy requirement among all sub-data, then the data partition with first priority and non-lossable packet loss attribute matches the processing requirement for that sub-data. If the processing requirement for sub-data 2 has the highest timeliness requirement among all sub-data, and the accuracy requirement is second only to the highest accuracy requirement and packet loss is allowed, then the data partition with first priority and lossable packet loss attribute matches the processing requirement for sub-data 2. If the processing requirement for sub-data 2 has the highest timeliness requirement among all sub-data, and the accuracy requirement is second only to the highest accuracy requirement and packet loss is not allowed, then the data partition with first priority and non-lossable packet loss attribute matches the processing requirement for sub-data 2. If the processing requirements for sub-data 3 have a timeliness requirement second only to the highest timeliness requirement and an accuracy requirement that is the highest among all sub-data, then the data partition with the highest priority and the packet loss attribute (non-packet loss) matches the processing requirements of that sub-data.

[0098] Step S503: The first distributed node sends the sub-data, the identifier of the target data partition corresponding to the sub-data, the priority information of the target data partition, and the packet loss attribute information to the second distributed node.

[0099] After the first distributed node determines the sub-data and the target data partition corresponding to the sub-data, it sends the identifiers of the sub-data and the target data partition, as well as the priority information and packet loss attribute information of the target data partition, to the second distributed node.

[0100] Step S504: The second distributed node writes the priority information and packet loss attribute information of the sub-data and the target data partition into the target data partition.

[0101] After receiving the sub-data, the identifier of the target data partition corresponding to the sub-data, and the priority information and packet loss attribute information of the target data partition, the second distributed node writes the sub-data, the priority information and packet loss attribute information of the target data partition into the target data partition indicated by the identifier of the target data partition.

[0102] Step S505: The second distributed node processes the sub-data in the target data partition according to the priority information of the target data partition and the packet loss attribute information of the target data partition.

[0103] For example, if the priority information of the target data partition indicates that its priority is the highest (first priority) and its packet loss attribute is non-droppable, then the second distributed node will process the sub-data in the target data partition first. Alternatively, if the priority information indicates that the target data partition has the highest priority (first priority) and its packet loss attribute is droppable, then the second distributed node will process the data in the first priority data partition (with non-droppable packet loss) before processing the sub-data in the target data partition. Or, if the priority information indicates that the target data partition has the second priority and its packet loss attribute is droppable (second priority is lower than first priority), then the second distributed node will process the sub-data in the first priority data partition and the sub-data in the second priority data partition (with non-droppable packet loss) before processing the sub-data in the target data partition.

[0104] The method in this embodiment can process data differently according to the different processing requirements, ensuring that sub-data with high processing requirements can be processed during peak network access periods or when the network is congested.

[0105] The following is combined with Figure 8 and Figure 9 This section describes a data processing method for another scenario. This scenario involves a first distributed node where the node receiving the source data input, the node processing the source data, and the node outputting the processed source data are all part of this scenario.

[0106] Figure 8 This is the flow of the data processing method provided in this application. Figure 1 See Figure 8 The method in this embodiment includes:

[0107] Step S801: The first distributed node determines the priority of each data partition corresponding to the data processing task. The data processing task includes: the processing operations that need to be performed on the source data.

[0108] Step S802: The first distributed node determines the target data partition that matches the processing requirements of the sub-data included in the source data and the priority of each data partition.

[0109] Step S803: The first distributed node writes the priority information of the sub-data and the target data partition into the target data partition; or, the first distributed node writes the sub-data into the target data partition.

[0110] Step S804: The first distributed node processes the sub-data included in the target data partition according to the priority of the target data partition.

[0111] For a specific implementation of this embodiment, please refer to... Figure 2 The descriptions of the relevant content in the illustrated embodiments will not be repeated in this embodiment.

[0112] The method in this embodiment can process data differently according to the different processing requirements, ensuring that sub-data with high processing requirements can be processed during peak network access periods or when the network is congested.

[0113] Figure 9 This is the flow of the data processing method provided in this application. Figure 2 See Figure 9 The method in this embodiment includes:

[0114] Step S901: The first distributed node determines the priority and packet loss attributes of each data partition corresponding to the data processing task. The data processing task includes: the processing operations that need to be performed on the source data.

[0115] Step S902: The first distributed node determines the target data partition that matches the processing requirements of the sub-data included in the source data, the priority of each data partition, and the packet loss attributes.

[0116] Step S903: The first distributed node writes the priority information and packet loss attribute information of the sub-data and the target data partition into the target data partition; or, the first distributed node writes the sub-data into the target data partition.

[0117] Step S904: The first distributed node processes the sub-data included in the target data partition according to the priority and packet loss attributes of the target data partition.

[0118] For a specific implementation of this embodiment, please refer to... Figure 5 The descriptions of the relevant content in the illustrated embodiments will not be repeated in this embodiment.

[0119] The method in this embodiment can process data differently according to the different processing requirements, ensuring that sub-data with high processing requirements can be processed during peak network access periods or when the network is congested.

[0120] The data processing method provided in this application has been described above. The data processing apparatus provided in this application will be described below.

[0121] Figure 10 This is a block diagram of a data processing apparatus used to implement embodiments of this application. For example... Figure 10 As shown, another embodiment of this application provides a data processing apparatus 100. The data processing apparatus 100 can be a first distributed node or a component of the first distributed node (e.g., an integrated circuit, a chip, etc.). The data processing apparatus 100 can also be a second distributed node or a component of the second distributed node (e.g., an integrated circuit, a chip, etc.). The data processing apparatus 100 can also be other data processing modules used to implement the methods of the first distributed node or the method corresponding to the first distributed node in the method embodiments of this application. The data processing apparatus 100 can include: a processing module 1001 (processing unit) and a transceiver module 1002 (transceiver unit).

[0122] In one possible design, such as Figure 10 One or more modules may be implemented by one or more processors; or by one or more processors and transceivers, which is not limited in this application embodiment. The processors and transceivers can be set individually or integrated.

[0123] Optionally, each module in the data processing apparatus of this application embodiment can be used to execute the functions described in this application embodiment. Figure 2 The methods described in the illustrated embodiments; or the various modules in the data processing apparatus of this application embodiments can be used to execute the methods described in the embodiments of this application. Figure 5 The methods described in the illustrated embodiments.

[0124] In one embodiment, the data processing apparatus 100 includes a processing module 1001 and a transceiver module 1002.

[0125] The processing module 1001 is used to determine the priority of each data partition corresponding to the data processing task, the data processing task including: processing operations to be performed on the source data; the processing module 1001 is also used to determine the target data partition that matches the processing requirements of the sub-data included in the source data and the priority of each data partition according to the processing requirements of the sub-data and the priority of each data partition; the transceiver module 1002 is used to send the sub-data, the identifier of the target data partition and the priority information of the target data partition to the second distributed node.

[0126] Optionally, the processing module 1001 is specifically used to: determine the priority of each data partition according to the priority setting information.

[0127] Optionally, the priority setting information includes: first information, which indicates a preset ratio between the number of data partitions of various priorities; or, second information, which indicates a preset ratio of the number of data partitions of various priorities to the total number of data partitions.

[0128] Optionally, the processing module 1001 is further configured to: determine the packet loss attributes of each data partition corresponding to the current data processing task; the transceiver module 1002 is further configured to send the packet loss attribute information of the target data partition to the second distributed node; correspondingly, the processing module 1001 is specifically configured to: determine the target data partition that matches the processing requirements of the sub-data included in the source data, the priority of each data partition and the packet loss attributes.

[0129] Optionally, the processing module 1001 is specifically used to: determine the packet loss attribute of each data partition according to the packet loss attribute setting information, the priority of each data partition and the preset conditions; wherein, the preset conditions include: if there is a data partition with a packet loss attribute of packet loss in the first priority data partition, then the packet loss attribute of the data partitions in the second priority is packet lossable, and the first priority is higher than the second priority.

[0130] Optionally, the packet loss attribute setting information includes: third information, which indicates a preset ratio between the number of data partitions for various packet loss attributes; or, fourth information, which indicates a preset ratio of the number of data partitions for various packet loss attributes to the total number of data partitions.

[0131] The data processing device in the above method can be the first distributed node or a component of the first distributed node. It can be used to execute the technical solution corresponding to the first distributed node in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0132] In another embodiment, the data processing device 100 includes a processing module 1001 and a transceiver module 1002.

[0133] The transceiver module 1002 is used to receive sub-data, the identifier of the target data partition, and the priority information of the target data partition sent by the first distributed node, wherein the target data partition is a data partition that matches the processing requirements of the sub-data; the processing module 1001 is used to write the sub-data and the priority information of the target data partition into the target data partition.

[0134] Optionally, the processing module 1001 is further configured to process sub-data in the target data partition according to the priority of the target data partition indicated by the priority information of the target data partition.

[0135] Optionally, the transceiver module 1002 is further configured to receive packet loss attribute information of the target data partition sent by the first distributed node; the processing module 1001 is further configured to write the packet loss attribute information of the target data partition into the target data partition.

[0136] Optionally, the processing module 1001 is specifically used to: process the sub-data in the target data partition according to the priority of the target data partition indicated by the priority information of the target data partition and the packet loss attribute of the target data partition indicated by the packet loss attribute information of the target data partition.

[0137] The data processing device in the above method can be a second distributed node or a component within a second distributed node. It can be used to execute the technical solution corresponding to the second distributed node in the above method embodiments, and its implementation principle and technical effects are similar, so they will not be repeated here.

[0138] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0139] like Figure 11 The diagram shown is a block diagram of an electronic device for a data processing method according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0140] like Figure 11As shown, the electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 11 Take a processor 1101 as an example.

[0141] The memory 1102 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the data processing method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the data processing method provided in this application.

[0142] Memory 1102, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to the data processing method in the embodiments of this application (e.g., appendix). Figure 10 The acquisition module 1001, verification module 1002, and determination module 1003 are shown. The processor 1101 executes various functional applications and data processing of the server by running non-transient software programs, instructions, and modules stored in the memory 1102, thereby implementing the data processing method in the above method embodiments.

[0143] Memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device according to the data processing method. Furthermore, memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1102 may optionally include memory remotely located relative to processor 1101, and these remote memories can be connected to the electronic device implementing the data processing method via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The electronic device implementing the data processing method may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0145] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0146] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0149] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0150] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0151] According to the technical solution of the embodiments of this application, the node receiving the source data determines the priority of the data partition, and determines the data partition whose priority matches the processing requirements of the sub-data of the source data and the priority of each data partition, so that the sub-data can be written into the data partition whose priority matches the processing requirements of the sub-data. In this way, when the node processing the data processes the data according to the priority of the data partition, it can process the data differently according to the processing requirements of the data. During peak network access or network congestion, it can ensure that data with high processing requirements can be processed.

[0152] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: The first distributed node determines the priority and the packet loss attribute of each data partition corresponding to the data processing task, wherein the data processing task comprises a processing operation required to be performed on source data; The first distributed node determines a target data partition matching the processing requirement of the sub-data according to the processing requirement of the sub-data in the source data, the priority and the packet loss attribute of each data partition; The first distributed node sends the sub-data, the identifier of the target data partition, the priority information and the packet loss attribute information of the target data partition to a second distributed node.

2. The method of claim 1, wherein, The first distributed node determines the priority of each data partition corresponding to the current data processing task, comprising: The first distributed node determines the priority of each data partition according to the priority setting information.

3. The method of claim 2, wherein, The priority setting information comprises: First information for indicating a preset ratio between the number of data partitions of various priorities; or Second information for indicating a preset proportion of the total number of data partitions occupied by the number of data partitions of various priorities.

4. The method of claim 1, wherein, The first distributed node determines the packet loss attribute of each data partition corresponding to the data processing task, comprising: The first distributed node determines the packet loss attribute of each data partition according to the packet loss attribute setting information, the priority of each data partition and a preset condition; The preset condition comprises: if there is a data partition with a packet loss attribute of packet loss tolerance in a data partition of a first priority, the packet loss attribute of a data partition of a second priority is packet loss tolerance, and the first priority is higher than the second priority.

5. The method of claim 4, wherein, The packet loss attribute setting information comprises: Third information for indicating a preset ratio between the number of data partitions of various packet loss attributes; or Fourth information for indicating a preset proportion of the total number of data partitions occupied by the number of data partitions of various packet loss attributes.

6. A data processing method, characterized by, The method comprises the following steps: The second distributed node receives the sub-data, the identifier of the target data partition, the priority information and the packet loss attribute information of the target data partition sent by the first distributed node, wherein the target data partition is a data partition matching the processing requirement of the sub-data; The second distributed node writes the sub-data, the priority information and the packet loss attribute information of the target data partition into the target data partition; The second distributed node processes the sub-data in the target data partition according to the priority of the target data partition indicated by the priority information of the target data partition and the packet loss attribute of the target data partition indicated by the packet loss attribute information of the target data partition.

7. A data processing apparatus, characterized by The method comprises the following steps: The processing module is configured to determine the priority and the packet loss attribute of each data partition corresponding to the data processing task, wherein the data processing task comprises a processing operation required to be performed on source data; The processing module is further configured to determine a target data partition matching the processing requirement of the sub-data according to the processing requirement of the sub-data in the source data, the priority and the packet loss attribute of each data partition; The transceiving module is configured to send the sub-data, the identifier of the target data partition, the priority information of the target data partition, and the packet loss attribute information to a second distributed node.

8. A data processing apparatus, characterized by, The method comprises the following steps: The transceiving module is configured to receive the sub-data, the identifier of the target data partition, the priority information of the target data partition, and the packet loss attribute information sent by the first distributed node, wherein the target data partition is a data partition matching the processing requirement of the sub-data. The processing module is configured to write the sub-data, the priority information of the target data partition, and the packet loss attribute information into the target data partition. The sub-data in the target data partition is processed according to the priority of the target data partition indicated by the priority information of the target data partition and the packet loss attribute of the target data partition indicated by the packet loss attribute information of the target data partition.

9. An electronic device, comprising: The method comprises the following steps: At least one processor; And A memory connected with the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any one of claims 1-5 or the method in claim 6.

10. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to execute the method in any one of claims 1-5 or the method in claim 6.

11. A data processing method, characterized by, The method comprises the following steps: The first distributed node determines the priority and the packet loss attribute of each data partition corresponding to a data processing task, wherein the data processing task comprises a processing operation required to be performed on source data. The first distributed node determines a target data partition matching the processing requirement of the sub-data according to the processing requirement of the sub-data in the source data, the priority of each data partition, and the packet loss attribute.

Images