Data distribution method and system

Abstract

This application discloses a data distribution method, which includes: receiving row record data from a data source and determining the data's ownership tag; determining the target thread corresponding to the data based on the pre-aggregation result of each tag in the previous preset time period or the distributed data information recorded in the circular array corresponding to each thread; and distributing the data to the target thread. This application also discloses a data distribution system, electronic device, and computer-readable storage medium. Therefore, it is possible to evenly allocate corresponding downstream threads to data traffic based on the pre-aggregation result or records in the circular array, achieving balanced processing of burst traffic and reducing the number of small files generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data distribution method and system, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In a data warehouse layered architecture system, data cleaning, desensitization, columnar compression and other steps need to be performed in data conversion from an ODS layer to a DWD layer. In the conversion process of data (for example, user behavior point-in-time data) from the ODS layer to the DWD layer, in order to solve the problem of serious resource consumption caused by repeated data ingestion under a super large incremental scale, a real-time shunting mechanism is introduced, and through a pre-set table partitioning rule, in the form of multi-table and multi-partition control, the data ingestion quantity of a downstream task file is maximally reduced to reduce resource consumption. However, since the real-time shunting generates a large number of small files when performing file cutting each time, the read-write pressure is increased, and the resource consumption is also increased in the directory reading process of the downstream, which causes the task execution to be timed out. However, through analysis, many small files are caused by unreasonable partition concurrency degree allocation. SUMMARY

[0003] The main purpose of the present application is to provide a data distribution method and system, an electronic device and a computer readable storage medium, which aims to solve the problem of how to realize burst traffic balancing processing and reduce the number of small files generated during data shunting.

[0004] To achieve the above purpose, the embodiments of the present application provide a data distribution method, which comprises the following steps:

[0005] receiving row record data issued by a data source, and determining the ownership label of the data;

[0006] determining the target thread corresponding to the data according to the pre-aggregation result of each label in the last preset time period or the distributed data information recorded in the ring array corresponding to each thread;

[0007] distributing the data to the target thread.

[0008] Optionally, the method further comprises the following steps:

[0009] at the end of the current preset time period, uniformly aggregating the distributed data information recorded in the ring array reported by each thread to provide the pre-aggregation result of each label for the next preset time period.

[0010] Optionally, the method further comprises the following steps after determining the ownership label of the data:

[0011] calculating the size of the data.

[0012] Optionally, the determining the target thread corresponding to the data according to the pre-aggregated result of each label in the last preset time period or the distributed data information recorded in the ring array corresponding to each thread comprises:

[0013] obtaining a pre-aggregated result in the last preset time period;

[0014] in the case where the label of the data is contained in the pre-aggregated result, selecting one from the thread distribution interval corresponding to the label in the pre-aggregated result as the target thread of the data;

[0015] in the case where the label of the data is not contained in the pre-aggregated result, assigning a corresponding target thread to the data according to the distributed data information recorded in the ring array.

[0016] Optionally, the pre-aggregated result comprises a thread distribution interval assigned to each label according to the distributed data information of all labels in all threads in the last preset time period; each thread corresponds to a ring array, and the ring array records the distributed data information in the thread in the preset time period as a window.

[0017] Optionally, the assigning a corresponding target thread to the data according to the distributed data information recorded in the ring array comprises:

[0018] in the case where the label has a historical distribution record and the thread corresponding to the historical distribution record does not reach a preset threshold, taking the thread as the target thread corresponding to the data;

[0019] in the case where the label has a historical distribution record and the thread corresponding to the historical distribution record reaches a preset threshold, or the label has no historical distribution record, newly applying a thread as the target thread corresponding to the data.

[0020] Optionally, the method further comprises, in the case where the label of the data is not contained in the pre-aggregated result:

[0021] determining the target thread of the data according to a label distribution rule preset by a user.

[0022] Optionally, the uniformly aggregating the distributed data information recorded in the ring array reported by each thread to provide a pre-aggregated result of each label for a next preset time period comprises:

[0023] summarizing all distributed labels and the distributed data size of each label in the current preset time period, and adopting a signed knapsack algorithm to assign a corresponding thread distribution interval to each label according to the proportion of the distributed data size of all labels and a preset threshold of each thread.

[0024] Optionally, the unified aggregation according to the distributed data information recorded in the ring array reported by each thread further comprises:

[0025] On the basis of the signing knapsack algorithm, all tags are uniformly sorted, and each tag is allocated a corresponding thread distribution interval according to the sorting, the proportion of the distributed data size of all tags, and the preset threshold of each thread.

[0026] Optionally, the unified aggregation according to the distributed data information recorded in the ring array reported by each thread further comprises:

[0027] The distributed data size of all tags in the current preset time period is summarized, and all tags are allocated to all threads according to the proportion of the distributed data size between tags, to obtain a thread distribution interval corresponding to each tag.

[0028] In addition, to achieve the above-mentioned purpose, the embodiment of the present application further provides a data distribution system, the system comprises:

[0029] A receiving module is configured to receive row record data issued by a data source and determine the home tag of the data.

[0030] A distribution module is configured to determine the target thread corresponding to the data according to the pre-aggregation result of each tag in the last preset time period or the distributed data information recorded in the ring array corresponding to each thread.

[0031] A distribution module is configured to distribute the data to the target thread.

[0032] To achieve the above-mentioned purpose, the embodiment of the present application further provides an electronic device, which comprises a memory, a processor and a data distribution program stored in the memory and executable on the processor, and the data distribution program is executed by the processor to realize the data distribution method as described above.

[0033] To achieve the above-mentioned purpose, the embodiment of the present application further provides a computer readable storage medium, which stores a data distribution program, and the data distribution program is executed by a processor to realize the data distribution method as described above.

[0034] The data distribution method, system, electronic device and computer readable storage medium provided by the embodiments of the present application can allocate corresponding downstream threads to data traffic according to the pre-aggregation result or the record in the ring array, avoid unreasonable partition concurrent degree allocation of data traffic, realize balanced processing of burst traffic, reduce the number of small file generation, and reduce file read-write pressure and downstream resource consumption. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 An application environment architecture diagram for implementing various embodiments of the present application;

[0036] Figure 2 A flowchart of a data distribution method according to the first embodiment of the present application;

[0037] Figure 3 A refinement flowchart of step S202 in the first embodiment of the present application; Figure 2

[0038] Figure 4 Another refinement flowchart of step S202 in the first embodiment of the present application; Figure 2

[0039] A flowchart of a data distribution method according to the second embodiment of the present application; Figure 5

[0040] Another form of flowchart of the data distribution method according to the second embodiment of the present application; Figure 6

[0041] A hardware architecture diagram of an electronic device according to the third embodiment of the present application; Figure 7

[0042] A module diagram of a data distribution system according to the fourth embodiment of the present application; Figure 8

[0043] A module diagram of a data distribution system according to the fifth embodiment of the present application. Figure 9 DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0045] ​​It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0046] The following provides an explanation of the terms used in this application:

[0047] The data warehouse layered architecture mainly includes ODS (Operational Data Store), DWD (data warehouse details), DWM (Data Warehouse Middle), DWS (Data Warehouse Service), and ADS (Application Data Service).

[0048] ODS Layer: Operational Data Storage Layer, also known as the raw data layer or source layer. Data from the data source enters this layer after being extracted, cleaned, and transferred (ETL). The ODS layer serves as a preparation area for the subsequent data warehouse layer, providing raw data to the DWD layer and minimizing the impact on business systems.

[0049] The DWD layer serves as an isolation layer between the business layer and the data warehouse, maintaining the same data granularity as the ODS layer. The DWD layer primarily performs data cleaning and normalization operations on the ODS layer, such as removing empty data, dirty data, and outliers.

[0050] Shuffle: An operator in this application that is located between the data source and the data output, performs aggregation calculations based on data distributed in the past time period (five minutes) and provides predictions (i.e., pre-aggregation) for the next time period (five minutes), and performs data distribution.

[0051] Rowdata: Also known as row record data, in this application it refers to the record corresponding to the selected row that is related to the user's event tracking behavior data issued by the data source.

[0052] A circular array is a data structure used to represent a fixed-size, contiguous buffer, suitable for caching data streams. In this application, it is used to record data information distributed in each downstream thread (Subtask).

[0053] Please see Figure 1 , Figure 1 This application provides an application environment architecture diagram to implement various embodiments of the present application. The present application can be applied to application environments including, but not limited to, data source 2, Shuffle operator 4, and Writer operator 6. The data source 2, Shuffle operator 4, and Writer operator 6 can be applied to a data warehouse layered architecture.

[0054] The data source 2 is used to send row data to the shuffle operator 4.

[0055] The Shuffle operator 4 is used to perform aggregation calculations based on data distributed in the past preset time period (e.g., five minutes) and provide a prediction (pre-aggregation result) for the next preset time period, and perform data distribution. It aims to achieve balanced processing of burst traffic through pre-aggregation and reduce the number of small files generated.

[0056] The Writer operator 6 is used to receive the data processed by the Shuffle operator 4.

[0057] In this application, a traffic splitting scheme can be used for user tracking behavior data or other data during the data transformation from the ODS layer to the DWD layer. However, real-time traffic splitting generates a large number of small files each time file slicing is performed, leading to increased read and write pressure and increased resource consumption during downstream directory reading, causing task execution timeouts. Analysis revealed that many small files are due to unreasonable partition concurrency allocation. Therefore, by adding the Shuffle operator 4, the Rowdata from the data source 2 is evenly distributed to multiple downstream threads, resulting in more reasonable partition concurrency and reducing the number of small files generated.

[0058] Example 1

[0059] like Figure 2 The diagram shown is a flowchart of a data distribution method proposed in the first embodiment of this application. It is understood that the flowchart in this method embodiment is not intended to limit the order of execution steps. Some steps in the flowchart can be added to or removed as needed. It is worth noting that the execution entity of this method can be an electronic device containing the Shuffle operator, or it can be the Shuffle operator itself.

[0060] The method includes the following steps:

[0061] S200: Receive row record data from the data source and determine the tag of the data.

[0062] The label is used to mark the group to which the data belongs. The group can be user-defined, for example, which type of business the data belongs to.

[0063] In this embodiment, users can define their own rules for calculating the attribution tags of the data. Once the attribution tags of the data are determined according to the user-defined calculation rules, the data can be distributed based on the tags.

[0064] In a preferred embodiment, it is also necessary to calculate the size of the data, that is, to calculate the number of bytes required, so as to record the size of the distributed data in subsequent recordings.

[0065] S202, determine the target thread (SubtaskId) corresponding to the data based on the pre-aggregation results or records in the circular array.

[0066] To balance data traffic and reduce the number of small files generated, the data needs to be distributed to different downstream threads according to certain rules. The thread to which the data is distributed is the corresponding target thread. In one embodiment, the target thread of the data can first be inferred based on the pre-aggregation results of the previous preset time period (e.g., five minutes). If there are no corresponding pre-aggregation results, the data can then be distributed according to the records in the circular array corresponding to each thread.

[0067] For more details, please refer to the following: Figure 3 This is a detailed flowchart of step S202 described above. It is understood that this flowchart is not intended to limit the order of execution steps. Steps in this flowchart can be added to or removed as needed. In this embodiment, step S202 specifically includes:

[0068] S2020: Obtain the pre-aggregation results of the previous preset time period.

[0069] In this embodiment, each thread maintains a record of the data distribution corresponding to all tags, which is recorded in a circular array. The array is rolled once every preset time period (e.g., five minutes), and the distributed data information (ArrayInfo) recorded in the circular array is reported to the JobManager for aggregation calculation.

[0070] JobManager summarizes the distributed data information of all threads in the previous preset time period, divides the concurrency according to the proportion of all tags, that is, based on the distributed data information of the previous preset time period (first five minutes), it infers the thread distribution range corresponding to each tag in the next preset time period (last five minutes) and obtains the pre-aggregation result.

[0071] S2022, when the pre-aggregation result contains the label of the data, select one from the thread distribution interval corresponding to the label in the pre-aggregation result as the target thread of the data.

[0072] If the pre-aggregation result contains the tag, it means that the data corresponding to the tag has been distributed in the previous preset time period, and JobManager has inferred the thread distribution range corresponding to the tag. Therefore, a thread can be randomly selected from the thread distribution range as the target thread for the data.

[0073] S2024, when the pre-aggregation result does not contain the label of the data, the corresponding target thread is assigned to the data according to the distributed data information recorded in the circular array.

[0074] In this embodiment, each thread corresponds to a circular array. The circular array uses a preset time period (e.g., five minutes) as a scrolling window to record the data information distributed within that thread during that time period. The key-value pairs maintained in the circular array include the thread ID (SubTaskId), array index, tag name, and the amount of data allocated to each tag. Furthermore, the circular array cleans up data from five minutes ago based on the current millisecond's minute and the recorded data's TTL (Time To Live). The circular array maintains two pointers, using the movement of the left and right pointers to determine the start and end of the window. The circular array also provides an interface for external access to the distributed data information.

[0075] Specifically, the process first determines whether the tag has a historical distribution record, that is, whether the tag's data has ever been distributed to a particular thread. If a historical distribution record exists, the process then determines whether the distributed data in that thread has reached a preset threshold (e.g., 8GB) based on the size of the distributed data recorded in the circular array of the thread corresponding to that historical distribution record. The threshold can be the total amount of data that the downstream process can handle divided by the number of threads. If the threshold has not been reached, it means that the thread can process more data, and the thread is designated as the target thread for the data. If the threshold has been reached, it means that the thread can no longer process more data, and a new thread is created as the target thread for the data. If the tag has no historical distribution record, that is, if the tag is appearing for the first time, a new thread is also created as the target thread for the data.

[0076] In another embodiment, user-preset tag distribution rules can also be supported. First, the target thread of the data is inferred based on the pre-aggregation results of the previous preset time period (five minutes). If there is no corresponding pre-aggregation result, the data is then distributed according to the preset tag distribution rules. If there are no preset tag distribution rules, the data is finally distributed according to the records in the circular array corresponding to each thread.

[0077] Further reading Figure 4 This is another detailed flowchart of step S202 described above. It is understood that this flowchart is not intended to limit the order of execution steps. Steps in this flowchart can be added to or removed as needed. In this embodiment, step S202 specifically includes:

[0078] S2020: Obtain the pre-aggregation results of the previous preset time period.

[0079] S2022, when the pre-aggregation result contains the label of the data, select one from the thread distribution interval corresponding to the label in the pre-aggregation result as the target thread of the data.

[0080] S2023, when the pre-aggregation result does not contain the label of the data, the target thread of the data is determined according to the user-preset label distribution rules.

[0081] Specifically, the preset tag distribution rule may include the thread distribution interval corresponding to each tag. Therefore, according to the preset tag distribution rule, a thread can be randomly selected from the thread distribution interval corresponding to the tag as the target thread of the data.

[0082] S2025, when the preset tag distribution rule does not exist, the corresponding target thread is assigned to the data according to the distributed data information recorded in the circular array.

[0083] Back Figure 2 S204, the data is distributed to the target thread.

[0084] Once the target thread corresponding to the data is determined according to the above steps, the data can be distributed to the target thread for processing.

[0085] The data distribution method proposed in this embodiment can evenly allocate corresponding downstream threads to data traffic based on the pre-aggregation results or records in the circular array, avoiding unreasonable partitioning and concurrency allocation of data traffic, achieving balanced processing of burst traffic, reducing the number of small files generated, and reducing file read / write pressure and downstream resource consumption.

[0086] Example 2

[0087] likeFigure 5 The diagram shown is a flowchart of a data distribution method according to a second embodiment of this application. In the second embodiment, the data distribution method further includes step S306, based on the first embodiment described above. It is understood that the flowchart in this method embodiment is not intended to limit the order of execution steps. Depending on the needs, some steps in the flowchart can be added to or removed.

[0088] The method includes the following steps:

[0089] S300: Receive row record data from the data source and determine the ownership tag of the data.

[0090] The label is used to mark the group to which the data belongs. The group can be user-defined, for example, which type of business the data belongs to.

[0091] In this embodiment, users can define their own rules for calculating the attribution tags of the data. Once the attribution tags of the data are determined according to the user-defined calculation rules, the data can be distributed based on the tags.

[0092] In a preferred embodiment, it is also necessary to calculate the size of the data, that is, to calculate the number of bytes required.

[0093] S302, determine the target thread corresponding to the data based on the pre-aggregation result or the records in the circular array.

[0094] To balance data traffic and reduce the number of small files generated, the data needs to be distributed to different downstream threads according to certain rules. The thread to which the data is distributed is the corresponding target thread. In one embodiment, the target thread of the data can first be inferred based on the pre-aggregation results of the previous preset time period (e.g., five minutes). If there are no corresponding pre-aggregation results, the data can then be distributed according to the records in the circular array corresponding to each thread.

[0095] In another embodiment, user-preset tag distribution rules can also be supported. First, the target thread of the data is inferred based on the pre-aggregation results of the previous preset time period (five minutes). If there is no corresponding pre-aggregation result, the data is then distributed according to the preset tag distribution rules. If there are no preset tag distribution rules, the data is finally distributed according to the records in the circular array corresponding to each thread.

[0096] S304, the data is distributed to the target thread.

[0097] Once the target thread corresponding to the data is determined according to the above steps, the data can be distributed to the target thread for processing.

[0098] S306, at the end of the current preset time period, perform aggregation calculations based on the distributed data information recorded in the circular array reported by each thread, and provide pre-aggregation results for the next preset time period.

[0099] In this embodiment, each thread maintains a record of the data distribution corresponding to all tags, recorded in a circular array. This array is rolled every preset time interval (e.g., five minutes), and the distributed data information recorded in the circular array is reported to the JobManager for aggregation calculation. In other words, at the end of the current preset time interval, the distributed data information recorded in each circular array is reported to the JobManager.

[0100] JobManager summarizes the distributed data information of all threads in the current preset time period, divides the concurrency according to the proportion of all tags, that is, based on the distributed data information of the current preset time period (five minutes), it infers the thread distribution range corresponding to each tag in the next preset time period (the next five minutes) and obtains the pre-aggregation result.

[0101] Specifically, the pre-polymerization result can be obtained in the following three ways.

[0102] (1) Using the signature-based knapsack algorithm:

[0103] First, all distributed tags and their distributed data size for the current preset time period are aggregated. Then, the allocated size for each tag is summed up to a preset threshold (8GB) for each thread, thus completing the knapsack problem. Next, the required concurrency (number of threads) for each tag is allocated proportionally based on the distributed data sizes of all tags. In other words, for each tag, the next thread is allocated only after the preset threshold for a thread is reached, determining how many threads need to be allocated to that tag. These allocated threads constitute the thread distribution range corresponding to the tag.

[0104] The advantage of this approach is that it can greatly reduce the number of small files generated during off-peak data periods. However, the disadvantage is that adjusting the ratio will increase the number of files, making it susceptible to back pressure and reducing processing capacity.

[0105] (2) Allocation based on threshold:

[0106] Based on the aforementioned signature knapsack algorithm, all tags are uniformly sorted. Then, according to the sorting, the proportion of distributed data size for all tags, and the preset threshold for each thread, a corresponding thread distribution interval is assigned to each tag. The sorting can be based on the unique code of each tag, the purpose of which is to ensure that each tag is assigned to a relatively stable thread distribution interval each time. That is to say, in addition to assigning the next thread to the tag after the distributed data size reaches the preset threshold for each thread to determine how many threads need to be allocated to the tag, this method also allocates threads to each tag according to the sorting. Therefore, each tag is assigned to a similar thread distribution interval each time, maintaining stability.

[0107] The advantage of this approach is that it can greatly reduce the number of small files generated during off-peak data periods, and the impact on files is not significant when the ratio is adjusted. However, the disadvantage is that it is easily affected by back pressure, which reduces processing capacity.

[0108] (3) Divide according to proportion:

[0109] First, all distributed tags and the amount of distributed data for each tag are aggregated in the current preset time period. Then, concurrency (number of threads) is allocated according to the ratio of the distributed data sizes among the tags to obtain the thread distribution range corresponding to each tag. If the calculated concurrency of a tag is 0, it is forcibly allocated 1 to ensure data distribution for that tag.

[0110] In other words, the first two methods prioritize ensuring that each assigned thread reaches a threshold (each thread is fully loaded), meaning the number of assigned threads may be relatively small. This method, however, does not consider the signature knapsack algorithm or the preset threshold for each thread. Instead of prioritizing ensuring each assigned thread reaches its threshold (not fully loaded), it allocates all tags to all threads according to the proportion among all tags (each thread is assigned a portion of the task).

[0111] The advantage of this method is that it is not affected by back pressure and has a strong ability to handle wave crests. However, the disadvantage is that the proportional distribution during the stacking recovery process can easily generate noise, resulting in a reduction in the stacking capacity.

[0112] All three methods can uniformly aggregate the distributed data information of all threads and all tags in the current preset time period, and then infer the thread distribution range corresponding to each tag in the next preset time period, that is, obtain the pre-aggregation result (Tag Allocated List) of each tag in the next preset time period. After generating the pre-aggregation result according to one of the above three methods, it can be distributed to each thread, providing the basis for determining the target thread corresponding to the data based on the pre-aggregation result in step S302 of the next preset time period.

[0113] like Figure 6 The diagram shown is a flowchart of another form of the data distribution method proposed in this embodiment. Figure 6 In this context, the Shuffle operator's processing flow for row record data sent from the data source mainly includes two parts: a synchronous process and an asynchronous process.

[0114] The synchronization process includes:

[0115] (1) Determine the home tag of the received Rowdata.

[0116] (2) Calculate the size of the Rowdata.

[0117] (3) Determine the target thread corresponding to the Rowdata, including:

[0118] (3-1) When the pre-aggregation result of the previous preset period contains the label of the Rowdata, select one from the thread distribution interval corresponding to the label in the pre-aggregation result as the target thread of the Rowdata.

[0119] (3-2) When the pre-aggregation result does not contain the tag of the Rowdata, the target thread of the Rowdata is determined according to the user-preset tag distribution rule.

[0120] (3-3) When the preset tag distribution rule does not exist, the corresponding target thread is assigned to the Rowdata according to the distributed data information recorded in the circular array.

[0121] First, it is determined whether the tag has a historical distribution record, that is, whether the tag's data has ever been distributed to a particular thread. If a historical distribution record exists, the distributed data size recorded in the circular array of the thread corresponding to the historical distribution record is used to determine whether the distributed data in that thread has reached a preset threshold (e.g., 8GB). If the threshold has not been reached, it means that the thread can process more data, and this thread is designated as the target thread for the Rowdata. If the threshold has been reached, it means that the thread can no longer process more data, and a new thread is requested as the target thread for the Rowdata. If the tag has no historical distribution record, that is, if the tag is appearing for the first time, a new thread is also requested as the target thread for the Rowdata.

[0122] The asynchronous process includes:

[0123] (1) Each thread maintains a record of the data distribution corresponding to all tags, which is recorded in a circular array. It is rolled once every preset time period (e.g., five minutes) and the distributed data information recorded in the circular array is reported to the JobManager for aggregation calculation.

[0124] (2) The JobManager summarizes the distributed data information of all threads in the current preset time period, and divides the concurrency according to the proportion of all tags. That is, it infers the thread distribution range corresponding to each tag in the next preset time period (last five minutes) based on the distributed data information of the previous preset time period (first five minutes), and obtains the pre-aggregation result. There are three ways to implement this, which will not be elaborated here.

[0125] The data distribution method proposed in this embodiment can perform aggregation calculations based on the distributed data information recorded in the circular array reported by each thread at the end of each preset time period, providing pre-aggregation results for the next preset time period. Furthermore, based on the pre-aggregation results or the records in the circular array, downstream threads are evenly allocated to the data traffic, avoiding unreasonable partitioning and concurrency distribution of data traffic, achieving balanced handling of burst traffic, reducing the number of small files generated, and decreasing file read / write pressure and downstream resource consumption.

[0126] Example 3

[0127] like Figure 7 The diagram shown illustrates the hardware architecture of an electronic device 20 according to a third embodiment of this application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23, which are interconnected via a system bus. It should be noted that... Figure 7 Only the electronic device 20 with components 21-23 is shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented alternatively. In this embodiment, the electronic device 20 is an electronic device containing the Shuffle operator, such as a server device. The server may be a rack server, blade server, tower server, or cabinet server, and may be a standalone server or a server cluster consisting of multiple servers.

[0128] The memory 21 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the electronic device 20, such as the hard disk or memory of the electronic device 20. In other embodiments, the memory 21 may also be an external storage device of the electronic device 20, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 20. Of course, the memory 21 may include both the internal storage unit and the external storage device of the electronic device 20. In this embodiment, the memory 21 is typically used to store the operating system and various application software installed on the electronic device 20, such as the program code of the data distribution system 60. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or will be output.

[0129] In some embodiments, the processor 22 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 22 is typically used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is used to run program code stored in the memory 21 or process data, for example, to run the data distribution system 60.

[0130] The network interface 23 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the electronic device 20 and other electronic devices.

[0131] Example 4

[0132] like Figure 8 The diagram shown is a modular schematic of a data distribution system 60 according to a fourth embodiment of this application. The data distribution system 60 can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program module referred to in the embodiments of this application refers to a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0133] In this embodiment, the data distribution system 60 includes:

[0134] The receiving module 600 is used to receive row record data sent by the data source and determine the ownership tag of the data.

[0135] The label is used to mark the group to which the data belongs. The group can be user-defined, for example, which type of business the data belongs to.

[0136] In this embodiment, users can define their own rules for calculating the attribution tags of the data. Once the attribution tags of the data are determined according to the user-defined calculation rules, the data can be distributed based on the tags.

[0137] In a preferred embodiment, it is also necessary to calculate the size of the data, that is, to calculate the number of bytes required.

[0138] The allocation module 602 is used to determine the target thread corresponding to the data based on the pre-aggregation result or the records in the circular array.

[0139] To balance data traffic and reduce the number of small files generated, the data needs to be distributed to different downstream threads according to certain rules. The thread to which the data is distributed is the corresponding target thread. In one embodiment, the target thread of the data can first be inferred based on the pre-aggregation results of the previous preset time period (e.g., five minutes). If there are no corresponding pre-aggregation results, the data can then be distributed according to the records in the circular array corresponding to each thread.

[0140] In another embodiment, user-preset tag distribution rules can also be supported. First, the target thread of the data is inferred based on the pre-aggregation results of the previous preset time period (five minutes). If there is no corresponding pre-aggregation result, the data is then distributed according to the preset tag distribution rules. If there are no preset tag distribution rules, the data is finally distributed according to the records in the circular array corresponding to each thread.

[0141] The specific process by which the allocation module 602 allocates the data to the target thread is described in the first embodiment above, and will not be repeated here.

[0142] The distribution module 604 is used to distribute the data to the target thread.

[0143] Once the allocation module 602 determines the target thread corresponding to the data, the distribution module 604 can distribute the data to the target thread for processing.

[0144] The data distribution system proposed in this embodiment can evenly allocate corresponding downstream threads to data traffic based on pre-aggregation results or records in a circular array, avoiding unreasonable partitioning and concurrency allocation of data traffic, achieving balanced processing of burst traffic, reducing the number of small files generated, and reducing file read / write pressure and downstream resource consumption.

[0145] Example 5

[0146] like Figure 9 The diagram shown is a schematic representation of a data distribution system 60 according to the fifth embodiment of this application. In this embodiment, the data distribution system 60 includes, in addition to the receiving module 600, the allocation module 602, and the distribution module 604 described in the fourth embodiment, an aggregation module 606.

[0147] The aggregation module 606 is used to perform aggregation calculations based on the distributed data information recorded in the circular array reported by each thread at the end of the current preset time period, so as to provide pre-aggregation results for the next preset time period.

[0148] In this embodiment, each thread maintains a record of the data distribution corresponding to all tags, recorded in a circular array. This array is rolled every preset time interval (e.g., five minutes), and the distributed data information recorded in the circular array is reported to the JobManager (aggregation module 606) for aggregation calculation. That is, at the end of the current preset time interval, the distributed data information recorded in each circular array is reported to the JobManager.

[0149] The JobManager aggregates the distributed data information of all threads for the current preset time period and divides the concurrency according to the proportion of all tags. That is, it infers the thread distribution range corresponding to each tag for the next preset time period (the next five minutes) based on the distributed data information of the current preset time period (five minutes), thus obtaining the pre-aggregation result. Specifically, the pre-aggregation result can be obtained in three ways. For a detailed description of each method, please refer to the second embodiment above, which will not be repeated here.

[0150] The data distribution system proposed in this embodiment can perform aggregation calculations based on the distributed data information recorded in the circular array reported by each thread at the end of each preset time period, providing pre-aggregation results for the next preset time period. Furthermore, based on the pre-aggregation results or the records in the circular array, it evenly distributes the corresponding downstream threads to the data traffic, avoiding unreasonable partitioning and concurrency allocation of data traffic, achieving balanced handling of burst traffic, reducing the number of small files generated, and decreasing file read / write pressure and downstream resource consumption.

[0151] Example 6

[0152] This application also provides another embodiment, namely, providing a computer-readable storage medium storing a data distribution program, which can be executed by at least one processor to cause the at least one processor to perform the steps of the data distribution method as described above.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0155] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0156] The above are merely preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structural or procedural transformations made using the description and drawings of the present application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present application.

Claims

1. A data distribution method, characterized in that, The method includes: Receive row record data from the data source and determine the attribution tag of the data; Based on the pre-aggregation results of each tag in the previous preset time period or the distributed data information recorded in the circular array corresponding to each thread, determine the target thread corresponding to the data; Distribute the data to the target thread; The step of determining the target thread corresponding to the data based on the pre-aggregation results of each tag in the previous preset time period or the distributed data information recorded in the circular array corresponding to each thread includes: Obtain the pre-aggregation results of the previous preset time period, wherein the pre-aggregation results include the thread distribution interval corresponding to each tag; If the pre-aggregation result contains the label of the data, select one from the thread distribution interval corresponding to the label in the pre-aggregation result as the target thread of the data; If the label of the data is not included in the pre-aggregation result, a corresponding target thread is assigned to the data based on the distributed data information recorded in the circular array.

2. The data distribution method according to claim 1, characterized in that, The method further includes: At the end of the current preset time period, the distributed data information recorded in the circular array reported by each thread is uniformly aggregated to provide pre-aggregation results for each tag for the next preset time period.

3. The data distribution method according to claim 1 or 2, characterized in that, The method further includes, after determining the attribution label of the data: Calculate the size of the data.

4. The data distribution method according to claim 1, characterized in that, The pre-aggregation result includes a thread distribution interval allocated to each tag based on the distributed data information of all threads and all tags in the previous preset time period; each thread corresponds to a circular array, and the circular array records the distributed data information of the thread within the preset time period with the preset time period as the window.

5. The data distribution method according to claim 1, characterized in that, The step of allocating a corresponding target thread to the data based on the distributed data information recorded in the circular array includes: If the tag has historical distribution records and the thread corresponding to the historical distribution record has not reached a preset threshold, the thread will be used as the target thread corresponding to the data. If the tag has a historical distribution record and the thread corresponding to the historical distribution record has reached a preset threshold, or if the tag has no historical distribution record, a new thread is requested as the target thread corresponding to the data.

6. The data distribution method according to claim 1, characterized in that, In the case where the label of the data is not included in the pre-aggregation result, the method further includes: The target thread for the data is determined based on the user-preset tag distribution rules.

7. The data distribution method according to claim 2, characterized in that, The step of uniformly aggregating the distributed data information recorded in the circular array reported by each thread to provide pre-aggregation results for each tag for the next preset time period includes: The algorithm summarizes all distributed tags and the size of distributed data for each tag in the current preset time period. Using the tagging knapsack algorithm, it assigns a corresponding thread distribution interval to each tag based on the proportion of distributed data size for all tags and the preset threshold for each thread.

8. The data distribution method according to claim 7, characterized in that, The step of uniformly aggregating the distributed data information recorded in the circular array reported by each thread to provide pre-aggregation results for each tag for the next preset time period also includes: Based on the aforementioned tagging knapsack algorithm, all tags are uniformly sorted, and each tag is assigned a corresponding thread distribution interval according to the sorting, the proportion of the distributed data size of all tags, and the preset threshold of each thread.

9. The data distribution method according to claim 7 or 8, characterized in that, The step of uniformly aggregating the distributed data information recorded in the circular array reported by each thread to provide pre-aggregation results for each tag for the next preset time period also includes: Summarize all distributed tags and the distributed data size of each tag in the current preset time period. Based on the ratio of the distributed data sizes among the tags, allocate all tags to all threads to obtain the thread distribution range corresponding to each tag.

10. A data distribution system, characterized in that, The system includes: The receiving module is used to receive row record data sent by the data source and determine the ownership tag of the data; The allocation module is used to determine the target thread corresponding to the data based on the pre-aggregation result of each tag in the previous preset time period or the distributed data information recorded in the circular array corresponding to each thread. The distribution module is used to distribute the data to the target thread; The allocation module is also used to obtain the pre-aggregation result of the previous preset time period, the pre-aggregation result including the thread distribution interval corresponding to each tag; when the pre-aggregation result contains the tag of the data, one is selected from the thread distribution interval corresponding to the tag in the pre-aggregation result as the target thread of the data; when the pre-aggregation result does not contain the tag of the data, the corresponding target thread is allocated to the data according to the distributed data information recorded in the circular array.

11. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a data distribution program stored in the memory and executable on the processor, wherein the data distribution program, when executed by the processor, implements the data distribution method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data distribution program, which, when executed by a processor, implements the data distribution method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.

Images