Data processing method and electronic device

Abstract

An embodiment of the present application provides a data processing method and electronic device, wherein the method includes: determining historical task execution information completed by a distributed file system within a preset time period; and determining an adjustment strategy based on the execution information, wherein the adjustment strategy is used to adjust the data partitioning strategy implemented for data to be stored during the current data preprocessing phase, so that after the data to be stored in the next preset time period is partitioned using the adjusted data partitioning strategy, the data volume in each partition meets a preset storage capacity condition. The data processing method of the embodiment of the present application can effectively prevent the distributed file system from generating a large number of small files, thereby avoiding affecting the operating speed and performance of the system.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of smart devices, and in particular to a data processing method and an electronic device. Background Art

[0002] When a large number of small files exist in some distributed storage systems, such as HDFS (Hadoop Distributed File System, referred to as distributed file system in this article), the system's scalability and actual operational performance will be seriously affected. There are many ways for such storage systems to generate a large number of small files. For example, taking HDFS as an example, when a client writes Hive table data in HDFS based on Spark-SQL in Spark (a data computing engine), if the Shuffle Partition (referred to as data partitioning strategy in this article) in Spark is not set properly, it will directly cause HDFS to generate a large number of small files when storing the written data. Therefore, in order to avoid damage to the scalability and actual operational performance of the storage system, it is very necessary to avoid the generation of a large number of small files. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method and an electronic device applying the method that can effectively prevent a distributed file system from generating a large number of small files.

[0004] In order to solve the above technical problems, an embodiment of the present application provides a data processing method, which includes:

[0005] Determine the historical task execution information completed by the distributed file system within a preset time period;

[0006] An adjustment strategy is determined based on the execution information, wherein the adjustment strategy is used to adjust the data partitioning strategy implemented for the data to be stored in the current data preprocessing stage, so that after the data to be stored in the next preset time period is partitioned according to the adjusted data partitioning strategy, the data volume in each zone meets the preset storage volume condition.

[0007] Preferably, determining the adjustment strategy based on the execution information includes:

[0008] determining an adjustment strategy based at least on the execution information; or

[0009] The adjustment strategy is determined based at least on the execution information and current system configuration parameters.

[0010] Preferably, the adjustment strategy includes at least a first adjustment strategy, a second adjustment strategy and a third adjustment strategy;

[0011] The execution information includes at least the execution time of two historical tasks completed successively within the preset time period, and system stability information indicating whether the system is stable when executing the historical tasks;

[0012] The current configuration parameters of the system include at least the maximum single execution time of the historical task, the coefficient of change of the number of partitions, and the maximum number of partitions.

[0013] Preferably, determining the adjustment strategy based at least on the execution information includes:

[0014] If the execution time of the historical task is 0s, it is determined that the first adjustment strategy is adopted to adjust the current data partition strategy.

[0015] Preferably, determining the adjustment strategy based at least on the execution information and current system configuration parameters includes:

[0016] If the execution time of the historical task is greater than 0s, determining whether the system is stable based on the system stability information;

[0017] If it is unstable, determining whether the execution time of the historical task is greater than the single maximum execution time;

[0018] If so, it is determined that the current data partitioning strategy is adjusted by the first adjustment strategy.

[0019] Preferably, determining the adjustment strategy based at least on the execution information and current system configuration parameters includes:

[0020] If the execution time of the most recently executed first historical task is greater than 0s, determining whether the system is stable based on the system stability information;

[0021] If it is unstable, determining whether the execution time of the first historical task is greater than the single maximum execution time;

[0022] If it is greater, determine whether the execution time of the second historical task executed earlier is 0s, or whether the execution time of the first historical task is less than the execution time of the second historical task;

[0023] If so, determining to adjust the current data partitioning strategy by the second adjustment strategy;

[0024] The second adjustment strategy at least includes replacing the current partition value with the product of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

[0025] As an option, it also includes:

[0026] If the execution time of the first historical task is greater than the execution time of the second historical task, determining to adjust the current data partitioning strategy by the third adjustment strategy;

[0027] The third adjustment strategy at least includes replacing the current partition value with the quotient of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

[0028] As an option, it also includes:

[0029] Determining data information capable of representing an amount of data change in the distributed file system within the preset time period;

[0030] Determining a predicted number of partitions based on the data information and a default storage capacity of the storage block;

[0031] The first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, thereby achieving adjustment of the current data partition strategy.

[0032] Preferably, the first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, including:

[0033] Determining whether the predicted number of partitions is greater than the maximum number of partitions;

[0034] If it is greater, the maximum number of partitions will replace the current number of partitions;

[0035] If it is smaller, the predicted number of partitions will replace the current number of partitions.

[0036] The present application also provides an electronic device, including:

[0037] A distributed file system for distributed storage of pre-processed and partitioned data;

[0038] a data computing engine configured to determine at least historical task execution information completed by the distributed file system within a preset time period, and determine an adjustment strategy based at least on the execution information;

[0039] Among them, the adjustment strategy is used to adjust the data partitioning strategy implemented for the data to be stored in the current data preprocessing stage, so that after the data to be stored in at least the next preset time period is partitioned according to the adjusted data partitioning strategy, the data volume in each zone meets the preset storage volume conditions.

[0040] Based on the disclosure of the above embodiments, it can be known that the beneficial effect of the embodiments of the present application is to determine the adjustment strategy for adjusting the data partitioning strategy through the historical task execution information of the distributed file system, so that the data partitioning strategy adjusted based on the adjustment strategy can at least make the data volume of each data area within the next preset time period meet the preset storage volume conditions, avoiding the situation where the data volume in some data areas is small, resulting in the formation of small files after storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the data processing method in an embodiment of the present application.

[0042] Figure 2 This is a flowchart of the data processing method in another embodiment of the present application (P_num in the figure is the number of partitions of the system when the first historical task is executed, F_stable is the system stability information, F_time1 is the execution time of the first historical task, F_time2 is the execution time of the second historical task, T_MAX is the maximum execution time allowed by the system, NTimes is the coefficient of change of the number of partitions, N is the theoretical number of partitions, N_tmp is the current number of partitions / temporary number of partitions, and P_MAX is the maximum number of partitions).

[0043] Figure 3 This is a structural block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0044] Below, specific embodiments of the present application are described in detail with reference to the accompanying drawings, but are not intended to limit the present application.

[0045] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

[0046] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0047] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0048] It should also be understood that although the present application has been described with reference to certain specific examples, those skilled in the art will be able to implement many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0049] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0050] Specific embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repetitive functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant detail. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but rather serve merely as a basis and representative basis for the claims to teach those skilled in the art to variously employ the present disclosure with substantially any suitable detailed structure.

[0051] This description may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," each of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0052] Below, the embodiments of the present application are described in detail with reference to the accompanying drawings.

[0053] like Figure 1 As shown, an embodiment of the present application provides a data processing method, which includes:

[0054] Determine the historical task execution information completed by the distributed file system within a preset time period;

[0055] An adjustment strategy is determined based on the execution information, wherein the adjustment strategy is used to adjust the data partitioning strategy implemented for the data to be stored in the current data preprocessing stage, so that after the data to be stored in the next preset time period is partitioned according to the adjusted data partitioning strategy, the data volume in each zone meets the preset storage volume conditions.

[0056] The beneficial effect of the embodiment of the present application is that an adjustment strategy for adjusting the data partitioning strategy is determined through the historical task execution information of the distributed file system, so that the data partitioning strategy adjusted based on the adjustment strategy can at least ensure that the data volume of each data area in the next preset time period can meet the preset storage volume conditions, thereby avoiding the situation where the data volume in some data areas is small, resulting in small files being easily formed after storage. That is, it achieves the technical effect of avoiding the distributed file system from generating a large number of small files when storing and processing data, thereby affecting its work efficiency and scalability.

[0057] For example, a distributed file system periodically obtains its historical task completion information in the past week, and then determines whether the current data partitioning strategy needs to be adjusted based on the historical task completion information, determines the corresponding adjustment strategy, and applies the adjustment strategy to the current data partitioning strategy. When the system processes the data tasks of the next week starting from the next second, it can partition the data based on the adjusted data partitioning strategy to ensure that the amount of data after partitioning meets the preset storage capacity conditions, thereby effectively ensuring that the system will not generate a large number of small files during the storage process due to insufficient data in the data area.

[0058] The aforementioned preset storage capacity conditions are not unique. In this embodiment, it is preferred that the data capacity of each data area must be no less than the default storage capacity of each storage block used to store data in the distributed file system. That is, the data capacity of each data area partitioned by the adjusted data partitioning strategy in this embodiment is no less than the default storage capacity of each storage block in the distributed file system. The specific value of this default storage capacity is not unique and can be determined based on actual circumstances. Through the settings in this embodiment, each storage block can be fully stored with data, eliminating the problem of insufficient storage capacity causing a large number of small files to be generated, which could affect system processing speed.

[0059] Furthermore, in this embodiment, determining the adjustment strategy based on the execution information includes:

[0060] determining an adjustment strategy based at least on the execution information; or

[0061] An adjustment strategy is determined based at least on the execution information and current configuration parameters of the system.

[0062] That is, the system may directly determine the adjustment strategy based only on the execution information of historical tasks, or may jointly determine the adjustment strategy based on the execution information of historical tasks and the configuration parameters of the current system.

[0063] The adjustment strategies in this embodiment include at least a first adjustment strategy, a second adjustment strategy, and a third adjustment strategy. The execution information includes at least the execution times of two historical tasks that were completed successively within a preset time period, as well as system stability information indicating whether the system was stable when executing the historical tasks. The current system configuration parameters include at least the maximum single execution time of the historical tasks, the coefficient of variation of the number of partitions, and the maximum number of partitions. Of course, the above information is not exclusive, and other parameters may be added, such as information related to three or four historical tasks that were completed successively within a preset time period, etc.

[0064] Specifically, if Figure 2 As shown, in this embodiment, determining the adjustment strategy based on the execution information includes:

[0065] If the execution time of the historical task is 0s, it is determined that the first adjustment strategy is adopted to adjust the current data partition strategy.

[0066] For example, if the system retrieves and checks the execution information of a historical task and finds that the execution time of the historical task is 0s, then it can be determined that the system has not actually executed the task before. Therefore, it can be determined that the current data partitioning strategy should be adjusted according to the first adjustment strategy. The current partitioning strategy can be the default partitioning strategy or no data partitioning strategy, and the execution time of the historical task can be the execution time of the most recently executed historical task, or the execution time of two historical tasks can be viewed simultaneously.

[0067] Furthermore, when the system determines the adjustment strategy based on the execution information and the current configuration parameters of the system, the system includes:

[0068] If the execution time of the historical task is greater than 0s, the system stability is determined based on the system stability information;

[0069] If it is unstable, determine whether the execution time of the historical task is greater than the maximum single execution time;

[0070] If not, it is determined that the current data partitioning strategy is adjusted by the first adjustment strategy.

[0071] For example, after the system retrieves the execution information of a historical task, it first checks the execution time of the historical task. After determining that the execution time of the historical task is greater than 0s, it checks the system stability information. If the information indicates that the system is in an unstable state when executing the historical task, then it can be determined whether the execution time of the historical task exceeds the parameter value of the single maximum execution time in the system configuration parameters. If not, it is determined that the current data partitioning strategy is adjusted by the first adjustment strategy.

[0072] The aforementioned historical task execution time and system stability information can be the execution time of the most recently executed historical task, or can be the execution time and system stability information of two previous historical tasks. In the above process of determining whether the system is in a stable state when executing historical tasks, if the judgment result is yes, then the system can determine that the current number of data partitions meets the requirements and the data volume of each data partition meets the preset storage conditions, and no adjustment is required.

[0073] Further, continue to combine Figure 2 As shown, the method in this embodiment also includes:

[0074] Determining data information that can represent the amount of data changes in the distributed file system within a preset time period;

[0075] Determining the predicted number of partitions based on data information and a default storage capacity of a storage block;

[0076] The first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, thereby achieving adjustment of the current data partition strategy.

[0077] For example, the data information that can characterize the amount of data change in the distributed file system within a preset time period can be obtained by the data increment of the Hive table in the distributed file system within the preset time period. That is, the amount of data change that can characterize the distributed file system within the preset time period is determined by calculating the data increment of the Hive table within the preset time period. After the system obtains the data information, it can determine the predicted number of partitions by calculating it with the default storage capacity of the storage block. In this embodiment, the final predicted number of partitions is obtained by the formula: N (predicted number of partitions) = 1 + (data increment of the Hive table / preset time period / default storage capacity of the storage block). The first adjustment strategy in this embodiment is to replace the current number of data partitions with the predicted number of partitions, thereby realizing the adjustment of the data partition strategy.

[0078] Furthermore, in this embodiment, when adjustment is performed based on the first adjustment strategy, that is, the first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, including:

[0079] Determine whether the predicted number of partitions is greater than the maximum number of partitions;

[0080] If it is greater, the maximum number of partitions will replace the current number of partitions;

[0081] If it is smaller, the predicted number of partitions will replace the current number of partitions.

[0082] That is, to ensure that the calculated predicted number of partitions is feasible, it needs to be tested to determine whether it is greater than the system parameter - the maximum number of partitions. If it is greater, the system will replace the current number of partitions with the maximum number of partitions, or directly use it as the current number of partitions (applicable to the case where there are currently no partitions). If it is less than, it can be executed as planned, that is, the calculated predicted number of partitions will replace the current number of partitions, or directly use it as the current number of partitions (applicable to the case where there are currently no partitions). Moreover, after the setting is completed, the system stability parameter can be directly changed to stable.

[0083] Further, continue to combine Figure 2 In this embodiment, when determining the adjustment strategy based on the execution information and the current configuration parameters of the system, the following steps are also included:

[0084] If the execution time of the most recently executed first historical task is greater than 0s, determine whether the system is stable based on the system stability information;

[0085] If it is unstable, determine whether the execution time of the first historical task is greater than the single maximum execution time;

[0086] If it is greater, determine whether the execution time of the second historical task executed earlier is 0s, or whether the execution time of the first historical task is less than the execution time of the second historical task;

[0087] If so, determining to adjust the current data partitioning strategy by the second adjustment strategy;

[0088] The second adjustment strategy at least includes replacing the current partition value with the product of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

[0089] For example, of the two historical tasks, the historical task closest to the current one is the first historical task, and the other historical task is the second historical task. The system determines by retrieving the historical task execution information that if the execution time of the first historical task is greater than 0s, and the system is in an unstable state when executing the first historical task, and the execution time of the first historical task also exceeds the single maximum execution time allowed by the system, then it determines whether the execution time of the second historical task is 0s, or determines whether the execution time of the first historical task is less than the execution time of the second historical task. If the judgment result is yes, it indicates that the current number of partitions is too many and the number of partitions should be reduced. At this time, the system determines to adjust the current data partition strategy by the second adjustment strategy. The second adjustment strategy in this embodiment is to replace the current partition value with the product of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment. Among them, the partition change coefficient in this embodiment is a value less than 1, and its specific value is uncertain, and can be calculated based on the corresponding data of the actual situation.

[0090] Furthermore, in combination with the above embodiment, the method further includes:

[0091] If the execution time of the first historical task is greater than the execution time of the second historical task, determining to adjust the current data partitioning strategy by the third adjustment strategy;

[0092] The third adjustment strategy at least includes replacing the current partition value with the quotient of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

[0093] For example, also taking the above embodiment as an example, when the system determines the relationship between the execution time of the first historical task and the execution time of the second historical task, the result obtained is that the execution time of the first historical task is greater than the execution time of the second historical task, then it can be determined that the current number of partitions is too small and the number of partitions needs to be increased, so it is determined to adopt the third adjustment strategy to adjust the current number of partitions. In this embodiment, the third adjustment strategy includes dividing the current data partition value by the partition change coefficient, and using the calculated result as the current partition value, thereby adjusting the number of partitions. Moreover, after the system adjusts the partition strategy based on the second adjustment strategy or the third adjustment strategy, it will change the system stability parameter to stable, and then the system will return to normal and continue to receive and process data.

[0094] like Figure 3 As shown, an embodiment of the present application also provides an electronic device, which includes:

[0095] A distributed file system for distributed storage of pre-processed and partitioned data;

[0096] a data computing engine configured to determine at least historical task execution information completed by the distributed file system within a preset time period, and determine an adjustment strategy based at least on the execution information;

[0097] Among them, the adjustment strategy is used to adjust the data partitioning strategy implemented for the data to be stored in the current data preprocessing stage, so that after the data to be stored in at least the next preset time period is partitioned by the adjusted data partitioning strategy, the data volume in each zone meets the preset storage volume conditions.

[0098] The beneficial effect of the embodiment of the present application is that the data computing engine determines the adjustment strategy for adjusting the data partitioning strategy by obtaining the historical task execution information of the distributed file system, so that the data partitioning strategy adjusted based on the adjustment strategy can at least make the data volume of each data area meet the preset storage volume conditions within the next preset time period, avoiding the situation where the data volume in some data areas is small, resulting in small files being easily formed after storage. That is, it achieves the technical effect of avoiding the distributed file system from generating a large number of small files when storing and processing data, affecting its work efficiency and scalability.

[0099] For example, the data computing engine is Spark-SQL, which regularly obtains the historical task completion information of the distributed file system (HDFS) in the past week, and then determines whether the current data partitioning strategy needs to be adjusted based on the historical task completion information, and determines the corresponding adjustment strategy, and applies the adjustment strategy to the current data partitioning strategy. Specifically, after the number of partitions is determined, it can be handed over to the Spark executor for execution, so that when the system processes the data tasks of the next week starting from the next second, it can partition the data based on the adjusted data partitioning strategy to ensure that the data volume after partitioning meets the preset storage volume conditions, thereby effectively ensuring that the system will not generate a large number of small files due to insufficient data volume in the data area during the storage process.

[0100] The aforementioned preset storage capacity conditions are not unique. In this embodiment, it is preferred that the data capacity of each data area must be no less than the default storage capacity of each storage block used to store data in the distributed file system. That is, the data capacity of each data area partitioned by the adjusted data partitioning strategy in this embodiment is no less than the default storage capacity of each storage block in the distributed file system. The specific value of this default storage capacity is not unique and can be determined based on actual circumstances. Through the settings in this embodiment, each storage block can be fully stored with data, eliminating the problem of insufficient storage capacity causing a large number of small files to be generated, which could affect system processing speed.

[0101] Furthermore, in this embodiment, determining the adjustment strategy based on the execution information includes:

[0102] determining an adjustment strategy based at least on the execution information; or

[0103] An adjustment strategy is determined based at least on the execution information and current configuration parameters of the system.

[0104] That is, the system may directly determine the adjustment strategy based only on the execution information of historical tasks, or may jointly determine the adjustment strategy based on the execution information of historical tasks and the configuration parameters of the current system.

[0105] The adjustment strategies in this embodiment include at least a first adjustment strategy, a second adjustment strategy, and a third adjustment strategy. The execution information includes at least the execution times of two historical tasks that were completed successively within a preset time period, as well as system stability information indicating whether the system was stable when executing the historical tasks. The current system configuration parameters include at least the maximum single execution time of the historical tasks, the coefficient of variation of the number of partitions, and the maximum number of partitions. Of course, the above information is not exclusive, and other parameters may be added, such as information related to three or four historical tasks that were completed successively within a preset time period, etc.

[0106] Specifically, in this embodiment, determining the adjustment strategy based on the execution information includes:

[0107] If the execution time of the historical task is 0s, it is determined that the first adjustment strategy is adopted to adjust the current data partition strategy.

[0108] For example, if the system retrieves and checks the execution information of a historical task and finds that the execution time of the historical task is 0s, then it can be determined that the system has not actually executed the task before. Therefore, it can be determined that the current data partitioning strategy should be adjusted according to the first adjustment strategy. The current partitioning strategy can be the default partitioning strategy or no data partitioning strategy, and the execution time of the historical task can be the execution time of the most recently executed historical task, or the execution time of two historical tasks can be viewed simultaneously.

[0109] Furthermore, when the system determines the adjustment strategy based on the execution information and the current configuration parameters of the system, the system includes:

[0110] If the execution time of the historical task is greater than 0s, the system stability is determined based on the system stability information;

[0111] If it is unstable, determine whether the execution time of the historical task is greater than the maximum single execution time;

[0112] If not, it is determined that the current data partitioning strategy is adjusted by the first adjustment strategy.

[0113] For example, after the system retrieves the execution information of a historical task, it first checks the execution time of the historical task. After determining that the execution time of the historical task is greater than 0s, it checks the system stability information. If the information indicates that the system is in an unstable state when executing the historical task, then it can be determined whether the execution time of the historical task exceeds the parameter value of the single maximum execution time in the system configuration parameters. If not, it is determined that the current data partitioning strategy is adjusted by the first adjustment strategy.

[0114] The aforementioned historical task execution time and system stability information can be the execution time of the most recently executed historical task, or can be the execution time and system stability information of two previous historical tasks. In the above process of determining whether the system is in a stable state when executing historical tasks, if the judgment result is yes, then the system can determine that the current number of data partitions meets the requirements and the data volume of each data partition meets the preset storage conditions, and no adjustment is required.

[0115] Furthermore, the data calculation engine in this embodiment is also used to:

[0116] Determining data information that can represent the amount of data changes in the distributed file system within a preset time period;

[0117] Determining the predicted number of partitions based on data information and a default storage capacity of a storage block;

[0118] The first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, thereby achieving adjustment of the current data partition strategy.

[0119] For example, the data information that can characterize the amount of data change in the distributed file system within a preset time period can be obtained by the data increment of the Hive table in the distributed file system within the preset time period. That is, the amount of data change that can characterize the distributed file system within the preset time period is determined by calculating the data increment of the Hive table within the preset time period. After the system obtains the data information, it can determine the predicted number of partitions by calculating it with the default storage capacity of the storage block. In this embodiment, the final predicted number of partitions is obtained by the formula: N (predicted number of partitions) = 1 + (data increment of the Hive table / preset time period / default storage capacity of the storage block). The first adjustment strategy in this embodiment is to replace the current number of data partitions with the predicted number of partitions, thereby realizing the adjustment of the data partition strategy.

[0120] Furthermore, in this embodiment, when adjustment is performed based on the first adjustment strategy, that is, the first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, including:

[0121] Determine whether the predicted number of partitions is greater than the maximum number of partitions;

[0122] If it is greater, the maximum number of partitions will replace the current number of partitions;

[0123] If it is smaller, the predicted number of partitions will replace the current number of partitions.

[0124] That is, to ensure that the calculated predicted number of partitions is feasible, it needs to be tested to determine whether it is greater than the system parameter - the maximum number of partitions. If it is greater, the system will replace the current number of partitions with the maximum number of partitions, or directly use it as the current number of partitions (applicable to the case where there are currently no partitions). If it is less than, it can be executed as planned, that is, the calculated predicted number of partitions will replace the current number of partitions, or directly use it as the current number of partitions (applicable to the case where there are currently no partitions). Moreover, after the setting is completed, the system stability parameter can be directly changed to stable.

[0125] Furthermore, in this embodiment, when determining the adjustment strategy based on the execution information and the current configuration parameters of the system, the following steps are further included:

[0126] If the execution time of the most recently executed first historical task is greater than 0s, determine whether the system is stable based on the system stability information;

[0127] If it is unstable, determine whether the execution time of the first historical task is greater than the single maximum execution time;

[0128] If it is greater, determine whether the execution time of the second historical task executed earlier is 0s, or whether the execution time of the first historical task is less than the execution time of the second historical task;

[0129] If so, determining to adjust the current data partitioning strategy by the second adjustment strategy;

[0130] The second adjustment strategy at least includes replacing the current partition value with the product of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

[0131] For example, of the two historical tasks, the historical task closest to the current one is the first historical task, and the other historical task is the second historical task. The system determines by retrieving the historical task execution information that if the execution time of the first historical task is greater than 0s, and the system is in an unstable state when executing the first historical task, and the execution time of the first historical task also exceeds the single maximum execution time allowed by the system, then it determines whether the execution time of the second historical task is 0s, or determines whether the execution time of the first historical task is less than the execution time of the second historical task. If the judgment result is yes, it indicates that the current number of partitions is too many and the number of partitions should be reduced. At this time, the system determines to adjust the current data partition strategy by the second adjustment strategy. The second adjustment strategy in this embodiment is to replace the current partition value with the product of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment. Among them, the partition change coefficient in this embodiment is a value less than 1, and its specific value is uncertain, and can be calculated based on the corresponding data of the actual situation.

[0132] Furthermore, in combination with the above embodiment, the data calculation engine is further used to:

[0133] If the execution time of the first historical task is greater than the execution time of the second historical task, determining to adjust the current data partitioning strategy by the third adjustment strategy;

[0134] The third adjustment strategy at least includes replacing the current partition value with the quotient of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

[0135] For example, also taking the above embodiment as an example, when the system determines the relationship between the execution time of the first historical task and the execution time of the second historical task, the result obtained is that the execution time of the first historical task is greater than the execution time of the second historical task, then it can be determined that the current number of partitions is too small and the number of partitions needs to be increased, so it is determined to adopt the third adjustment strategy to adjust the current number of partitions. In this embodiment, the third adjustment strategy includes dividing the current data partition value by the partition change coefficient, and using the calculated result as the current partition value, thereby adjusting the number of partitions. Moreover, after the system adjusts the partition strategy based on the second adjustment strategy or the third adjustment strategy, it will change the system stability parameter to stable, and then the system will return to normal and continue to receive and process data.

[0136] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A data processing method, wherein: include: Determine historical task execution information completed by the distributed file system within a preset time period; wherein the execution information at least includes the execution time; An adjustment strategy is determined based on the execution information, wherein the adjustment strategy is used to adjust the data partitioning strategy implemented for the data to be stored in the current data preprocessing stage, so that after the data to be stored in the next preset time period is partitioned according to the adjusted data partitioning strategy, the data volume in each zone meets the preset storage volume condition.

2. The method according to claim 1, wherein Determining the adjustment strategy based on the execution information includes: determining an adjustment strategy based at least on the execution information; or The adjustment strategy is determined based at least on the execution information and current system configuration parameters.

3. The method according to claim 2, wherein: The adjustment strategy includes at least a first adjustment strategy, a second adjustment strategy and a third adjustment strategy; The execution information includes at least the execution time of two historical tasks completed successively within the preset time period, and system stability information indicating whether the system is stable when executing the historical tasks; The current configuration parameters of the system include at least the maximum single execution time of the historical task, the coefficient of change of the number of partitions, and the maximum number of partitions.

4. The method according to claim 3, wherein: Determining the adjustment strategy at least based on the execution information includes: If it is determined that the execution time of the historical task is 0s, it is determined that the first adjustment strategy is adopted to adjust the current data partition strategy.

5. The method according to claim 3, wherein Determining the adjustment strategy based at least on the execution information and current system configuration parameters includes: Determining that the execution time of the historical task is greater than 0s, determining whether the system is stable based on the system stability information; If it is unstable, determining whether the execution time of the historical task is greater than the single maximum execution time; If so, it is determined that the current data partitioning strategy is adjusted by the first adjustment strategy.

6. The method according to claim 3, wherein: Determining the adjustment strategy based at least on the execution information and current system configuration parameters includes: If the execution time of the most recently executed first historical task is greater than 0s, determining whether the system is stable based on the system stability information; If it is unstable, determining whether the execution time of the first historical task is greater than the single maximum execution time; If it is greater, determine whether the execution time of the second historical task executed earlier is 0s, or whether the execution time of the first historical task is less than the execution time of the second historical task; If so, determining to adjust the current data partitioning strategy by the second adjustment strategy; The second adjustment strategy at least includes replacing the current partition value with the product of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

7. The method according to claim 6, characterized in that Also includes: If the execution time of the first historical task is greater than the execution time of the second historical task, determining to adjust the current data partitioning strategy by the third adjustment strategy; The third adjustment strategy at least includes replacing the current partition value with the quotient of the partition value in the current data partition strategy and the partition change coefficient, thereby achieving adjustment of the current data partition strategy.

8. The method according to claim 4 or 5, wherein: Also includes: Determining data information capable of representing an amount of data change in the distributed file system within the preset time period; Determining a predicted number of partitions based on the data information and a default storage capacity of the storage block; The first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, thereby achieving adjustment of the current data partition strategy.

9. The method according to claim 8, wherein The first adjustment strategy at least includes replacing the number of partitions in the current data partition strategy with the predicted number of partitions, including: Determining whether the predicted number of partitions is greater than the maximum number of partitions; If it is greater, the maximum number of partitions will replace the current number of partitions; If it is smaller, the predicted number of partitions will replace the current number of partitions.

10. An electronic device, wherein: include: A distributed file system for distributed storage of pre-processed and partitioned data; a data computing engine configured to at least determine historical task execution information completed by the distributed file system within a preset time period, and determine an adjustment strategy based at least on the execution information; wherein the execution information at least includes execution time; Among them, the adjustment strategy is used to adjust the data partitioning strategy implemented for the data to be stored in the current data preprocessing stage, so that after the data to be stored in at least the next preset time period is partitioned according to the adjusted data partitioning strategy, the data volume in each zone meets the preset storage volume conditions.

Images