Data processing dynamic adjustment method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of data request processing, and discloses a data processing dynamic adjustment method, which comprises the following steps of: dividing data to be processed into a plurality of data blocks, and configuring computing resources for each data block; calling the allocated computing resources to run each data block, recording the run data volume and the non-run data volume in each data block, and calculating the execution speed of each data block according to the execution time of running the run data volume by the computing resources; dynamically adjusting the data blocks or the computing resources according to the execution speed and the non-running data volume, namely screening the data blocks of which the non-running data volume is smaller than a first preset threshold value, combining the screened data blocks of which the data volume is smaller than the first preset threshold value in pairs, screening the data blocks of which the execution speed is smaller than a second preset threshold value, and combining the data blocks of which the execution speed is smaller than the second preset threshold value; and adjusting the computing resources of the data blocks smaller than the second preset threshold after screening. According to the invention, through dynamic adjustment of data processing, the requirements of different data volumes and operating environments are met.

Description

Technical Field

[0001] The present invention relates to the field of data request processing, and in particular to a data processing dynamic adjustment method, device, electronic equipment and storage medium. Background Art

[0002] With the advent of the big data era, Apache Spark, as an open source large-scale data processing framework, has become one of the mainstream choices for banking data analysis and processing due to its high performance, ease of use and rich feature set. In the banking and financial field, Apache Spark is widely used in credit card transaction analysis, customer behavior analysis, risk management and other scenarios. However, in Apache Spark, setting the number of data blocks has an important impact on its performance and efficiency.

[0003] The number of data blocks in Apache Spark usually needs to be manually set based on the developer's experience, and this number of data blocks is usually fixed during the job operation. If the number of data blocks is too small, the bank's computing resources will not be fully utilized, resulting in inefficient processing, especially when processing large amounts of transaction data, which may cause severe delays and increased response times. On the contrary, if the number of data blocks is too large, although the degree of parallelism can be improved, it will also bring excessive scheduling overhead and memory consumption, especially during peak processing periods, which may lead to resource waste and system performance degradation.

[0004] In practical applications, this static data block method often cannot meet the needs of different data volumes and operating environments. Therefore, it is an urgent technical problem to dynamically adjust data processing through automation to meet the needs of different data volumes and operating environments. Summary of the invention

[0005] In view of the above, it is necessary to provide a method for dynamic adjustment of data processing, the purpose of which is to dynamically adjust data processing in an automated manner to meet the needs of different data volumes and operating environments.

[0006] The data processing dynamic adjustment method provided by the present invention comprises:

[0007] receiving a request for data partitioning of data to be processed, dividing the data to be processed into a plurality of data blocks, and configuring computing resources for each data block, wherein the computing resources include one of a server node, a number of CPU cores, and a memory capacity;

[0008] Calling the allocated computing resources to run each data block, and recording the amount of data that has been run and the amount of data that has not been run in each data block, and calculating the execution speed of each data block according to the execution time of the computing resources running the run data amount;

[0009] Dynamically adjusting the data blocks or computing resources according to the execution speed and the amount of unrun data includes: screening, from the multiple data blocks, data blocks whose amount of unrun data is less than a first preset threshold, merging the data blocks whose amount of data is less than the first preset threshold after screening in pairs, screening, from the multiple data blocks, data blocks whose execution speed is less than a second preset threshold, and adjusting the computing resources for the data blocks whose amount of data is less than the second preset threshold after screening, wherein the first preset threshold and the second preset threshold have different measurement units.

[0010] Optionally, after dividing the data to be processed into a plurality of data blocks, the method further includes:

[0011] Get the metadata of each data block and calculate the hash value of the metadata of each data block;

[0012] The hash value of the metadata of each data block is used as the identifier of the corresponding data block.

[0013] Optionally, before calling the allocated computing resources to run each data block, the method further includes:

[0014] Open a preset monitoring tool and use the preset monitoring tool to configure an event log for each data block;

[0015] The event log is used to record key indicators of each data block, wherein the key indicators include the amount of data that has been executed, the amount of data that has not been executed, and the execution time.

[0016] Optionally, recording the amount of data that has been run and the amount of data that has not been run in each data block, and running the execution time of the run data amount according to the computing resource, includes:

[0017] respectively setting a first recording task and a second recording task on the event recording log;

[0018] The first recording task is used to record the amount of data that has been executed and the amount of data that has not been executed in each data block, and the second recording task is used to record the execution time required for each data block to execute the executed amount of data.

[0019] Optionally, merging two by two of the data blocks whose values ​​are smaller than the first preset threshold after screening includes:

[0020] sorting the data blocks that are smaller than the first preset threshold after screening in ascending order according to the identifiers of the data blocks;

[0021] Two adjacent data blocks in the sorting are merged, and computing resources of any one of the two adjacent data blocks are released.

[0022] Optionally, merging two by two of the data blocks whose values ​​are smaller than the first preset threshold after screening includes:

[0023] Sorting the filtered data blocks that are smaller than the first preset threshold in ascending order according to the data volume to obtain a sorted merge list;

[0024] From the sorted merge list, the data blocks with the largest data volume and the data blocks with the smallest data volume are selected in turn to be merged in pairs, until all the data blocks in the sorted merge list are merged.

[0025] Optionally, the method further includes:

[0026] The step of dynamically adjusting the data blocks or computing resources according to the execution speed and the amount of unexecuted data is repeatedly performed at every preset time period until the execution task of the data to be processed is completed.

[0027] In order to solve the above problem, the present invention further provides a data processing dynamic adjustment device, the device comprising:

[0028] A receiving module, configured to receive a request for partitioning the data to be processed, divide the data to be processed into a plurality of data blocks, and configure computing resources for each data block, wherein the computing resources include one of a server node, a number of CPU cores, and a memory capacity;

[0029] A recording module, used to call the allocated computing resources to run each data block, and record the amount of data that has been run and the amount of data that has not been run in each data block, and calculate the execution speed of each data block according to the execution time of the computing resources running the run data amount;

[0030] An adjustment module is used to dynamically adjust the data blocks or computing resources according to the execution speed and the amount of unrun data, including: from the multiple data blocks, screening the data blocks whose amount of unrun data is less than a first preset threshold, merging the data blocks whose amount of data is less than the first preset threshold after screening in pairs, screening the data blocks whose execution speed is less than a second preset threshold from the multiple data blocks, and adjusting the computing resources for the data blocks whose amount of data is less than the second preset threshold after screening, wherein the first preset threshold and the second preset threshold have different measurement units.

[0031] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0032] at least one processor; and,

[0033] a memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores a data processing dynamic adjustment program executable by the at least one processor, and the data processing dynamic adjustment program is executed by the at least one processor so that the at least one processor can perform the above-mentioned data processing dynamic adjustment method.

[0035] In order to solve the above problems, the present invention also provides a computer-readable storage medium, on which a data processing dynamic adjustment program is stored. The data processing dynamic adjustment program can be executed by one or more processors to implement the above data processing dynamic adjustment method.

[0036] Compared with the prior art, when the present invention receives a request for data to be processed, it divides the data to be processed into multiple data blocks and allocates corresponding computing resources (such as server nodes, number of CPU cores, memory capacity, etc.) to each data block. Monitor the running status of each data block, record the amount of data that has been run, the amount of data that has not been run, and the execution time of each partition, and dynamically adjust the data blocks according to the execution speed and the amount of data that has not been run. If the amount of data that has not been run in some partitions is less than a pre-set first preset threshold, these partitions are merged to reduce redundancy; conversely, if the execution speed is less than a second preset threshold, additional computing resources are allocated to these partitions.

[0037] The present invention dynamically adjusts data blocks according to the execution speed of each data block and the amount of unrun data, which can more reasonably allocate computing resources, reduce unnecessary resource waste, maximize the advantages of parallel processing, shorten the total time of data processing, and automatically adjust the number of partitions to reduce the need for manual intervention, reduce the complexity of operation and maintenance, and make the system easier to manage and maintain. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a flow chart of a method for dynamic adjustment of data processing provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a module of a data processing dynamic adjustment device provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the structure of an electronic device for implementing a method for dynamic adjustment of data processing provided by an embodiment of the present invention;

[0041] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0044] Reference Figure 1 FIG. 1 is a flow chart of a method for dynamically adjusting data processing provided by an embodiment of the present invention. The method is executed by an electronic device.

[0045] In this embodiment, the data processing dynamic adjustment method includes:

[0046] S1. Receive a request for data partitioning of data to be processed, divide the data to be processed into multiple data blocks, and configure computing resources for each data block, wherein the computing resources include one of a server node, a number of CPU cores, and a memory capacity.

[0047] In this embodiment, a request for data partitioning of data to be processed is received from a user or an application program, requiring the data to be processed to be divided into a plurality of data blocks. The data to be processed is divided into a plurality of smaller parts according to a preset strategy, each of which is a data block. The purpose of partitioning the data to be processed is to be able to process the data in parallel on different computing nodes, thereby improving processing efficiency.

[0048] Data to be processed refers to a set of data waiting to be analyzed, processed, or transformed. In Apache Spark, data to be processed is usually encapsulated into RDD (Resilient Distributed Dataset) or DataFrame / DataSet objects, which support parallel processing in the cluster.

[0049] Data chunking refers to the division of a large data set into multiple smaller, manageable parts (also called partitions or shards) in a distributed computing environment.

[0050] After the data to be processed is divided into multiple data blocks, a certain amount of computing resources is allocated to each data block to ensure that multiple pieces of data to be processed in each data block can be effectively processed.

[0051] After computing resources are allocated to each data block, the tasks for these data blocks begin to be executed. The data of each data block will be loaded onto the computing resources allocated to it, and the corresponding computing tasks will be executed on this basis.

[0052] Computing resources can refer to different types of computing resources, such as server nodes (i.e., servers responsible for executing computing tasks), number of CPU cores, memory capacity, etc. The selection of these resources depends on the specific computing requirements and available resources of the cluster, and the present invention does not limit this in any way.

[0053] Take the financial scenario as an example:

[0054] Suppose a bank wants to analyze its customers' credit card transaction data. The bank's application developer or system administrator sends a data block request to the backend Apache Spark cluster, requesting that the transaction data files (data to be processed) stored on HDFS (Hadoop Distributed File System) be divided into multiple data blocks.

[0055] After Spark receives the data block request, it divides the data to be processed into multiple corresponding data blocks according to the size of the data to be processed, the computing power of the cluster and other factors. For example, if the total data size of the data to be processed is 4GB, it is divided into 20 data blocks, each of which is 200M. Each data block will be assigned to one or more computing nodes in the cluster for processing. In this way, the bank can use multiple servers in the cluster to process different parts of the data at the same time, thereby significantly speeding up data analysis.

[0056] In one embodiment, dividing the data to be processed into a plurality of data blocks includes:

[0057] Acquire the data volume of the data to be processed, query a preset data block level table according to the data volume of the data to be processed, and obtain the number of data blocks for dividing the data to be processed, wherein the data block level table records the number of partitions corresponding to different data volume ranges;

[0058] The data to be processed is divided according to the number of the data blocks to obtain a plurality of data blocks.

[0059] The data volume (file size) of the data to be processed is determined according to the number of bytes or rows contained in the data to be processed, and the preset data block level table is queried according to the file size of the acquired data set to obtain the corresponding number of data blocks.

[0060] The preset data block level table is a predefined table or configuration file that lists the optimal number of data blocks for different data size ranges. For example, if the data size is between 1GB and 10GB, the recommended number of data blocks may be 100; if the data size exceeds 10GB but does not exceed 100GB, the recommended number of data blocks may be 500, and so on.

[0061] The data to be processed is divided into multiple data blocks according to the number of data blocks obtained. The amount of data contained in each data block should be relatively uniform to ensure the efficiency of parallel processing.

[0062] The present invention can dynamically determine the optimal number of data blocks according to the actual size of the data to be processed, thereby optimizing the efficiency of data processing. This method is particularly suitable for processing data sets of different sizes and can flexibly adjust the data block strategy according to different data volumes.

[0063] In other embodiments, the data to be processed may be divided into a plurality of data blocks according to average division, dynamic size division or based on data type, etc. The present invention does not impose any limitation on the division method.

[0064] In one embodiment, after dividing the data to be processed into a plurality of data blocks, the method further includes:

[0065] Get the metadata of each data block and calculate the hash value of the metadata of each data block;

[0066] The hash value of the metadata of each data block is used as the identifier of the corresponding data block.

[0067] Metadata refers to some basic information that describes a data block, such as the starting position, end position, or other information that helps locate or describe the data block. Obtaining metadata can better manage and identify each data block. Applying a hash function to the metadata of each data block generates a unique hash value. Using the hash value as the identifier of the data block ensures that each data block has a unique identifier, which helps with subsequent data block management, tracking, and data consistency checks. A hash value is a fixed-length string generated by a hash function, which is usually used to quickly find, compare, or verify the integrity of data.

[0068] In this way, each data block has a unique identifier, which makes it easy to identify and manage each data block in subsequent processing. For example, if you need to query a specific data block, you can use its hash value identifier to quickly locate it; if you need to verify whether the data block has changed, you can also confirm it by comparing the hash value.

[0069] In other embodiments, the path name in the file system may be used as an identifier, a timestamp + a random number (for example, an identifier is generated by combining the current timestamp and a random number), or an index-based identifier (if the data is divided into blocks in sequence, the index of the block may be directly used as an identifier. For example, block 1, block 2, block 3, etc.), etc. The present invention does not impose any limitation on the method of obtaining the identifier of the data block.

[0070] In step S1, through automated data block and resource allocation, it is ensured that each data block can obtain sufficient resource support, which can reduce the necessity of manual intervention and make the data processing process simpler and easier to manage.

[0071] S2. Call the allocated computing resources to run each data block, and record the amount of data that has been run and the amount of data that has not been run in each data block. According to the execution time of the computing resources to run the run data amount, the execution speed of each data block is calculated.

[0072] In this embodiment, a preset monitoring tool is enabled in the Spark application, for example, the preset monitoring tool includes the Spark UI and other third-party tools, and an event log is configured for each data block using the preset monitoring tool. The event log is used to record key indicators of each data block, including the amount of data that has been run, the amount of data that has not been run, and the execution time. Recording the amount of data for each data block helps to understand the workload of each data block, thereby better evaluating the overall performance and resource allocation of the system.

[0073] Corresponding computing resources are allocated to each data block, including but not limited to server nodes, number of CPU cores, and memory capacity. The execution time required to process the data in the data block using computing resources is recorded. The execution time refers to the time period from the start of processing a data block to the completion of processing.

[0074] Within a preset time period (e.g., 10 minutes), record the amount of data that has been run and the amount of data that has not been run for each data block within the preset time period, and record the execution time required to run the amount of data that has been run using computing resources.

[0075] The execution speed of each data block is calculated based on the amount of data that has been run and the execution time in the past 10 minutes. The execution speed of each data block in the preset time period can be obtained by dividing the amount of data that has been run by the execution time. For example, the execution speed of each data block per minute is 10M when the amount of data that has been run (100M) is divided by the execution time (10 minutes).

[0076] By calculating the execution speed, we can understand the average load of each data block over a period of time. If the execution speed of some data blocks is too high or too low, we can dynamically adjust or reallocate computing resources based on this information data processing.

[0077] In one embodiment, before calling the allocated computing resources to run each data block, the method further includes:

[0078] Open a preset monitoring tool and use the preset monitoring tool to configure an event log for each data block;

[0079] The event log is used to record key indicators of each data block, wherein the key indicators include the amount of data that has been executed, the amount of data that has not been executed, and the execution time.

[0080] Before the Spark application starts running, you need to enable the preset monitoring tools. For example, you can start the Spark UI that comes with Spark. The Spark UI provides a web interface that allows you to view the running status of each data block of the application in real time.

[0081] Configure an event log for each data block. The event log will record the key indicators of each data block, including the amount of data that has been run, the amount of data that has not been run, and the execution time. Specifically, the event log will record the amount of data that has been run, the amount of data that has not been run, and the execution time of each data block.

[0082] Event logging refers to the process of recording key events and operations in a Spark application. It is used to monitor the running status of each data block, debug problems, perform performance analysis, and audit.

[0083] By configuring and using event logging, you can ensure that Spark applications can be effectively monitored during operation and quickly locate and resolve problems when they occur.

[0084] In one embodiment, recording the amount of data that has been run and the amount of data that has not been run in each data block, and running the execution time of the run data amount according to the computing resource, includes:

[0085] respectively setting a first recording task and a second recording task on the event recording log;

[0086] The first recording task is used to record the amount of data that has been executed and the amount of data that has not been executed in each data block, and the second recording task is used to record the execution time required for each data block to execute the executed amount of data.

[0087] A first recording task and a second recording task are respectively set on the event recording log. The first recording task is used to record the amount of data that has been run and the amount of data that has not been run in each data block. The second recording task is used to record the execution time required for each data block to run the amount of data that has been run.

[0088] By setting the first and second recording tasks to record the amount of data that has been run, the amount of data that has not been run, and the execution time, you can better monitor the processing efficiency of each data block, dynamically adjust resource allocation, and optimize the data processing process. If the data volume of a data block is abnormal or the execution time is too long, you can also use these records to locate the problem.

[0089] In one embodiment, after calling the allocated computing resources to run each data block and executing the execution time of the data volume using the computing resources, the method further comprises:

[0090] Constructing an initial database table, wherein the initial database table at least includes the identifier of each data block, the amount of data, the execution time and the field items corresponding to the computing resources allocated to the data block;

[0091] The data volume and execution time of each data block at different time points are associated with the computing resources allocated to the data block and filled into the corresponding field items to obtain the target database table.

[0092] Use SQL code to create an initial database table, which is a database table used to store information related to each data block. The initial database table contains at least the following fields: Data block identifier: used to uniquely identify each data block. Data volume: the amount of data that has been run and the amount of data that has not been run contained in each data block, which can be the number of records, file size, etc. Execution time: the time required to process the data block. Computing resources: the computing resources allocated to the data block, such as server nodes, number of CPU cores, and memory capacity.

[0093] The amount of data that has been run, the amount of data that has not been run, and the execution time of each data block at different time points are associated with the computing resources allocated to the data block. Through data association, the running status of the Spark application can be monitored in real time. It can be used to analyze the performance of the Spark application in different time periods, help understand the changing trend of the Spark application load, and then make more reasonable resource allocation decisions.

[0094] Fill these associated data into the corresponding field items of the initial database table to generate a target database table. By building the target database table and recording the data volume, execution time and computing resources of each data block, the processing efficiency of each data block can be monitored in real time.

[0095] Although the key indicators of each data block can be recorded from the event log, there is a risk that the data recorded in the event log will be deleted or overwritten. Therefore, the present invention can be used to generate a detailed performance analysis report based on the data in the target database table to help system administrators and technical teams better understand the operating status of the system. When a problem occurs, it can be traced back to the specific data block and time point, and the responsibility can be clearly identified, which is convenient for the subsequent division of responsibilities and the formulation of improvement measures.

[0096] In step S2, by recording the amount of data that has been run, the amount of data that has not been run, and the execution time of each data block, the processing status of each data block can be monitored in real time, which is particularly important for a distributed computing environment because processing bottlenecks or abnormal situations can be discovered in a timely manner. Recording the amount of data that has been run, the amount of data that has not been run, and the execution time at each time point can form historical data for subsequent analysis and backtracking.

[0097] S3. Dynamically adjust the data blocks or computing resources according to the execution speed and the amount of unrun data, including: screening out data blocks whose amount of unrun data is less than a first preset threshold from the multiple data blocks, merging the data blocks whose amount of data is less than the first preset threshold after screening in pairs, screening out data blocks whose execution speed is less than a second preset threshold from the multiple data blocks, and adjusting the computing resources for the data blocks whose amount of data is less than the second preset threshold after screening, wherein the first preset threshold and the second preset threshold have different measurement units.

[0098] In this embodiment, the execution speed of each data block within the preset time period reflects the data processing load of each data block, and the amount of unexecuted data of each data block reflects the processing efficiency of each data block. According to the calculated execution speed and the amount of unexecuted data, the system automatically adjusts the data blocks to optimize the overall processing efficiency.

[0099] The first preset threshold and the second preset threshold have different measurement units. The first preset threshold is expressed in units such as bytes, kilobytes (KB), megabytes (MB), and gigabytes (GB). The second preset threshold is expressed in the amount of data processed per second (such as MB / s or GB / s), or in the time required to process unit data (such as milliseconds / ms or seconds / s).

[0100] Specifically, according to the set first preset threshold, the data blocks whose unrun data volume is less than the first preset threshold are screened and merged in pairs to form new data blocks. By merging the screened data blocks whose volume is less than the first preset threshold in pairs, it is possible to optimize the use of resources, reduce the number of unnecessary data blocks, and improve the overall processing efficiency of the system.

[0101] A predefined numerical standard is used as the first preset threshold, which is used to determine whether the amount of unrun data is low enough. The first preset threshold can be set according to actual needs, such as 50M or 100M.

[0102] The data blocks that are smaller than the first preset threshold after screening are merged in pairs. There are several specific merging methods: 1. Sequential merging: In the order of data blocks, the data blocks with adjacent sequence numbers are merged in pairs in sequence. 2. Balanced load merging: All data blocks that are smaller than the first preset threshold are sorted from small to large according to the data volume, and the data blocks with the largest data volume and the data blocks with the smallest data volume are selected from the sorting in turn to merge them in pairs until all the data blocks in the sorting are merged. 3. Random merging: Randomly select two data blocks to merge. 4. Priority merging: Select data blocks for merging according to certain priority rules (such as data volume, execution time, data block number, etc.).

[0103] Additional computing resources are allocated to the data blocks whose screening execution speed is less than the second preset threshold to improve their processing efficiency. The computing resources may include more CPU cores, larger memory space, faster storage devices, etc. The second preset threshold is used to determine whether the execution speed is too low. The second preset threshold can be set according to actual needs and is usually used to identify data blocks with low processing efficiency, such as 5M / sec or 10M / sec.

[0104] The present invention can reduce the number of unnecessary data blocks by merging low-load data blocks, making the load of the entire system more balanced. Merging data blocks can reduce redundant management overhead and improve the simplicity and maintainability of the system. Allocating additional computing resources for data blocks with low processing efficiency can improve their processing efficiency and reduce processing time.

[0105] For example:

[0106] The system has been running for a while and has collected the amount of data that has been run, the amount of data that has not been run, and the execution time of each data block. Now, the system needs to decide which data blocks should be merged based on this data.

[0107] Assume that the first preset threshold is set to 100 MB, which means that any data block with an unrunning data volume less than 100 MB will be regarded as a candidate for merging, and all data blocks with an unrunning data volume less than 100 MB will be screened out.

[0108] Assume that the selected data blocks are as follows: Data block A: 80MB of unrunning data. Data block B: 90MB of unrunning data. Data block C: 120MB of unrunning data. Data block D: 70MB of unrunning data. In this example, data blocks A, B, and D will all be considered for merging, while data block C will not be considered because its unrunning data exceeds the first preset threshold.

[0109] The system also needs to check which data blocks have an execution speed lower than the second preset threshold. Assume that the second preset threshold is set to process 50MB of data per minute. Filter out all data blocks with an execution speed lower than 50MB / minute. Assume that the filtered data blocks are as follows: Data block E: Execution speed 45MB / minute. Data block F: Execution speed 40MB / minute. These data blocks require additional computing resources to increase the processing speed due to their low processing efficiency.

[0110] In one embodiment, merging two by two of the data blocks whose values ​​are smaller than the first preset threshold after screening includes:

[0111] sorting the data blocks that are smaller than the first preset threshold after screening in ascending order according to the identifiers of the data blocks;

[0112] Two adjacent data blocks in the sorting are merged, and computing resources of any one of the two adjacent data blocks are released.

[0113] Filter out data blocks whose unrun data volume is less than a first preset threshold, sort them from small to large according to the identifiers of the data blocks (usually numeric or alphabetical numbers), merge the two adjacent data blocks after sorting, and release the computing resources of one of the data blocks during the merging process.

[0114] By merging data blocks with smaller data volumes and shorter processing times and releasing the computing resources of one of the data blocks, unnecessary resource usage can be reduced and resource utilization can be improved. The merged data blocks still maintain a lower processing time while having a moderate data volume, thereby improving overall processing efficiency.

[0115] In one embodiment, merging two by two of the data blocks whose values ​​are smaller than the first preset threshold after screening includes:

[0116] Sorting the filtered data blocks that are smaller than the first preset threshold in ascending order according to the data volume to obtain a sorted merge list;

[0117] From the sorted merge list, the data blocks with the largest data volume and the data blocks with the smallest data volume are selected in turn to be merged in pairs, until all the data blocks in the sorted merge list are merged.

[0118] Filter out the data blocks whose unrun data volume is less than the first preset threshold, sort them from small to large according to the data volume, and obtain a sorted merge list. The sorted merge list provides a clear organization of the data blocks, which is convenient for management and operation. Through the list form, the information of all data blocks can be managed uniformly, which is convenient for subsequent data processing and resource allocation.

[0119] From the sorted sorted merge list, select the data block with the largest data volume and the data block with the smallest data volume in turn to merge them in pairs until all data blocks are merged.

[0120] In one embodiment, merging two by two of the data blocks whose values ​​are smaller than the first preset threshold after screening includes:

[0121] According to the preset type attribute of the data block, the data blocks that are smaller than the first preset threshold value after screening are classified to obtain multiple categories;

[0122] Two data blocks with adjacent identifiers in the same category are merged, and computing resources of any one of the two adjacent data blocks are released.

[0123] Data blocks whose amount of unrun data is less than a first preset threshold are screened, and the screened data blocks are classified according to preset type attributes of the data blocks (for example, data blocks are classified according to time periods, geographical locations or other characteristics) to obtain multiple categories. Within the same category, two data blocks with adjacent identifiers are merged, and computing resources of one of the data blocks are released.

[0124] Classification and merging based on type attributes has many advantages, especially when processing large-scale data sets and optimizing system performance:

[0125] 1. Data blocks of the same type often have similar data characteristics and processing requirements. Merging these data blocks together can better achieve load balancing, because the merged data blocks are more evenly distributed in terms of data volume and processing time. The merged data blocks can more reasonably allocate computing resources according to their type characteristics, thereby improving resource utilization.

[0126] 2. If data blocks of the same type are stored on the same physical node, the merged data blocks can reduce the data transmission overhead between different nodes, thereby increasing the speed of data processing. In a distributed computing environment, the merged data blocks can better adapt to the parallel processing framework and improve processing efficiency.

[0127] Take the banking and finance scenario as an example:

[0128] Banks process a large amount of transaction data every day, which is usually divided into multiple data blocks for processing. By merging data blocks with smaller data volumes and classifying and merging them according to type attributes, processing delays can be reduced and transaction processing efficiency can be improved.

[0129] Assume that the bank's transaction data is divided into multiple data blocks, each of which contains transaction records within a day and is classified according to time periods (such as morning transactions, lunch transactions, and evening transactions). By merging data blocks with smaller data volumes within the same time period, the data processing time can be reduced and some computing resources can be freed up for other important tasks.

[0130] Banks need to manage a large amount of customer information, which is usually divided into multiple data blocks for storage and processing. By merging data blocks with smaller data volumes and classifying and merging them according to type attributes, the customer information management system can be optimized and query efficiency can be improved.

[0131] The method further comprises:

[0132] The step of dynamically adjusting the data blocks or computing resources according to the execution speed and the amount of unexecuted data is repeatedly performed at every preset time period until the execution task of the data to be processed is completed.

[0133] In this embodiment, the system will repeat the operation of dynamically adjusting data processing every preset time period (for example, every 10 minutes), calculate the execution speed of each data block and the amount of unrun data, and dynamically adjust the number of data blocks according to the execution speed and the amount of unrun data. For example, low-load data blocks can be merged to reduce the number of unnecessary data blocks, and additional computing resources can be allocated to data blocks with lower processing efficiency to optimize processing efficiency and resource utilization until all running tasks of the data to be processed are completed.

[0134] By dynamically adjusting data processing at preset time intervals, data processing efficiency and resource utilization can be continuously optimized. As data volume and processing requirements change, dynamic adjustments can better adapt to these changes and ensure that the system is always in the best condition.

[0135] By timely merging low-load data blocks or allocating additional computing resources for data blocks with low processing efficiency, bottlenecks in the system can be reduced and the overall processing speed can be improved.

[0136] Based on the above analysis of steps S1-S3, the present invention dynamically adjusts data blocks according to the execution speed of each data block and the amount of unrun data, which can more reasonably allocate computing resources and reduce unnecessary waste of resources. It can maximize the advantages of parallel processing and shorten the total time of data processing. The automatic adjustment of the number of partitions reduces the need for manual intervention and the complexity of operation and maintenance, making the system easier to manage and maintain.

[0137] like Figure 2 FIG. 1 is a schematic diagram of a module of a data processing dynamic adjustment device provided in an embodiment of the present invention.

[0138] The data processing dynamic adjustment device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the data processing dynamic adjustment device 100 can include a receiving module 110, a recording module 120, and an adjustment module 130. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0139] In this embodiment, the functions of each module / unit are as follows:

[0140] The receiving module 110 is used to receive a request for data partitioning of the data to be processed, divide the data to be processed into a plurality of data blocks, and configure computing resources for each data block, wherein the computing resources include one of a server node, a number of CPU cores, and a memory capacity;

[0141] The recording module 120 is used to call the allocated computing resources to run each data block, and record the amount of data that has been run and the amount of data that has not been run in each data block, and calculate the execution speed of each data block according to the execution time of the computing resources running the run data amount;

[0142] The adjustment module 130 is used to dynamically adjust the data blocks or computing resources according to the execution speed and the amount of unrun data, including: from the multiple data blocks, screening the data blocks whose amount of unrun data is less than a first preset threshold, merging the data blocks whose amount is less than the first preset threshold after screening in pairs, screening the data blocks whose execution speed is less than a second preset threshold from the multiple data blocks, and adjusting the computing resources for the data blocks whose amount is less than the second preset threshold after screening, wherein the first preset threshold and the second preset threshold have different measurement units.

[0143] like Figure 3 , which is a schematic diagram of the structure of an electronic device for implementing a method for dynamic adjustment of data processing provided by an embodiment of the present invention.

[0144] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13, which can be interconnected and communicated via a system bus. The memory 11 stores a data processing dynamic adjustment program 10, and the data processing dynamic adjustment program 10 can be executed by the processor 12. Figure 3 Only the electronic device 1 having the components 11-13 and the data processing dynamic adjustment program 10 is shown. It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0145] Among them, the memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. In this embodiment, the readable storage medium of the memory 11 is generally used to store an operating system and various application software installed on the electronic device 1, such as storing the code of the data processing dynamic adjustment program 10 in one embodiment of the present invention. In addition, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.

[0146] The processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the data processing dynamic adjustment program 10.

[0147] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a terminal (not shown in the figure).

[0148] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0149] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0150] The data processing dynamic adjustment program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 12, it can achieve:

[0151] receiving a request for data partitioning of data to be processed, dividing the data to be processed into a plurality of data blocks, and configuring computing resources for each data block, wherein the computing resources include one of a server node, a number of CPU cores, and a memory capacity;

[0152] Calling the allocated computing resources to run each data block, and recording the amount of data that has been run and the amount of data that has not been run in each data block, and calculating the execution speed of each data block according to the execution time of the computing resources running the run data amount;

[0153] Dynamically adjusting the data blocks or computing resources according to the execution speed and the amount of unrun data includes: screening, from the multiple data blocks, data blocks whose amount of unrun data is less than a first preset threshold, merging the data blocks whose amount of data is less than the first preset threshold after screening in pairs, screening, from the multiple data blocks, data blocks whose execution speed is less than a second preset threshold, and adjusting the computing resources for the data blocks whose amount of data is less than the second preset threshold after screening, wherein the first preset threshold and the second preset threshold have different measurement units.

[0154] Specifically, the specific implementation method of the processor 12 for the above data processing dynamic adjustment program 10 can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0155] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0156] The computer-readable storage medium stores a data processing dynamic adjustment program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the above-mentioned data processing dynamic adjustment method embodiments, and will not be repeated here.

[0157] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0158] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0160] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0161] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0162] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A data processing dynamic adjustment method, characterized in that: The method comprises: receiving a request for data partitioning of data to be processed, dividing the data to be processed into a plurality of data blocks, and configuring computing resources for each data block, wherein the computing resources include one of a server node, a number of CPU cores, and a memory capacity; Calling the allocated computing resources to run each data block, and recording the amount of data that has been run and the amount of data that has not been run in each data block, and calculating the execution speed of each data block according to the execution time of the computing resources running the run data amount; Dynamically adjusting the data blocks or computing resources according to the execution speed and the amount of unrun data includes: screening, from the multiple data blocks, data blocks whose amount of unrun data is less than a first preset threshold, merging the data blocks whose amount of data is less than the first preset threshold after screening in pairs, screening, from the multiple data blocks, data blocks whose execution speed is less than a second preset threshold, and adjusting the computing resources for the data blocks whose amount of data is less than the second preset threshold after screening, wherein the first preset threshold and the second preset threshold have different measurement units.

2. The data processing dynamic adjustment method according to claim 1, characterized in that: After dividing the data to be processed into a plurality of data blocks, the method further includes: Get the metadata of each data block and calculate the hash value of the metadata of each data block; The hash value of the metadata of each data block is used as the identifier of the corresponding data block.

3. The data processing dynamic adjustment method according to claim 1, characterized in that: Before calling the allocated computing resources to run each data block, the method further includes: Open a preset monitoring tool and use the preset monitoring tool to configure an event log for each data block; The event log is used to record key indicators of each data block, wherein the key indicators include the amount of data that has been executed, the amount of data that has not been executed, and the execution time.

4. The data processing dynamic adjustment method according to claim 1 or 3, characterized in that: The recording of the amount of data that has been run and the amount of data that has not been run in each data block, and the execution time of running the amount of data that has been run according to the computing resource, includes: respectively setting a first recording task and a second recording task on the event recording log; The first recording task is used to record the amount of data that has been executed and the amount of data that has not been executed in each data block, and the second recording task is used to record the execution time required for each data block to execute the executed amount of data.

5. The data processing dynamic adjustment method according to claim 1 or 2, characterized in that: The step of merging two by two of the data blocks that are smaller than the first preset threshold after screening includes: sorting the data blocks that are smaller than the first preset threshold after screening in ascending order according to the identifiers of the data blocks; Two adjacent data blocks in the sorting are merged, and computing resources of any one of the two adjacent data blocks are released.

6. The data processing dynamic adjustment method according to claim 1, characterized in that: The step of merging two by two of the data blocks that are smaller than the first preset threshold after screening includes: Sorting the filtered data blocks that are smaller than the first preset threshold in ascending order according to the data volume to obtain a sorted merge list; From the sorted merge list, the data blocks with the largest data volume and the data blocks with the smallest data volume are selected in turn to be merged in pairs, until all the data blocks in the sorted merge list are merged.

7. The data processing dynamic adjustment method according to claim 1, characterized in that: The method further comprises: The step of dynamically adjusting the data blocks or computing resources according to the execution speed and the amount of unexecuted data is repeatedly performed at every preset time period until the execution task of the data to be processed is completed.

8. A data processing dynamic adjustment device, characterized in that: The device comprises: A receiving module, configured to receive a request for partitioning the data to be processed, divide the data to be processed into a plurality of data blocks, and configure computing resources for each data block, wherein the computing resources include one of a server node, a number of CPU cores, and a memory capacity; A recording module, used to call the allocated computing resources to run each data block, and record the amount of data that has been run and the amount of data that has not been run in each data block, and calculate the execution speed of each data block according to the execution time of the computing resources running the run data amount; An adjustment module is used to dynamically adjust the data blocks or computing resources according to the execution speed and the amount of unrun data, including: from the multiple data blocks, screening the data blocks whose amount of unrun data is less than a first preset threshold, merging the data blocks whose amount of data is less than the first preset threshold after screening in pairs, screening the data blocks whose execution speed is less than a second preset threshold from the multiple data blocks, and adjusting the computing resources for the data blocks whose amount of data is less than the second preset threshold after screening, wherein the first preset threshold and the second preset threshold have different measurement units.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a data processing dynamic adjustment program that can be executed by the at least one processor, and the data processing dynamic adjustment program is executed by the at least one processor so that the at least one processor can execute the data processing dynamic adjustment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data processing dynamic adjustment program, which can be executed by one or more processors to implement the data processing dynamic adjustment method according to any one of claims 1 to 7.

Images