Data processing method, device and storage medium based on big data platform

Abstract

The embodiment of the application relates to the technical field of big data platforms, and provides a data processing method, equipment and storage medium based on a big data platform. The data processing method based on the big data platform comprises the following steps: determining a plurality of first data sets of a target task, the first data set being a data set read remotely when the target task is executed; determining a hot data set in the plurality of first data sets; determining a plurality of target computing nodes in the big data platform; storing the hot data set into memories of the plurality of target computing nodes and generating mapping information; and the mapping information is used for indicating a corresponding relationship between the hot data set and the plurality of target computing nodes. The embodiment of the application pre-identifies the corresponding hot data set when a computing engine executes a big data-based computing task, and pre-stores the hot data set in a memory of a computing server, so that the computing server can directly read data from the memory, and the system performance of the big data platform is improved.

Description

Technical Field

[0001] This application relates to the field of big data platform technology, and in particular to a data processing method, device and storage medium based on a big data platform. Background Technology

[0002] In the digital economy era, data has become a new means of production, and the focus has shifted from data management to data operation, with big data playing an increasingly important role.

[0003] When big data first emerged, mainstream network bandwidth was only 100Mb, making remote data access too slow. To address the issue of fast data access, a big data storage-compute convergence architecture was proposed. In this architecture, computing and storage are coupled and deployed on the same machine, avoiding significant resource overhead from input / output (IO).

[0004] With improvements in I / O optimization and network performance, network bandwidth has increased from 100Mb to 10Gb. Various efficient compression algorithms and data storage formats have further reduced I / O data volume, shifting the bottleneck of big data from I / O to computation. Therefore, a storage-compute separation architecture for big data has been proposed. In this architecture, computation and storage are decoupled and deployed on different machines, allowing for separate adjustments to storage and computation resources. This architecture enables big data clusters to fully utilize both storage and computation resources, resulting in strong system scalability, low expansion costs, flexible networking, and better alignment with cloud computing characteristics.

[0005] However, after the separation of storage and computing, the data storage location is moved to a remote location, and the data needs to be retrieved from the remote location, which leads to a significant decrease in the system performance of the big data platform. Summary of the Invention

[0006] This application provides a data processing method, device, and storage medium based on a big data platform, which improves the system performance of the big data platform.

[0007] In a first aspect, a data processing method based on a big data platform is provided, comprising: determining multiple first datasets for a target task, wherein the first datasets are datasets remotely read when executing the target task; determining hot datasets among the multiple first datasets; determining multiple target computing nodes in the big data platform; storing the hot datasets in the memory of the multiple target computing nodes and generating mapping information; the mapping information is used to indicate the correspondence between the hot datasets and the multiple target computing nodes.

[0008] The first aspect provides a data processing method based on a big data platform. By defining multiple primary datasets for the target task, it can pre-identify the hot datasets corresponding to the big data-based computing tasks executed by the computing engine, and store these hot datasets in the memory of the computing server. In this way, when the computing server executes big data-based computing tasks, it can directly read data from memory, increasing the degree of data localization and avoiding the need for the computing engine to fetch all the required data from remote locations. This fully utilizes computing and storage resources and improves the system performance of the big data platform.

[0009] One possible implementation also includes displaying the mapping information to the user via an electronic device.

[0010] One possible implementation also includes: obtaining modification instructions input by the user through an electronic device, the modification instructions being used to modify the correspondence between the hotspot dataset indicated by the mapping information and multiple target computing nodes.

[0011] One possible implementation also includes: distributing the target task to multiple target computing nodes based on the mapping information.

[0012] One possible implementation involves storing the hotspot dataset in the memory of multiple target computing nodes, including: obtaining file partitioning levels; partitioning the hotspot dataset according to the preset file size corresponding to the file partitioning levels to obtain multiple file packages; and storing the multiple file packages in the memory of multiple target computing nodes.

[0013] In one possible implementation, obtaining the file classification level includes: obtaining the proportion of files stored on the big data platform within multiple preset file size ranges during the second time period; and obtaining the file classification level corresponding to the preset file size range with the largest proportion.

[0014] One possible implementation involves storing multiple file packages in the memory of multiple target computing nodes, including: using a three-replica mechanism to store multiple file packages in the memory of multiple target computing nodes.

[0015] In one possible implementation, the target task includes multiple SQL tasks, and multiple first datasets for the target task are determined, including: obtaining first historical data within a first time period; the first historical data is used to indicate the first dataset corresponding to each SQL task in the target task; and determining the first dataset corresponding to each SQL task in the target task based on the first historical data.

[0016] In one possible implementation, hot datasets are identified among multiple first datasets, including: for each first dataset corresponding to each SQL task in the target task, determining the block popularity of the first dataset relative to the SQL task based on the IO rate, the number of times the SQL task reads the first dataset, and the size of the first dataset; for each first dataset, determining the file popularity of the first dataset by summing the block popularity of the first dataset relative to all SQL tasks in the target task; and identifying hot datasets based on the file popularity corresponding to the multiple first datasets respectively.

[0017] In one possible implementation, the hot data set is determined based on the file popularity corresponding to multiple first data sets, including: determining the first data set whose file popularity is greater than a preset threshold as the hot data set.

[0018] In one possible implementation, determining multiple target computing nodes in a big data platform includes: obtaining performance parameters of computing nodes in the big data platform; and determining multiple target computing nodes based on the performance parameters of the computing nodes.

[0019] In one possible implementation, the performance parameters include at least one of the following: CPU utilization, CPU load, system idle time, memory utilization, or I / O rate.

[0020] In one possible implementation, before determining multiple first datasets for the target task, the method further includes: obtaining multiple SQL tasks corresponding to the target business; dividing the multiple SQL tasks into multiple target tasks; the target tasks include some SQL tasks from the multiple SQL tasks, and the SQL tasks included in different target tasks do not overlap.

[0021] Secondly, a data processing device based on a big data platform is provided, comprising: a hotspot analysis module for determining multiple first datasets of a target task and identifying hotspot datasets among the multiple first datasets; the first datasets are datasets remotely read when executing the target task; a node orchestration module for determining multiple target computing nodes in the big data platform; and a preheating module for storing the hotspot datasets in the memory of the multiple target computing nodes and generating mapping information; the mapping information is used to indicate the correspondence between the hotspot datasets and the multiple target computing nodes.

[0022] One possible implementation also includes a cache orchestration module, which is used to display mapping information to the user via an electronic device.

[0023] One possible implementation also includes a cache orchestration module, which is used to: obtain modification instructions input by the user through an electronic device, and modify the correspondence between the hot dataset indicated by the mapping information and multiple target computing nodes.

[0024] One possible implementation also includes a task distribution module, which is used to distribute the target task to multiple target computing nodes according to the mapping information.

[0025] In one possible implementation, the preheating module is used to: obtain file partitioning levels; partition the hotspot dataset according to the preset file size corresponding to the file partitioning level to obtain multiple file packages; and store the multiple file packages in the memory of multiple target computing nodes.

[0026] In one possible implementation, the preheating module is used to: obtain the proportion of files stored on the big data platform within multiple preset file size ranges during the second time period; and obtain the file classification level corresponding to the preset file size range with the largest proportion.

[0027] In one possible implementation, the warm-up module is used to: store multiple file packages in the memory of multiple target computing nodes using a three-replica mechanism.

[0028] In one possible implementation, the target task includes multiple SQL tasks, and the hotspot analysis module is used to: obtain the first historical data within a first time period; the first historical data is used to indicate the first dataset corresponding to each SQL task in the target task; and determine the first dataset corresponding to each SQL task in the target task based on the first historical data.

[0029] In one possible implementation, the hotspot analysis module is used to: for each first dataset corresponding to each SQL task in the target task, determine the block heat of the first dataset relative to the SQL task based on the input / output IO rate, the number of times the SQL task reads the first dataset, and the size of the first dataset; for each first dataset, determine the file heat of the first dataset by summing the block heat of the first dataset relative to all SQL tasks in the target task; and determine the hot dataset based on the file heat corresponding to multiple first datasets respectively.

[0030] In one possible implementation, the hotspot analysis module is used to: identify the first dataset whose file popularity exceeds a preset threshold as the hotspot dataset.

[0031] In one possible implementation, the node orchestration module is used to: obtain the performance parameters of computing nodes in the big data platform; and determine multiple target computing nodes based on the performance parameters of the computing nodes.

[0032] In one possible implementation, the performance parameters include at least one of the following: CPU utilization, CPU load, system idle time, memory utilization, or I / O rate.

[0033] In one possible implementation, the hotspot analysis module is also used to: obtain multiple SQL tasks corresponding to the target business before determining multiple first datasets of the target task; divide the multiple SQL tasks into multiple target tasks; the target tasks include some SQL tasks among the multiple SQL tasks, and the SQL tasks included in different target tasks do not overlap.

[0034] Thirdly, a big data platform server is provided, including a processor, which is coupled to a memory and reads instructions from the memory and causes the big data platform server to execute the method provided in the first aspect according to the instructions.

[0035] Fourthly, a program is provided that, when executed by a processor, performs the method provided in the first aspect.

[0036] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer or processor, implement the method provided in the first aspect.

[0037] In a sixth aspect, a program product is provided, comprising a computer program stored in a readable storage medium, wherein at least one processor of the device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the device to implement the method provided in the first aspect. Attached Figure Description

[0038] Figure 1 This is a physical architecture diagram of the big data platform system to which the embodiments of this application apply;

[0039] Figure 2 This is a software architecture diagram of a big data platform under a storage-compute separation architecture.

[0040] Figure 3 This is another software architecture diagram for a big data platform under a storage-compute separation architecture;

[0041] Figure 4 A software architecture diagram of a big data platform provided in this application embodiment;

[0042] Figure 5 A flowchart illustrating a data processing method based on a big data platform provided in this application embodiment;

[0043] Figure 6 A schematic diagram of the first dataset and the hotspot dataset provided in the embodiments of this application;

[0044] Figure 7 A schematic diagram of a cache orchestration interface displayed on an electronic device provided in an embodiment of this application;

[0045] Figure 8 Another schematic diagram of the cache orchestration interface displayed on the electronic device provided in the embodiments of this application;

[0046] Figure 9 A schematic diagram of the hotspot dataset and file package provided in the embodiments of this application;

[0047] Figure 10 A schematic diagram of the structure of a data processing device based on a big data platform provided in an embodiment of this application;

[0048] Figure 11 This is a schematic diagram of the device provided in an embodiment of this application. Detailed Implementation

[0049] The embodiments of this application are described below with reference to the accompanying drawings.

[0050] The data processing method based on a big data platform provided in this application is applicable to big data platforms employing a storage-compute separation architecture. Storage-compute separation, also known as storage-compute separation, refers to the decoupling of computing and storage, with each deployed on different servers. Computing resources can share storage resources, and both computing and storage resources can be expanded independently.

[0051] First, the physical architecture of the big data platform under the storage-compute separation architecture will be explained.

[0052] For example, Figure 1 This is a physical architecture diagram of a big data platform system to which the embodiments of this application apply. Figure 1 As shown, the big data platform system includes a big data platform and electronic devices 300. The big data platform includes a storage server 100 and a computing server 200. The storage server 100 is mainly used for the storage management of big data. The computing server 200 is mainly used for big data-based computing tasks, such as user profiling analysis, user consumption analysis, demographic analysis, etc. This embodiment does not limit the big data-based computing tasks. The number of storage servers 100 and the number of computing servers 200 can be individually set, increased, or decreased according to storage resource requirements. It should be noted that this embodiment does not limit the number, type, or name of the servers, nor does it limit the storage capacity or computing power of the servers. For example, a server can also be called a node.

[0053] In the big data platform system, the electronic device 300 can communicate with the server in the big data platform to realize functions such as information display and human-computer interaction between the user and the big data platform. Optionally, an application (AP) related to big data processing can be installed on the electronic device 300. This application embodiment does not limit the name and specific function of the APP. It should be noted that this application embodiment does not limit the number, type, and name of the electronic devices. For example, some examples of electronic devices include: desktop computers, laptops, tablet computers, mobile phones, etc.

[0054] The following section describes the software architecture of the big data platform under the storage-compute separation architecture.

[0055] For example, in one implementation, Figure 2 This is a software architecture diagram for a big data platform under a storage-compute separation architecture. For example... Figure 2 As shown, the software architecture of the big data platform includes a storage engine 21 at the bottom layer and a computing engine 22 at the top layer. Storage engine 21 is deployed on a storage server, and computing engine 22 is deployed on a computing server. Storage engine 21 can also be called a distributed file system, remote or cloud-based. Examples of storage engines 21 include: Amazon Web Services (also known as Amazon S3), Apache Hadoop Distributed File System (HDFS), Alibaba Cloud Object Storage Service (OBS), and Huawei Cloud Object Storage Service (OSS). Computing engine 22 can also be called a distributed computing framework. Examples of computing engines 22 include: Spark, MapReduce, HBase, Flink, and Hive. Computing engine 22 needs to remotely read and write data from storage engine 21 via the network.

[0056] For example, in another implementation, Figure 3 This is another software architecture diagram for a big data platform under a storage-compute separation architecture. Figure 2 Alluxio was introduced based on the software architecture shown. For example... Figure 3 As shown, the software architecture of the big data platform includes a storage engine 21, a computing engine 22, and a cache engine 31 (i.e., Alluxio) located between the storage engine 21 and the computing engine 22. Descriptions of the storage engine 21 and the computing engine 22 are provided above. Figure 2This will not be elaborated upon here. Alluxio is a memory-based virtual distributed storage system, serving as middleware between the underlying distributed file system and the upper-level distributed computing framework. Alluxio unifies data access methods, providing a universal data access interface. Computing engine 22 can access data in any underlying storage engine 21 through Alluxio. Furthermore, Alluxio provides data access services in memory as files. Figure 3 The big data platform shown is compared to Figure 2 The big data platform shown offers faster data access.

[0057] exist Figure 2 or Figure 3 In the compute-storage separation architecture shown, the data required by the computing engine is stored remotely, which means that the computing engine needs to pull the data from the remote location. This results in a significant loss of data locality, insufficient utilization of computing and storage resources, and a significant decline in the system performance of the big data platform.

[0058] To solve this technical problem, Figure 4 This is a software architecture diagram of a big data platform provided in an embodiment of this application. Figure 4 As shown, the software architecture of the big data platform includes a storage engine 21, a computing engine 22, a caching engine 31, and a cache orchestration engine 41. Descriptions of the storage engine 21, computing engine 22, and caching engine 31 can be found above. Figure 3 This will not be elaborated further here. In this embodiment, by adding a cache orchestration engine 41, the hot datasets corresponding to the computing engine's execution of big data-based computing tasks can be identified in advance, and these hot datasets can be pre-stored in the computing server's memory. In this way, when the computing server executes big data-based computing tasks, it can directly read data from memory, increasing the degree of data localization and avoiding the need for the computing engine to fetch all the data required from a remote location. This fully utilizes computing and storage resources and improves the system performance of the big data platform.

[0059] This application does not limit the application scenarios of the big data platform. For example, application scenarios include, but are not limited to: IO-intensive application scenarios, big data volume business scenarios, etc.

[0060] The technical solution of this application will be described in detail below through specific embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0061] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0063] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0064] Figure 5 This is a flowchart illustrating a data processing method based on a big data platform provided in an embodiment of this application. The data processing method based on a big data platform provided in this embodiment can be executed by a data processing device or server based on a big data platform, and the software framework is as follows: Figure 4 As shown. Optionally, the server can be a computing server or computing node in a big data platform. For example, this embodiment uses a computing server as the execution entity for illustration. Figure 5 As shown, the data processing method based on a big data platform provided in this embodiment may include:

[0065] S501. Determine multiple first datasets for the target task. The first datasets are datasets that are remotely read when the target task is executed.

[0066] This embodiment does not limit the business logic implemented by the target task. For example, the target task could implement a big data user profiling service. Optionally, the target task may include multiple Structured Query Language (SQL) tasks, and this embodiment does not limit the number of SQL tasks included in the target task. Each SQL task corresponds to at least one first dataset, and this embodiment does not limit the number of first datasets corresponding to each SQL task or the number of first datasets corresponding to the target task. The first dataset is a dataset remotely read by the computing server when executing the target task. That is, the first dataset is not currently stored on the computing server, but on a storage server within the big data platform, and the computing server needs to read the first dataset from the storage server via the network.

[0067] It's understandable that the first datasets corresponding to different SQL tasks may or may not overlap. This overlap can also be understood as an intersection. Let's illustrate with an example. Figure 6 This is a schematic diagram of the first dataset and the hotspot dataset provided in an embodiment of this application. Figure 6As shown, the target task includes four SQL tasks, labeled SQL1 to SQL4. SQL1 has three corresponding datasets: path1, path2, and path3. Here, path represents the path of the dataset. SQL2 has two corresponding datasets: path2 and path3. SQL3 has two corresponding datasets: path3 and path4. SQL4 has three corresponding datasets: path4, path5, and path6. It can be seen that the first datasets corresponding to SQL1 and SQL2 each have paths2 and path3, indicating overlap. The first datasets corresponding to SQL2 and SQL3 each have path3. The first datasets corresponding to SQL1 and SQL4 do not overlap.

[0068] S502. Identify hotspot datasets among multiple primary datasets.

[0069] A "hotspot dataset" refers to a dataset that is frequently used and repeatedly accessed during the execution of the target task. This embodiment does not limit the number of hotspot datasets in the first dataset.

[0070] Optionally, the computing server can interact with electronic devices to display hot datasets to users through the electronic devices' interface. Displaying hot datasets through electronic devices allows users to receive real-time analysis results of the first dataset for the target task, understand the data processing progress and intermediate results, facilitate manual intervention, and improve the user experience.

[0071] This embodiment does not limit the layout of the interface displayed on the electronic device. The following example illustrates this.

[0072] Assuming, such as Figure 6 As shown, the first dataset for the target task includes path1, path2, path3, path4, path5, and path6. The hotspot datasets identified within this first dataset include path1, path2, path4, and path6. Path1 and path2 can be processed in parallel, while path1 and path2, and path4 and path6 can be processed sequentially. The topology of the hotspot datasets is as follows: Figure 6 As shown below. For example, Figure 7 This is a schematic diagram of a cache orchestration interface displayed on an electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the cache orchestration interface 71 displayed on the electronic device includes a hotspot analysis control 72, a data warm-up control 73, a node orchestration control 74, and a task execution control 75. Users can click on the hotspot analysis control 72 to view the hotspot analysis structure in window 76, for example, to view the topology of the hotspot dataset.

[0073] S503. Identify multiple target computing nodes in the big data platform.

[0074] Specifically, a big data platform includes multiple computing nodes, and the computing node used to execute the target task is called the target computing node.

[0075] S504. Store the hotspot dataset in the memory of multiple target computing nodes and generate mapping information. The mapping information is used to indicate the correspondence between the hotspot dataset and the multiple target computing nodes.

[0076] Specifically, since hot datasets are datasets that are read frequently and utilized extensively when executing the target task, by pre-storing hot datasets in the memory of the target computing node, the target computing node can directly read hot datasets from memory when executing the target task, thereby improving memory hit rate, providing memory-level IO capabilities, and reducing the amount of data read remotely.

[0077] S503 and S504 can be executed serially or in parallel.

[0078] As can be seen, the data processing method based on a big data platform provided in this embodiment, by determining multiple first datasets for the target task, can pre-identify the hot datasets corresponding to the computing engine's execution of big data-based computing tasks within these datasets, and store these hot datasets in the computing server's memory beforehand. In this way, when the computing server executes big data-based computing tasks, it can directly read data from memory, increasing the degree of data localization, avoiding the need for the computing engine to fetch all the data required from a remote location, fully utilizing computing and storage resources, and improving the system performance of the big data platform.

[0079] Optionally, the data processing method based on a big data platform provided in this embodiment may further include, after S504:

[0080] S505. Display mapping information to the user via electronic devices.

[0081] Specifically, the computing server can send mapping information to electronic devices. Correspondingly, the electronic devices receive the mapping information sent by the computing server and display it on their interface.

[0082] Displaying mapping information via electronic devices allows users to know the storage relationship between hot datasets and computing nodes in real time. Users can also understand the data processing progress and intermediate processing results, which facilitates manual intervention and improves the user experience.

[0083] This embodiment does not limit the layout of the interface displayed on the electronic device. For example, Figure 8This is another schematic diagram of the cache orchestration interface displayed on the electronic device provided in an embodiment of this application. For example... Figure 8 As shown, the cache orchestration interface 71 displayed on the electronic device includes a hotspot analysis control 72, a data warm-up control 73, a node orchestration control 74, and a task execution control 75. Users can click the data warm-up control 73 to view mapping information in window 77. For example, window 77 includes four records. Each record corresponds to a hotspot dataset, along with the path of that hotspot dataset and its corresponding target computing nodes. For example, for the first record, the storage path of the hotspot dataset data1 is hdfs: / / table1 / path, and the corresponding target computing nodes are node1 and node2. That is, the hotspot dataset data1 is stored in the memory of computing nodes node1 and node2.

[0084] Optionally, the data processing method based on a big data platform provided in this embodiment may further include:

[0085] The modification instructions are obtained by acquiring user input through electronic devices. These instructions are used to modify the correspondence between the hotspot dataset indicated by the mapping information and multiple target computing nodes.

[0086] return Figure 8 For example, see the illustration. Figure 8 As shown, each record displayed in window 77 includes a modify button 78. Users can click button 78 to modify the mapping between the hotspot dataset and the target computing nodes. For example, clicking the modify button 78 in the first record changes the nodes from node1,node2 to node1,node3. Correspondingly, the electronic device can receive the modification command input by the user and send it to the computing server. The computing server receives the modification command sent by the electronic device and, based on the command, modifies the mapping information of the hotspot dataset data1 to correspond to the computing nodes node1,node3.

[0087] As can be seen, users can manually modify the storage relationship between hot datasets and computing nodes according to actual conditions, which improves the flexibility and accuracy of data processing and enhances the user experience.

[0088] Optionally, the data processing method based on a big data platform provided in this embodiment may further include:

[0089] The target task is distributed to multiple target computing nodes based on the mapping information.

[0090] The mapping information can be the mapping information determined in S504, or it can be the mapping information determined according to the user's modification instructions.

[0091] Since the mapping information indicates the correspondence between hot datasets and multiple target computing nodes, the target tasks are distributed to multiple target computing nodes according to the mapping information, which further improves the memory hit rate of the computing server, further improves the data access speed, and improves the overall computing performance of the big data platform.

[0092] Optionally, the computing server can interact with electronic devices, displaying task distribution results to the user through the electronic device's interface. This embodiment does not limit the layout of the interface displayed on the electronic device. For example, see... Figure 7 or Figure 8 In the cache orchestration interface shown, users can click on task execution control 75 to view the task distribution results.

[0093] Optionally, the data processing method based on a big data platform provided in this embodiment may further include, before determining the multiple first datasets of the target task in S501:

[0094] Retrieve multiple SQL tasks corresponding to the target business.

[0095] Multiple SQL tasks are divided into multiple target tasks. Each target task includes a subset of the SQL tasks from the multiple SQL tasks, and the SQL tasks included in different target tasks do not overlap.

[0096] In this embodiment, there are no limitations on the number of SQL tasks corresponding to the target business, the number of target tasks corresponding to the target business, or the number of SQL tasks included in each target task. For example, suppose a big data user profiling business corresponds to 80 SQL tasks, and assume that each target task includes the same number of SQL tasks (4). Then there are a total of 80 / 4 = 20 target tasks. The 4 SQL tasks included in different target tasks are all different.

[0097] Optionally, the target task includes multiple SQL tasks. In S501, the multiple first datasets for determining the target task may include:

[0098] Retrieve the first historical data within the first time period. The first historical data is used to indicate the first dataset corresponding to each SQL task in the target task.

[0099] The first dataset corresponding to each SQL task in the target task is determined based on the first historical data.

[0100] In this embodiment, the value of the first time period is not limited; the value of the first time period can vary depending on the specific big data-based business. For example, if the processing cycle of the big data-based business is short, the first time period can be one day. If the processing cycle of the big data-based business is long, the first time period can be one week or one month.

[0101] Typically, in big data platforms, data storage locations follow a pattern; data for the same business is stored in fixed folders within fixed locations. Determining the primary dataset for the target task using historical data is both accurate and reasonable.

[0102] Optionally, in S501, determining multiple first datasets for the target task may include:

[0103] Multiple initial datasets of the target task are obtained through electronic devices by user input.

[0104] In this implementation, users can manually input the first dataset for executing the target task based on the actual situation. The obtained first dataset is more accurate and more in line with the actual business.

[0105] Optionally, in S502, identifying hotspot datasets among multiple first datasets may include:

[0106] For each first dataset corresponding to each SQL task in the target task, the block heat of the first dataset relative to the SQL task is determined based on the IO rate, the number of times the SQL task reads the first dataset, and the size of the first dataset.

[0107] For each first dataset, the sum of the block heat of the first dataset relative to all SQL tasks in the target task is determined as the file heat of the first dataset.

[0108] Hotspot datasets are determined based on the file popularity corresponding to multiple first datasets.

[0109] The following is combined Figure 6 An example is provided. Figure 6 As shown, the first dataset for SQL1 includes path1, path2, and path3. For path1 corresponding to SQL1, the block heat of path1 relative to SQL1 is determined based on the IO rate, the number of times SQL1 reads path1, and the size of path1, denoted as Q(path1, SQL1). Similarly, the block heat of path2 relative to SQL1 can be determined based on the IO rate, the number of times SQL1 reads path2, and the size of path2, denoted as Q(path2, SQL1). This process continues, resulting in the following block heat values, labeled as: Q(path1, SQL1), Q(path2, SQL1), Q(path3, SQL1), Q(path2, SQL2), Q(path3, SQL2), Q(path3, SQL3), Q(path4, SQL3), Q(path4, SQL4), Q(path5, SQL4), and Q(path6, SQL4).

[0110] For the first dataset path1, the sum of the block heat of path1 relative to all SQL tasks in the target task is determined as the file heat of path1, that is, the file heat of path1 = Q(path1, SQL1).

[0111] For the second dataset path2, the sum of the block heat of path2 relative to all SQL tasks in the target task is determined as the file heat of path2, that is, the file heat of path2 = Q(path2, SQL1) + Q(path2, SQL2).

[0112] And so on. The file popularity of path3 = Q(path3, SQL1) + Q(path3, SQL2) + Q(path3, SQL3). The file popularity of path4 = Q(path4, SQL3) + Q(path4, SQL4). The file popularity of path5 = Q(path5, SQL4). The file popularity of path6 = Q(path6, SQL4).

[0113] Finally, the hotspot dataset is determined based on the file popularity corresponding to path1 to path6 respectively.

[0114] Optionally, the block heat of the first dataset relative to the SQL task can be determined based on the IO rate, the number of times the SQL task reads the first dataset, and the size of the first dataset. This can include:

[0115] Based on the formula: Block heat = (Size of the first dataset / IO rate) * Number of times the SQL task reads the first dataset, the block heat of the first dataset relative to the SQL task is determined.

[0116] Optionally, the IO rate can be the average IO rate over a preset time period, or the IO rate determined based on historical data, or a preset IO rate.

[0117] Optionally, in one implementation, determining the hotspot dataset based on the file popularity corresponding to multiple first datasets may include:

[0118] The first dataset whose file popularity exceeds a preset threshold is identified as the hot dataset.

[0119] In this embodiment, the value of the preset threshold is not limited.

[0120] Optionally, in another implementation, determining the hotspot dataset based on the file popularity corresponding to multiple first datasets may include:

[0121] Sort multiple first datasets in descending order of file popularity, and determine the top M first datasets in the sorted list as hot datasets.

[0122] Where M is an integer greater than 1, and this embodiment does not limit the value of M.

[0123] Optionally, in S503, identifying multiple target computing nodes in the big data platform may include:

[0124] Obtain the performance parameters of computing nodes in the big data platform.

[0125] Multiple target computing nodes are determined based on the performance parameters of the computing nodes.

[0126] The performance parameters are used to reflect the computing performance of the computing nodes. Optionally, the performance parameters may include at least one of the following: central processing unit (CPU) utilization, CPU load, system idle time, memory utilization, or I / O rate.

[0127] In this implementation, multiple target computing nodes with good performance can be obtained based on the performance parameters of computing nodes in the big data platform, which improves the rationality and accuracy of determining target computing nodes.

[0128] Optionally, in S504, storing the hotspot dataset in the memory of multiple target computing nodes may include:

[0129] Get the file classification levels.

[0130] The hotspot dataset is divided into multiple file packages based on the preset file size corresponding to the file division level.

[0131] Multiple file packages are stored in the memory of multiple target computing nodes.

[0132] Specifically, each file partitioning level corresponds to a preset file size, which is used to divide the hot data set into multiple file packages. This embodiment does not limit the number of file partitioning levels or the preset file size for each level. For example, suppose there are three file partitioning levels, denoted as 0, 1, and 2. The preset file size for file partitioning level 0 is 100MB, for file partitioning level 1 it is 500MB, and for file partitioning level 2 it is 1GB.

[0133] See Figure 6 and Figure 9Suppose that hotspot dataset path1 is 100MB, hotspot dataset path2 is 250MB, hotspot dataset path4 is 350MB, and hotspot dataset path6 is 400MB, with a file partitioning level of 0 and a default file size of 100MB. Then, hotspot datasets path1, path2, path4, and path6 will be divided into 11 100MB file packages. It should be noted that when the size of the hotspot dataset is not an integer multiple of the default file size, you can first partition according to the default file size, with the last file package being smaller than the default file size. Alternatively, you can first determine the number of file packages based on the size of the hotspot dataset and the default file size, and then divide the hotspot dataset evenly based on the number of file packages.

[0134] By dividing the hot dataset into multiple file packages and storing these packages in the memory of multiple target computing nodes, the unbalanced load caused by the excessive size of the hot dataset is avoided. The hot dataset is distributed as evenly as possible across multiple target computing nodes, achieving load balancing and making full use of the computing resources of each computing server, thereby further improving the overall computing performance of the big data platform.

[0135] Optionally, obtaining file partition levels may include:

[0136] Obtain the percentage of files stored on the big data platform within multiple preset file size ranges during the second time period.

[0137] Get the file classification level corresponding to the preset file size range with the largest proportion.

[0138] In this embodiment, there are no limitations on the value of the second time period, the number of preset file size ranges, or the value of each preset file size range. For example, suppose the second time period is one month. There are three preset file size ranges: 10MB–100MB, 100MB–500MB, and >500MB. The preset file size range of 10MB–100MB corresponds to file classification level 0, the preset file size range of 100MB–500MB corresponds to file classification level 1, and the preset file size range >500MB corresponds to file classification level 2.

[0139] Assuming that within a month, 60% of the files stored on the big data platform are between 10MB and 100MB, 30% are between 100MB and 500MB, and 10% are >500MB, then the largest proportion is 60% corresponding to 10MB to 100MB, and the file classification level is 0.

[0140] It is evident that determining the file classification level based on the proportion of files stored on the big data platform within multiple preset file size ranges during the second time period allows the size of the divided file packages to be as similar as possible to the size of the files stored on the big data platform, further improving the overall computing performance of the big data platform.

[0141] Optionally, storing multiple file packages in the memory of multiple target computing nodes may include:

[0142] A three-replica mechanism is used to store multiple file packages in the memory of multiple target computing nodes.

[0143] The three-replica mechanism involves copying a data block twice, creating three copies of each other. These copies are then stored on different nodes within the big data platform according to a pre-defined distributed storage algorithm. By employing this three-replica mechanism to store multiple file packages in the memory of multiple target computing nodes, the failure of a single computing server will not affect the overall business operations, thus improving the computing stability of the big data platform.

[0144] It is understood that, in order to achieve the above-mentioned functions, data processing devices or servers based on big data platforms include hardware and / or software modules corresponding to the execution of each function. Based on the algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0145] This application embodiment can divide a data processing device or server based on a big data platform into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation. It should also be noted that the module names in this application embodiment are illustrative, and the names of the modules are not limited in actual implementation.

[0146] Figure 10 This is a schematic diagram of the structure of a data processing device based on a big data platform provided in an embodiment of this application. The data processing device based on a big data platform provided in this embodiment can be applied to a server and can be used for… Figure 4 The specific implementation of the cache orchestration engine in the software architecture shown. For example... Figure 10As shown, a data processing device based on a big data platform may include:

[0147] Hotspot analysis module 1001 is used to determine multiple first datasets for the target task, and to identify hotspot datasets among the multiple first datasets; the first datasets are datasets remotely read when the target task is executed;

[0148] The node orchestration module 1002 is used to determine multiple target computing nodes in the big data platform;

[0149] The preheating module 1003 is used to store the hotspot dataset into the memory of the multiple target computing nodes and generate mapping information; the mapping information is used to indicate the correspondence between the hotspot dataset and the multiple target computing nodes.

[0150] Optionally, it also includes a cache orchestration module 1005, which is used for:

[0151] The mapping information is displayed to the user via an electronic device.

[0152] Optionally, it also includes a cache orchestration module 1005, which is used for:

[0153] The modification instructions are obtained by acquiring user input through an electronic device. The modification instructions are used to modify the correspondence between the hotspot dataset indicated by the mapping information and the multiple target computing nodes.

[0154] Optionally, it also includes a task distribution module 1004, which is used for:

[0155] The target task is distributed to the multiple target computing nodes based on the mapping information.

[0156] Optionally, the preheating module 1003 is used for:

[0157] Get the file classification levels;

[0158] The hotspot dataset is divided into multiple file packages based on the preset file size corresponding to the file division level.

[0159] The multiple file packages are stored in the memory of the multiple target computing nodes.

[0160] Optionally, the preheating module 1003 is used for:

[0161] Obtain the percentage of files stored on the big data platform within multiple preset file size ranges during the second time period;

[0162] Get the file classification level corresponding to the preset file size range with the largest proportion.

[0163] Optionally, the preheating module 1003 is used for:

[0164] A three-replica mechanism is used to store the multiple file packages in the memory of the multiple target computing nodes.

[0165] Optionally, the target task includes multiple SQL tasks, and the hotspot analysis module 1001 is used for:

[0166] Obtain the first historical data within the first time period; the first historical data is used to indicate the first dataset corresponding to each SQL task in the target task;

[0167] Based on the first historical data, determine the first dataset corresponding to each SQL task in the target task.

[0168] Optionally, the hotspot analysis module 1001 is used for:

[0169] For each first dataset corresponding to each SQL task in the target task, the block heat of the first dataset relative to the SQL task is determined based on the input / output IO rate, the number of times the SQL task reads the first dataset, and the size of the first dataset.

[0170] For each of the first datasets, the sum of the block heat of the first dataset relative to all SQL tasks in the target task is determined as the file heat of the first dataset.

[0171] The hotspot dataset is determined based on the file popularity corresponding to the multiple first datasets.

[0172] Optionally, the hotspot analysis module 1001 is used for:

[0173] The first dataset whose file popularity exceeds a preset threshold is identified as the hot dataset.

[0174] Optionally, the node orchestration module 1002 is used for:

[0175] Obtain the performance parameters of the computing nodes in the big data platform;

[0176] The plurality of target computing nodes are determined based on the performance parameters of the computing nodes.

[0177] Optionally, the performance parameters include at least one of the following: CPU utilization, CPU load, system idle time, memory utilization, or input / output I / O rate.

[0178] Optionally, the hotspot analysis module 1001 is further used for:

[0179] Before determining the multiple first datasets for the target task, obtain multiple SQL tasks corresponding to the target business;

[0180] The plurality of SQL tasks are divided into a plurality of target tasks; the target tasks include some of the SQL tasks in the plurality of SQL tasks, and the SQL tasks included in different target tasks do not overlap.

[0181] The data processing device based on a big data platform provided in this embodiment is used to execute the data processing method based on a big data platform provided in the above method embodiment. The technical principles and technical effects are similar, and will not be described again here.

[0182] Please refer to Figure 11 This illustration shows the structure of a device provided in an embodiment of this application. The device can be a server in a big data platform, specifically a computing server. The device includes: a processor 1101, a communication port 1102, a memory 1103, and a bus 1104. The processor 1101 includes one or more processing cores. The processor 1101 executes various application and information processing functions by running software programs and modules. The communication port 1102 can be implemented as a communication component, which can be a chip. The memory 1103 is connected to the processor 1101 via the bus 1104. The memory 1103 can be used to store at least one program instruction, and the processor 1101 is used to execute at least one program instruction to implement the technical solution of the above embodiment. Its implementation principle and technical effects are similar to those of the related embodiments of the above method, and will not be repeated here.

[0183] Those skilled in the art will understand that, for ease of explanation, Figure 11 Only one memory and processor are shown. In actual devices, multiple processors and memories may exist. Memory may also be referred to as storage medium or storage device, etc., and this application does not limit this.

[0184] In the embodiments of this application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0185] In the embodiments of this application, the memory can be non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SS), or it can be volatile memory, such as random-access memory (RAM). Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, and is not limited thereto.

[0186] The memory in the embodiments of this application can also be a circuit or any other device capable of performing storage functions, used to store program instructions and / or data. The methods provided in the various embodiments of this application can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DWD), or semiconductor media (e.g., SSD), etc.

[0187] This application provides a computer program product that, when run on a device, causes the device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here. The device can be a server in a big data platform.

[0188] This application provides a computer-readable storage medium storing program instructions. When executed by a device, the program instructions cause the device to perform the technical solution described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here. The device can be a server in a big data platform.

[0189] In summary, the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method based on a big data platform, characterized in that, include: Determine multiple first datasets for the target task, wherein the first datasets are datasets remotely read when the target task is executed; Identify hotspot datasets from the plurality of first datasets; Identify multiple target computing nodes within the big data platform; The process of storing the hotspot dataset in the memory of the multiple target computing nodes and generating mapping information includes: obtaining file partitioning levels; partitioning the hotspot dataset according to the preset file size corresponding to the file partitioning levels to obtain multiple file packages; storing the multiple file packages in the memory of the multiple target computing nodes; and the mapping information is used to indicate the correspondence between the hotspot dataset and the multiple target computing nodes. The mapping information is displayed to the user via an electronic device.

2. The method according to claim 1, characterized in that, Also includes: The modification instructions are obtained by acquiring user input through an electronic device. The modification instructions are used to modify the correspondence between the hotspot dataset indicated by the mapping information and the multiple target computing nodes.

3. The method according to claim 1, characterized in that, Also includes: The target task is distributed to the multiple target computing nodes based on the mapping information.

4. The method according to claim 1, characterized in that, The process of obtaining file classification levels includes: Obtain the percentage of files stored on the big data platform within multiple preset file size ranges during the second time period; Get the file classification level corresponding to the preset file size range with the largest proportion.

5. The method according to claim 1, characterized in that, The step of storing the plurality of file packages into the memory of the plurality of target computing nodes includes: A three-replica mechanism is used to store the multiple file packages in the memory of the multiple target computing nodes.

6. The method according to any one of claims 1-3, characterized in that, The target task includes multiple Structured Query Language (SQL) tasks, and the multiple first datasets used to determine the target task include: Obtain the first historical data within the first time period; the first historical data is used to indicate the first dataset corresponding to each SQL task in the target task; Based on the first historical data, determine the first dataset corresponding to each SQL task in the target task.

7. The method according to claim 6, characterized in that, The step of identifying hotspot datasets among the plurality of first datasets includes: For each first dataset corresponding to each SQL task in the target task, the block heat of the first dataset relative to the SQL task is determined based on the input / output IO rate, the number of times the SQL task reads the first dataset, and the size of the first dataset. For each of the first datasets, the sum of the block heat of the first dataset relative to all SQL tasks in the target task is determined as the file heat of the first dataset. The hotspot dataset is determined based on the file popularity corresponding to the multiple first datasets.

8. The method according to claim 7, characterized in that, The step of determining the hotspot dataset based on the file popularity corresponding to the plurality of first datasets includes: The first dataset whose file popularity exceeds a preset threshold is identified as the hot dataset.

9. The method according to any one of claims 1-3, characterized in that, The determination of multiple target computing nodes in the big data platform includes: Obtain the performance parameters of the computing nodes in the big data platform; The plurality of target computing nodes are determined based on the performance parameters of the computing nodes.

10. The method according to claim 9, characterized in that, The performance parameters include at least one of the following: CPU utilization, CPU load, system idle time, memory usage, or input / output I / O rate.

11. The method according to any one of claims 1-3, characterized in that, Before determining the multiple first datasets for the target task, the method further includes: Obtain multiple Structured Query Language (SQL) tasks corresponding to the target business; Multiple SQL tasks are divided into multiple target tasks; each target task includes a portion of the SQL tasks among the multiple SQL tasks, and the SQL tasks included in different target tasks do not overlap.

12. A data processing device based on a big data platform, characterized in that, include: The hotspot analysis module is used to identify multiple first datasets for the target task and to identify hotspot datasets among the multiple first datasets. The first dataset is a dataset remotely read when the target task is executed; The node orchestration module is used to determine multiple target computing nodes in the big data platform; The preheating module is used to store the hotspot dataset into the memory of the multiple target computing nodes and generate mapping information; The mapping information is used to indicate the correspondence between the hotspot dataset and the multiple target computing nodes; the preheating module is used to: obtain file partitioning levels; The hotspot dataset is divided into multiple file packages according to the preset file size corresponding to the file division level; the multiple file packages are then stored in the memory of the multiple target computing nodes. The data processing device based on the big data platform further includes a cache orchestration module, which is used for: The mapping information is displayed to the user via an electronic device.

13. The apparatus according to claim 12, characterized in that, It also includes a cache orchestration module, which is used for: The modification instructions are obtained by acquiring user input through an electronic device. The modification instructions are used to modify the correspondence between the hotspot dataset indicated by the mapping information and the multiple target computing nodes.

14. The apparatus according to claim 12, characterized in that, It also includes a task distribution module, which is used for: The target task is distributed to the multiple target computing nodes based on the mapping information.

15. The apparatus according to claim 12, characterized in that, The preheating module is used for: Obtain the percentage of files stored on the big data platform within multiple preset file size ranges during the second time period; Get the file classification level corresponding to the preset file size range with the largest proportion.

16. The apparatus according to claim 12, characterized in that, The preheating module is used for: A three-replica mechanism is used to store the multiple file packages in the memory of the multiple target computing nodes.

17. The apparatus according to any one of claims 12-14, characterized in that, The target task includes multiple Structured Query Language (SQL) tasks, and the hotspot analysis module is used for: Obtain the first historical data within the first time period; the first historical data is used to indicate the first dataset corresponding to each SQL task in the target task; Based on the first historical data, determine the first dataset corresponding to each SQL task in the target task.

18. The apparatus according to claim 17, characterized in that, The hotspot analysis module is used for: For each first dataset corresponding to each SQL task in the target task, the block heat of the first dataset relative to the SQL task is determined based on the input / output IO rate, the number of times the SQL task reads the first dataset, and the size of the first dataset. For each of the first datasets, the sum of the block heat of the first dataset relative to all SQL tasks in the target task is determined as the file heat of the first dataset. The hotspot dataset is determined based on the file popularity corresponding to the multiple first datasets.

19. The apparatus according to claim 18, characterized in that, The hotspot analysis module is used for: The first dataset whose file popularity exceeds a preset threshold is identified as the hot dataset.

20. The apparatus according to any one of claims 12-14, characterized in that, The node orchestration module is used for: Obtain the performance parameters of the computing nodes in the big data platform; The plurality of target computing nodes are determined based on the performance parameters of the computing nodes.

21. The apparatus according to claim 20, characterized in that, The performance parameters include at least one of the following: CPU utilization, CPU load, system idle time, memory usage, or input / output I / O rate.

22. The apparatus according to any one of claims 12-14, characterized in that, The hotspot analysis module is also used for: Before determining the multiple first datasets for the target task, obtain multiple Structured Query Language (SQL) tasks corresponding to the target business; Multiple SQL tasks are divided into multiple target tasks; each target task includes a portion of the SQL tasks among the multiple SQL tasks, and the SQL tasks included in different target tasks do not overlap.

23. A big data platform server, characterized in that, The system includes a processor configured to be coupled to a memory, and to read instructions from the memory and, according to the instructions, cause the big data platform server to perform the method as described in any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on the device, cause the device to perform the method as described in any one of claims 1-11.

Images