Data processing method, online platform and big data platform

Abstract

The embodiment of the invention provides a data processing method, an online platform and a big data platform, which can continuously acquire column storage data from a storage device and convert the column storage data into row storage data, so that the row storage data is directly scanned from a local cache, and the service processing efficiency of the online platform is improved. The method comprises the steps that after an online platform generates a physical plan according to an online service request, corresponding column storage data are obtained from a storage device according to an IO task list of the physical plan, after the column storage data are converted into row storage data, the row storage data are stored in a local cache, then target row storage data are obtained from the local cache by using a scanning operator of the physical plan, and the target row storage data are stored in the storage device. And executing the online service according to the target row storage data.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a data processing method, an online platform and a big data platform. Background Art

[0002] Big data platforms include online and offline platforms. The online platform is used to execute actual business operations, while the offline platform is used for data cleansing. However, the data formats of the online and offline platforms are incompatible.

[0003] Currently, there is a data processing method that is roughly as follows: after the online platform receives an online business request, it generates a physical plan based on the online business request, obtains column-stored data from the storage device according to the scanning operator of the physical plan, converts the column-stored data into row-stored data, and processes the row-stored data according to other operators of the physical plan to obtain online business data.

[0004] However, when using the scanning operator to scan multiple partitions, after scanning the current partition, you have to wait until other operators are processed before scanning the next partition. This slows the scanning speed and makes the online platform's business processing efficiency low. Summary of the Invention

[0005] This application provides a data processing method that pre-reads column-stored data from a storage device before scanning the column-stored data and converts the column-stored data into row-stored data. This method enables scanning the row-stored data directly from a local cache, thereby increasing scanning speed and, in turn, improving the business processing efficiency of an online platform. This application also provides a corresponding online platform, a big data platform, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0006] A first aspect provides a data processing method, which includes: after an online platform generates a physical plan based on an online business request, extracting an input and output (IO) task list from the physical plan, and then sending an asynchronous IO request to a storage device based on the IO task list. After the storage device obtains the IO task list based on the asynchronous IO request, the online platform receives column-stored data corresponding to the IO task list sent by the storage device, converts the column-stored data corresponding to the IO task list into row-stored data corresponding to the IO task list and saves the row-stored data in a local cache. The online platform obtains target row-stored data from the row-stored data in the local cache based on a scan operator of the physical plan, and executes the online business based on the target row-stored data.

[0007] The IO task list includes multiple partitioned IO tasks, and the online platform sends asynchronous IO requests to the storage device accordingly, thereby continuously reading the column-stored data of multiple partitions without waiting for other operators to complete execution. This can significantly improve the efficiency of obtaining column-stored data. After obtaining the column-stored data, the column-stored data can be pre-converted into row-stored data. Therefore, the online platform can scan the row-stored data directly from the local cache, which is faster than the scanning speed of existing methods. Because asynchronous IO requests can be sent continuously, the online platform does not need to send the next IO request after receiving the column-stored data corresponding to an IO request, which can reduce the resource overhead occupied by IO requests.

[0008] In one possible implementation, the online platform saves the row-stored data corresponding to the partitioned IO task in the local cache and then clears the column-stored data corresponding to the partitioned IO task. After the online platform caches the row-stored data corresponding to the partitioned IO task, it no longer needs to use the column-stored data corresponding to the partitioned IO task. Clearing the column-stored data corresponding to the partitioned IO task can reduce redundant data and lower the online platform's storage overhead.

[0009] In another possible implementation, the online platform converts the column-stored data corresponding to the IO task list into the row-stored data corresponding to the IO task list, including: obtaining a predicate list after predicate pushdown, filtering the column-stored data corresponding to the IO task list based on the predicate list, and then converting the filtered column-stored data into the row-stored data corresponding to the IO task list. Using the predicate list after predicate pushdown can filter out unnecessary data from the column-stored data, thereby reducing the amount of data required for other operators to compute. This improves the execution efficiency of the physical plan and increases business processing speed.

[0010] In another possible implementation, metadata is received from a storage device, and an index for the row-stored data is created based on the metadata. A physical plan is generated based on the online service request and the index for the row-stored data. After the offline platform cleans the column-stored data, the online platform can obtain the corresponding metadata for the cleansed column-stored data and then create an index for the row-stored data based on the metadata. This allows the index for the row-stored data to be automatically created after the data is cleansed. A physical plan is generated based on the index for the row-stored data and the online service request. Executing the physical plan can obtain the online service data, enabling online services to be executed based on the cleansed column-stored data.

[0011] In another possible implementation, the target row data is the row data corresponding to the IO task list. This allows all row data corresponding to the IO task list to be scanned from the local cache, significantly improving the execution efficiency of the physical plan.

[0012] In another possible implementation, the data processing method of the present application also includes: when the scan operator of the physical plan is used in the local cache and the row storage data corresponding to the target IO task is not found, an IO request is sent to the storage device according to the target IO task, and after the storage device obtains the column storage data corresponding to the target IO task according to the IO request, the online platform receives the column storage data corresponding to the target IO task sent by the storage device, converts the column storage data corresponding to the target IO task into the row storage data corresponding to the target IO task, and then executes the online business according to the row storage data corresponding to the target IO task and the target row storage data. The target IO task can be any IO task in the IO task list, the number of target IO tasks can be one or more, and the target row storage data includes the row storage data corresponding to the non-target IO tasks in the IO task list. In this way, the row storage data corresponding to some IO tasks can be scanned out from the local cache, and the row storage data corresponding to other IO tasks can be read from the storage device, which can also significantly improve the execution efficiency of the physical plan.

[0013] In another possible implementation, the format of the column-stored data is an optimized rowcolumnar (ORC) format, and the format of the row-stored data and the format of the target row-stored data are both a column unit (CU) format.

[0014] The second aspect provides an online platform, which includes a planning module, an extraction module, a pre-reading module, a pre-conversion module, a cache module and an execution module. The planning module is used to generate a physical plan according to an online business request; the extraction module is used to extract an IO task list from the physical plan; the pre-reading module is used to send an asynchronous IO request to a storage device according to the IO task list; receive column-stored data corresponding to the IO task list sent by the storage device; the pre-conversion module is used to convert the column-stored data corresponding to the IO task list into row-stored data corresponding to the IO task list; the cache module is used to save the row-stored data corresponding to the IO task list in a local cache; the execution module is used to obtain target row-stored data from the local cache according to a scan operator of the physical plan, and execute online business according to the target row-stored data.

[0015] In another possible implementation, after the cache module saves the row-stored data corresponding to the partitioned IO task in the local cache, the cache module is further configured to clear the column-stored data corresponding to the partitioned IO task.

[0016] In another possible implementation, the pre-conversion module is specifically used to obtain a predicate list after predicate pushdown; filter the column-stored data corresponding to the IO task list according to the predicate list; and convert the filtered column-stored data into row-stored data corresponding to the IO task list.

[0017] In another possible implementation, the online platform also includes an indexing module, which is further used to receive metadata sent by the storage device and create an index of the row-stored data based on the metadata; the planning module is specifically used to generate a physical plan based on the online business request and the index of the row-stored data.

[0018] In another possible implementation, the online platform also includes a communication module and a conversion module. The communication module is used to send an IO request to the storage device according to the target IO task and receive the column-stored data corresponding to the target IO task sent by the storage device when the scan operator of the physical plan fails to find the row-stored data corresponding to the target IO task in the local cache; the conversion module is used to convert the column-stored data corresponding to the target IO task into the row-stored data corresponding to the target IO task; and the execution module is specifically used to execute online services based on the row-stored data corresponding to the target IO task and the target row-stored data.

[0019] For the explanation of terms in the second aspect, the specific steps and beneficial effects of each module, please refer to the corresponding description of the first aspect.

[0020] The third aspect provides a big data platform, which includes an offline platform, a storage device, and the online platform of the second aspect.

[0021] In one possible implementation, the offline platform cleans the column-stored data, generates metadata based on the cleansed column-stored data, sends the cleansed column-stored data and metadata to a storage device, and stores the cleansed column-stored data and metadata. The storage device also sends the metadata to the online platform, which then creates an index for the row-stored data based on the metadata. After the offline platform cleans the column-stored data, the online platform can use the data to create an index for the row-stored data, facilitating access to the cleansed data.

[0022] The fourth aspect provides a computing device cluster, which includes at least one computing device, the at least one computing device including a processor and a memory, and instructions are stored in the memory of the at least one computing device; the processor of the at least one device executes the instructions so that the computing device cluster executes the method described in the first aspect or any one implementation of the first aspect.

[0023] A fifth aspect provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method as described in the first aspect or any one of the implementations of the first aspect.

[0024] A sixth aspect provides a computer program product comprising instructions, which, when executed by a computing device cluster, causes the computing device cluster to execute the method described in the first aspect or any one of the implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of an application scenario in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of an existing data processing method;

[0027] Figure 3 A schematic diagram of a data processing method in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a data processing method in an embodiment of the present application;

[0029] Figure 5 This is a schematic diagram of the online platform in the embodiment of the present application;

[0030] Figure 6 This is a schematic diagram of the big data platform in the embodiment of this application;

[0031] Figure 7 A structural diagram of a computing device in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of a computing device cluster in an embodiment of the present application;

[0033] Figure 9 A schematic diagram of a computing device cluster in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The data processing method of this application can be applied to the online platform of a big data platform. In one embodiment, the big data platform includes an online platform, an offline platform, and a storage cluster. The storage cluster includes multiple storage devices. The storage devices include, but are not limited to, storage servers and disk arrays.

[0035] The online platform is used to execute actual business operations and can be, but is not limited to, a transaction table management platform. The offline platform is used for data cleansing and can be, but is not limited to, an analytical table management platform. The online platform processes row-based data, while the offline platform processes column-based data. Consequently, data formats between the online and offline platforms may be incompatible. Row-based data is organized and stored in rows, with each row representing a single record. Column-based data is organized and stored in columns, with each column representing a single record.

[0036] To solve this problem, data format conversion is required, also known as data fusion. The following is an introduction to the data fusion process. Figure 1 In one embodiment, the offline platform's read module reads ORC files from the storage cluster. The big data component Hive can clean the ORC files and then send the cleaned ORC files to the storage cluster via the write module. Furthermore, the offline platform's write module can generate metadata and send the metadata to the storage cluster. The storage cluster can send the metadata to the online platform, which stores the metadata in the index cache and creates an index (i.e., a B+ tree index) based on the row-stored data. The metadata can be, but is not limited to, CU metadata. In the online platform, metadata can be represented by the descriptive information (i.e., CUDesc) of the row-stored data.

[0037] After the online platform receives an online business request, the first reading module reads the row data from the row data cache based on the online business request and the index of the row data. If the row data corresponding to the online business request is not found in the row data cache, the second reading module reads the column data from the storage cluster, uses the format conversion module to convert the column data into row data, and writes the row data into the row data cache. After the first reading module reads the row data from the row data cache, the online database (such as the opengauss database) performs database operations on the row data to obtain the online business data. Optionally, the writing module of the online platform can send the row data to the storage cluster.

[0038] The following describes the process of converting formats on the big data platform in existing data processing methods. Figure 2 , the online platform generates database statements based on online business requests, the planning module can generate an optimized logical plan based on the database statements, generate a physical plan based on the optimized logical plan, and then execute the physical plan through the execution module. Specifically, the physical plan includes multiple operators, such as scan operators, join operators, and window operators. The communication module reads column-stored data from the column-stored files of the storage cluster, the conversion module converts the column-stored data into row-stored data, and processes the row-stored data through other operators (such as join operators and window operators) to obtain the execution results of the physical plan. The join operator can merge multiple data sets based on a specified key, and the window operator is used to split the data according to windows (such as time windows). It should be understood that the operators of this application are not limited to the above examples.

[0039] After the online platform uses the scan operator to scan a partition, it needs to wait for other operators to process it, which leads to low execution efficiency of the physical plan. To address this problem, this application provides a data processing method that can continuously read column-stored data from multiple partitions without waiting for other operators to calculate. In this way, the online platform can use the scan operator to scan row-stored data, which can greatly improve the efficiency of obtaining column-stored data and thus improve business processing efficiency.

[0040] See Figure 3 In one embodiment, after the online platform of the present application receives a big data request, it generates a database statement according to the online business request. The planning module can generate an optimized logical plan according to the database statement, generate a physical plan according to the optimized logical plan, and then execute the physical plan through the execution module. After the physical plan is generated, the extraction module of the online platform can extract the IO task list from the physical plan, send the IO task list to the pre-reading module, the pre-reading module sends an IO request to the storage device according to the IO task list, the storage device responds to the IO request, and feeds the column-stored data back to the pre-reading module. The pre-conversion module converts the column-stored data sent by the pre-reading module into row-stored data, and then stores the row-stored data in the cache module. When the execution module uses the scan operator to first scan the row-stored data of the cache module, when the cache is hit, the row-stored data is read from the cache module. When the cache is not hit, the communication module reads the column-stored data from the storage device, and then the conversion module converts the column-stored data into row-stored data.

[0041] Below Figure 3 For a detailed description of the data processing method shown in Figure 4 , an embodiment of the data processing method of the present application includes the following steps:

[0042] S401: The online platform generates a physical plan according to the online business request.

[0043] In this embodiment, online service requests include but are not limited to query requests. The online platform generates database statements based on the online service requests. After performing syntactic and semantic parsing on the database statements, it generates a logical plan based on the parsing results. After optimizing the logical plan, a physical plan is generated based on the optimized logical plan. Optimizing the logical plan is optional. The physical plan includes various operators, such as scan operators, join operators, and window operators. Each operator can be one or more. It should be understood that the operators in the physical plan are not limited to the examples listed above.

[0044] S402: The online platform extracts an IO task list from the physical plan. The IO task list includes multiple partition IO tasks.

[0045] S403: The online platform sends an asynchronous IO request to the storage device according to the IO task list.

[0046] The IO request may be, but is not limited to, a coroutine async IO request. This allows multiple IO requests to be sent consecutively, eliminating the need to send the next IO request after receiving the column-stored data corresponding to one IO request. This improves the efficiency of obtaining column-stored data and reduces the computing resources occupied by IO requests.

[0047] S404: The storage device obtains column storage data corresponding to the IO task list according to the IO request.

[0048] S405: The storage device sends the column storage data corresponding to the IO task list to the online platform.

[0049] In response to the I / O requests sent by the partitioned I / O tasks, the online platform reads the column-stored data from the corresponding partitions and sends the column-stored data corresponding to each partitioned I / O task to the storage device. The online platform can continuously retrieve the column-stored data corresponding to multiple partitioned I / O tasks, enabling rapid access to the column-stored data from the storage device.

[0050] S406: The online platform converts the column-stored data corresponding to the IO task list into row-stored data corresponding to the IO task list.

[0051] S407: The online platform saves the row data corresponding to the IO task list in a local cache.

[0052] Specifically, the online platform stores the row-stored data corresponding to multiple partitioned I / O tasks in a local cache. Optionally, after storing the row-stored data in the local cache, the online platform clears the column-stored data corresponding to the partitioned I / O tasks. After caching the row-stored data for a partitioned I / O task, the column-stored data corresponding to that partitioned I / O task is no longer needed. Clearing the column-stored data for that partitioned I / O task can reduce redundant data and lower the online platform's storage overhead.

[0053] S408: The online platform uses the scan operator of the physical plan to obtain the target row storage data from the local cache.

[0054] Optionally, the target row data is the row data corresponding to the IO task list. Alternatively, the target row data is the row data corresponding to some IO tasks in the IO task list.

[0055] S409: The online platform executes online business according to the stored data of the target bank.

[0056] Specifically, the online platform uses other operators of the physical plan to perform operations on the target row data to obtain the execution results of the physical plan. The execution results of the physical plan can be used as online business data, or online business data can be generated based on the execution results of the physical plan.

[0057] In this embodiment, the online platform can continuously retrieve column-stored data from the storage device according to the IO task list without waiting for other operators to complete execution, significantly improving the efficiency of obtaining column-stored data. After pre-converting the column-stored data into row-stored data, scanning the row-stored data from the local cache is faster than conventional scanning, thereby improving business processing efficiency.

[0058] Secondly, the row-stored data in the online platform is converted from the column-stored data in the storage device, so the storage device does not need to store the row-stored data. Compared with the method of storing one copy of data in two formats, the method of this application can reduce the storage overhead by half.

[0059] In an optional embodiment, S406 includes: the online platform obtains a predicate list after the predicate is pushed down; the online platform filters the column-stored data corresponding to the IO task list according to the predicate list; and the online platform converts the filtered column-stored data into row-stored data corresponding to the IO task list.

[0060] In this embodiment, the online platform can remove unnecessary data from the column-stored data based on the predicates pushed down, thereby reducing the amount of data to be converted and improving the conversion speed. After performing the above conversion, the online platform uses the operators of the physical plan to process less data, thereby improving the execution efficiency of the physical plan.

[0061] In another optional embodiment, the data processing method of the present application further includes: when the scan operator of the physical plan fails to find the row storage data corresponding to the target IO task in the local cache, the online platform uses the scan operator to read the column storage data corresponding to the target IO task from the storage device; converting the column storage data corresponding to the target IO task into the row storage data corresponding to the target IO task, and executing the online business according to the row storage data corresponding to the target IO task and the target row storage data. The target IO task is any IO task in the IO task list, and the number of target IO tasks can be one or more. The target row storage data includes the row storage data corresponding to non-target IO tasks in the IO task list.

[0062] In this embodiment, when the network between the online platform and the storage device is congested, the online platform may not receive the column-stored data sent by the storage device. In this case, the online platform cannot pre-read the column-stored data or convert the column-stored data into row-stored data in advance. Alternatively, the online platform obtains the row-stored data corresponding to the target IO task later than scanning the row-stored data using the scan operator. In this case, the online platform does not pre-acquire the row-stored data corresponding to the target IO task. In the above case, the online platform can read the row-stored data corresponding to some IO tasks from the local cache and the column-stored data corresponding to other IO tasks from the storage device, and then convert the format to obtain all the row-stored data corresponding to the IO task list.

[0063] In another optional embodiment, the data processing method of the present application also includes: the offline platform cleans the column-stored data, and then generates metadata based on the cleaned column-stored data; the storage device can save the cleaned column-stored data and metadata and send the metadata to the online platform; the online platform can generate an index of the row-stored data based on the metadata, and generate a physical plan based on the online business request and the index of the row-stored data.

[0064] In this application, data cleaning includes data cleaning and / or data organization. After the offline platform cleans the column-stored data, the data content of the offline platform will be different from the data content of the online platform. In order to enable the online platform to access the column-stored data after data cleaning, the offline platform can generate metadata based on the column-stored data after data cleaning and save it in the storage device. After the online platform obtains the metadata from the storage device, it can respond to the index creation instruction input by the user, create an index for the row-stored data based on the metadata, and then generate a physical plan based on the index of the row-stored data and the online business request, and then obtain the online business data, so that online business can be performed based on the column-stored data after data cleaning. The metadata can be but not limited to CU metadata, and both the online platform and the offline platform can access the above-mentioned data after data cleaning based on the metadata.

[0065] See Figure 5 In one embodiment, the online platform of the present application includes: a planning module 501, an extraction module 502, a pre-reading module 503, a pre-conversion module 504, a cache module 505, an execution module 506, an indexing module 507, a communication module 508 and a conversion module 509.

[0066] In one embodiment, a planning module 501 is used to generate a physical plan based on an online business request; an extraction module 502 is used to extract an IO task list from the physical plan; a pre-reading module 503 is used to send an asynchronous IO request to a storage device based on the IO task list; and a column-stored data corresponding to the IO task list sent by the storage device is received; a pre-conversion module 504 is used to convert the column-stored data corresponding to the IO task list into the row-stored data corresponding to the IO task list; a cache module 505 is used to store the row-stored data corresponding to the IO task list in a local cache; and an execution module 506 is used to obtain target row-stored data from the local cache according to a scan operator of the physical plan and execute online business based on the target row-stored data.

[0067] It should be noted that, in other embodiments, the planning module 501 can be used to execute Figure 4 In any step performed by the online platform in the embodiment shown, the extraction module 502 can be used to perform Figure 4 In any step performed by the online platform in the embodiment shown, the pre-reading module 503 can be used to perform Figure 4In any step performed by the online platform in the embodiment shown, the pre-conversion module 504 can be used to perform Figure 4 In any step performed by the online platform in the embodiment shown, the cache module 505 can be used to perform Figure 4 In the embodiment shown, the execution module 506 can be used to execute any step performed by the online platform. Figure 4 In any step performed by the online platform in the embodiment shown, the index module 507 can be used to perform Figure 4 In any step performed by the online platform in the embodiment shown, the communication module 508 can be used to perform Figure 4 In any step performed by the online platform in the embodiment shown, the conversion module 509 can be used to perform Figure 4 For any steps executed by the online platform in the embodiment shown, the steps implemented by each module can be specified as needed, and the full functions of the online platform can be realized by implementing different steps in the data processing method through each module.

[0068] in addition, Figure 1 The function of the first reading module can be implemented by any one of the planning module 501, the extraction module 502, the pre-reading module 503, the pre-conversion module 504, the cache module 505 and the execution module 506; the function of the second reading module can be implemented by any one of the planning module 501, the extraction module 502, the pre-reading module 503, the pre-conversion module 504, the cache module 505 and the execution module 506; the function of the format conversion module can be implemented by any one of the planning module 501, the extraction module 502, the pre-reading module 503, the pre-conversion module 504, the cache module 505 and the execution module 506.

[0069] In an optional embodiment, after the cache module 505 stores the row-stored data corresponding to the partitioned IO task in the local cache, the cache module 505 is further configured to clear the column-stored data corresponding to the partitioned IO task.

[0070] In another optional embodiment, the pre-conversion module 504 is specifically configured to obtain a predicate list after predicate pushdown; filter the column-stored data corresponding to the IO task list according to the predicate list; and convert the filtered column-stored data into row-stored data corresponding to the IO task list.

[0071] In another optional embodiment, the index module 507 is used to receive metadata sent by the storage device and create an index of the row-stored data based on the metadata; the plan module 501 is specifically used to generate a physical plan based on the online business request and the index of the row-stored data.

[0072] In another optional embodiment, the communication module 508 is used to send an IO request to the storage device according to the target IO task and receive the column-stored data corresponding to the target IO task sent by the storage device when the scan operator of the physical plan fails to find the row-stored data corresponding to the target IO task in the local cache; the conversion module 509 is used to convert the column-stored data corresponding to the target IO task into the row-stored data corresponding to the target IO task; and the execution module 506 is specifically used to execute online business according to the row-stored data corresponding to the target IO task and the target row-stored data.

[0073] See Figure 6 , the present application provides a big data platform 600 , which includes an offline platform 601 , a storage device 602 and an online platform 500 .

[0074] In an optional embodiment, the offline platform 601 is used to clean the column-stored data and generate metadata based on the cleaned column-stored data; the cleaned column-stored data and metadata are sent to the storage device; the storage device 602 is used to save the cleaned column-stored data and metadata; the storage device 602 is also used to send metadata to the online platform; the online platform 500 is also used to create an index for the row-stored data based on the metadata.

[0075] The present application provides a computing device 700. Figure 7 In one embodiment, the computing device 700 of the present application includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 700.

[0076] The bus 702 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 704 may include a path for transmitting information between various components of the computing device 700 (eg, memory 706, processor 704, communication interface 708).

[0077] The processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0078] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory (NVM), such as ROM, flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0079] Memory 706 stores executable program code. Processor 704 executes the executable program code to implement the functions of the aforementioned planning module 501, extraction module 502, pre-reading module 503, pre-conversion module 504, cache module 505, execution module 506, indexing module 507, communication module 508, or conversion module 509, thereby implementing the data processing method. In other words, memory 706 stores instructions for executing the data processing method.

[0080] The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or a communication network.

[0081] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0082] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster may store the same instructions for executing the data processing method.

[0083] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 700 can jointly execute the instructions for executing the data processing method.

[0084] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 9 A possible implementation is shown. Figure 9 As shown, two computing devices 700A and 700B are connected via a network. Specifically, the connection to the network is achieved through a communication interface in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for executing the functions of the plan module 501, the extraction module 502, the pre-reading module 503, the pre-conversion module 504, the cache module 505, the execution module 506, the communication module 508, and the conversion module 509. Simultaneously, the memory 706 in computing device 700B stores instructions for executing the functions of the index module 507.

[0085] Figure 9 The connection mode between the computing device clusters shown may be considered to be that the data processing method provided by the present application requires a large amount of data storage, so the functions implemented by the storage device are considered to be handed over to the computing device 700B for execution. It should be understood that Figure 9 The functionality of the computing device 700A shown in FIG. 7 may also be implemented by multiple computing devices 700. Similarly, the functionality of the computing device 700B may also be implemented by multiple computing devices 700.

[0086] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a data processing method.

[0087] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to an online platform of a big data platform, the big data platform also including an offline platform and a storage device, the online platform is used to process row-stored data, and the storage device is used to store column-stored data, and the method includes: The online platform generates a physical plan based on the online business request; The online platform extracts an input and output IO task list from the physical plan, where the IO task list includes multiple partition IO tasks; The online platform sends an asynchronous IO request to the storage device according to the IO task list; The online platform receives the column-stored data corresponding to the IO task list sent by the storage device, where the column-stored data corresponding to the IO task list is obtained by the storage device according to the asynchronous IO request; The online platform converts the column-stored data corresponding to the IO task list into the row-stored data corresponding to the IO task list; The online platform stores the row data corresponding to the IO task list in a local cache; The online platform uses the scan operator of the physical plan to obtain target row data from the row data cached locally; The online platform executes online services according to the target row storage data.

2. The method according to claim 1, characterized in that The method further comprises: After the online platform saves the row-stored data corresponding to the partitioned IO task in the local cache, the online platform clears the column-stored data corresponding to the partitioned IO task.

3. The method according to claim 1 or 2, characterized in that The online platform converting the column-stored data corresponding to the IO task list into the row-stored data corresponding to the IO task list includes: The online platform obtains a predicate list after the predicates are pushed down; The online platform filters the column-stored data corresponding to the IO task list according to the predicate list; The online platform converts the filtered column-stored data into row-stored data corresponding to the IO task list.

4. The method according to any one of claims 1 to 3, characterized in that The method further includes: the online platform receiving metadata sent by the storage device, and creating an index of the row-stored data according to the metadata; The online platform generating a physical plan according to the online business request includes: the online platform generating a physical plan according to the online business request and the index of the row-stored data.

5. The method according to any one of claims 1 to 3, characterized in that The target row storage data is the row storage data corresponding to the IO task list.

6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When the scan operator of the physical plan fails to find the row-stored data corresponding to the target IO task in the local cache, the online platform sends an IO request to the storage device according to the target IO task, receives the column-stored data corresponding to the target IO task sent by the storage device; and converts the column-stored data corresponding to the target IO task into the row-stored data corresponding to the target IO task. The online platform executing online business according to the target row-stored data includes: the online platform executing online business according to the row-stored data corresponding to the target IO task and the target row-stored data, and the target row-stored data includes the row-stored data corresponding to the non-target IO task in the IO task list.

7. The method according to any one of claims 1 to 6, characterized in that The format of the column-stored data is an optimized row-column ORC format, and the format of the row-stored data and the format of the target row-stored data are both column unit CU formats.

8. An online platform, characterized in that include: Planning module, used to generate physical plans based on online business requests; An extraction module, configured to extract an input / output (IO) task list from the physical plan, the IO task list including a plurality of partitioned IO tasks; A pre-reading module, configured to send an asynchronous IO request to the storage device according to the IO task list; receiving column-stored data corresponding to the IO task list sent by the storage device, where the column-stored data corresponding to the IO task list is obtained by the storage device according to the asynchronous IO request; A pre-conversion module, configured to convert the column-stored data corresponding to the IO task list into the row-stored data corresponding to the IO task list; A cache module, configured to store the row data corresponding to the IO task list in a local cache; An execution module is configured to obtain target row-stored data from the local cache according to a scan operator of the physical plan, and execute an online business according to the target row-stored data.

9. The online platform according to claim 8, characterized in that After the cache module saves the row-stored data corresponding to the partitioned IO task in the local cache, the cache module is further configured to clear the column-stored data corresponding to the partitioned IO task.

10. The online platform according to claim 8 or 9, characterized in that: The pre-conversion module is specifically used to obtain a predicate list after predicate pushdown; filter the column-stored data corresponding to the IO task list according to the predicate list; and convert the filtered column-stored data into row-stored data corresponding to the IO task list.

11. The online platform according to any one of claims 8 to 10, characterized in that: The online platform also includes: An indexing module, configured to receive metadata sent by the storage device and create an index of the stored data according to the metadata; The planning module is specifically configured to generate a physical plan according to the online service request and the index of the row-stored data.

12. The online platform according to any one of claims 8 to 10, characterized in that: The target row storage data is the row storage data corresponding to the IO task list.

13. The online platform according to any one of claims 8 to 10, characterized in that: The online platform also includes: a communication module configured to, when the scan operator using the physical plan fails to find the row-stored data corresponding to the target IO task in the local cache, send an IO request to the storage device according to the target IO task, and receive the column-stored data corresponding to the target IO task sent by the storage device; A conversion module, configured to convert the column-stored data corresponding to the target IO task into the row-stored data corresponding to the target IO task; The execution module is specifically configured to execute an online business according to the row-stored data corresponding to the target IO task and the target row-stored data, where the target row-stored data includes row-stored data corresponding to non-target IO tasks in the IO task list.

14. The online platform according to any one of claims 8 to 13, characterized in that: The format of the column-stored data is the ORC format, and the format of the row-stored data and the format of the target row-stored data are both the CU format.

15. A big data platform, characterized in that: include: An offline platform, a storage device, and the online platform according to any one of claims 8 to 14.

16. The big data platform according to claim 15, characterized in that: The offline platform is used to clean the column-stored data, generate metadata based on the cleaned column-stored data; and send the cleaned column-stored data and metadata to the storage device; The storage device is used to store the column-stored data and the metadata after the data cleaning; The storage device is further configured to send metadata to the online platform; The online platform is further configured to create an index of the row-stored data based on the metadata.

17. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device includes a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method according to any one of claims 1 to 7.

18. A computer-readable storage medium, characterized in that The method comprises computer program instructions, which, when executed by a computing device cluster, perform the method according to any one of claims 1 to 7.

19. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 7.

Images