A data processing method, device and equipment

Abstract

The present application provides a data processing method, device and equipment, the method comprising: obtaining a data processing request, the data processing request comprising a first input format and a second output format; obtaining a target processing unit, the target conversion information of the target processing unit being first conversion information, the first conversion information being used to realize the conversion between the first input format and the second output format; obtaining first data in the first input format from a data source according to the data processing request, and outputting the first data to the target processing unit, so that the target processing unit converts the first data into second data in the second output format using the first conversion information; obtaining second data from the target processing unit, and outputting the second data. Through the technical solution of the present application, the computing resources of the data lake analysis system can be saved and the processing performance can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method, device and equipment. Background Art

[0002] Data Lake Analytics is used to provide users with serverless query and analysis services. It can analyze and query massive amounts of data in any dimension, and can support high concurrency, low latency (millisecond-level response), real-time online analysis, massive data query and other functions.

[0003] In the data lake analysis system, there are storage clusters and computing clusters. The storage cluster includes different types of data sources, which use different data formats. The computing cluster includes multiple computing nodes, and different computing nodes can use different data formats. Usually, the data format used by the data source is different from the data format used by the computing node, so the data format needs to be converted.

[0004] For example, data in data format A1 is read from the data source, converted into data format B1, and output to the computing node, which processes the data in data format B1. Since different types of data sources use different data formats and different computing nodes also use different data formats, the data lake analysis system needs to support the conversion of various data formats. The data lake analysis system needs to provide a large amount of computing resources to achieve data format conversion. As the number of users increases, the demand for computing resources also increases. Summary of the invention

[0005] The present application provides a data processing method, the method comprising:

[0006] Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format;

[0007] Acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format;

[0008] Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information;

[0009] The second data is acquired from the target processing unit, and the second data is output.

[0010] The present application provides a data processing method, which is applied to a data lake analysis system. The data lake analysis system is used to provide a serverless data processing service for a user. The method includes:

[0011] Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format;

[0012] Acquire a target processing unit from a plurality of processing units of the data lake analysis system; wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format;

[0013] Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information;

[0014] Acquire the second data from the target processing unit, and output the second data;

[0015] Among them, the data source includes a cloud database provided by the data lake analysis system.

[0016] The present application provides a data processing method, the method comprising:

[0017] Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format;

[0018] Acquire first data in a first input format from a data source according to the data processing request;

[0019] Outputting first data in the first input format to a target processing unit so that the target processing unit converts the first data into second data in a second output format;

[0020] The second data is acquired from the target processing unit, and the second data is output.

[0021] The present application provides a data processing method, which is applied to a data lake analysis system. For a processing unit among multiple processing units in the data lake analysis system, the processing unit includes multiple different conversion information, and the different conversion information is used to realize data conversion in different formats. The method includes:

[0022] The processing unit obtains first data in a first input format;

[0023] If the target conversion information of the processing unit is first conversion information, and the first conversion information is used to realize the conversion between the first input format and the second output format, converting the first data into second data in the second output format by using the first conversion information;

[0024] The processing unit outputs the second data.

[0025] The present application provides a data processing device, the device comprising:

[0026] an acquisition module, configured to acquire a data processing request, wherein the data processing request includes a first input format and a second output format; acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format;

[0027] a processing module, configured to obtain first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information;

[0028] The second data is acquired from the target processing unit, and the second data is output.

[0029] The present application provides a data processing device, comprising:

[0030] A processor and a machine-readable storage medium, wherein the machine-readable storage medium stores a plurality of computer instructions, and when the processor executes the computer instructions, the processor performs the following processing:

[0031] Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format;

[0032] Acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format;

[0033] Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information;

[0034] The second data is acquired from the target processing unit, and the second data is output.

[0035] Based on the above technical solution, in an embodiment of the present application, by setting the target conversion information of the target processing unit as the first conversion information, the target processing unit uses the first conversion information to convert the first data in the first input format into the second data in the second output format, that is, the data format conversion is realized by the target processing unit, and the target processing unit is usually realized by a logic chip, and the logic chip has very high processing performance. Therefore, the computing resources of the data lake analysis system (such as CPU (Central Processing Unit, central processing unit) resources, etc.) can be saved, and the overall processing performance of the data lake analysis system can be improved, and the overall usage efficiency and experience of the data lake analysis system can be improved, and data processing and computing performance can be accelerated. Hardware acceleration technology is combined to process data docking of the storage cluster and provide a data interface to the computing cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0037] Figure 1 It is a flowchart of a data processing method in one embodiment of the present application;

[0038] Figure 2 is a structural diagram of a data lake analysis system in one implementation of the present application;

[0039] Figure 3A-3E is a schematic diagram of a data scanning cluster in one embodiment of the present application;

[0040] Figure 4 is a schematic diagram of data format conversion in one implementation of the present application;

[0041] Figure 5A and Figure 5B is a structural diagram of a data scanning cluster in one embodiment of the present application;

[0042] Figure 6 It is a flowchart of a data processing method in one embodiment of the present application;

[0043] Figure 7 is a schematic diagram of the structure of a data processing device in one embodiment of the present application;

[0044] Figure 8 It is a structural schematic diagram of a data processing device in one embodiment of the present application. DETAILED DESCRIPTION

[0045] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0046] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0047] A data processing method is proposed in the embodiment of the present application, which can be applied to any device, such as any device of the data lake analysis system, see Figure 1 As shown, it is a flowchart of the method, and the method may include:

[0048] Step 101: Obtain a data processing request, which includes a first input format (ie, the format of data in a data source) and a second output format (ie, the format of data to be output).

[0049] Step 102, obtaining a target processing unit, wherein the target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize the conversion between the first input format and the second output format. Based on the first conversion information, the target processing unit can convert the data in the first input format into the data in the second output format.

[0050] Optionally, in one example, obtaining the target processing unit may include, but is not limited to: arbitrarily selecting a processing unit from multiple processing units in the data lake analysis system, and using the selected processing unit as the target processing unit. Alternatively, target conversion information of multiple processing units in the data lake analysis system may be obtained, and a processing unit may be selected from the multiple processing units using the target conversion information of each processing unit, and the selected processing unit may be used as the target processing unit.

[0051] In one example, for each processing unit of the data lake analysis system, the processing unit may be: a processing unit that is not currently working (i.e., the processing unit is not currently performing a data conversion operation), or a processing unit that is currently working (i.e., the processing unit is currently performing a data conversion operation).

[0052] In one example, the target conversion information of each processing unit is used to select a processing unit from the multiple processing units, and the selected processing unit is used as the target processing unit, which may include but is not limited to: if there is a processing unit whose target conversion information is the first conversion information (used to achieve conversion between a first input format and a second output format), the processing unit whose target conversion information is the first conversion information is determined as the target processing unit; or, if there is no processing unit whose target conversion information is the first conversion information, a processing unit is arbitrarily selected from the multiple processing units, and the selected processing unit is determined as the target processing unit.

[0053] Optionally, in one example, after obtaining the target processing unit, it may also include but is not limited to: if the target conversion information of the target processing unit is the first conversion information, then keeping the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or, if the target conversion information of the target processing unit is the second conversion information (the second conversion information is not used to achieve the conversion between the first input format and the second output format), then modifying the target conversion information of the target processing unit to the first conversion information according to the first input format and the second output format.

[0054] Optionally, in one example, for step 102, it may also be determined whether the data lake analysis system supports the conversion between the first input format and the second output format. If yes, that is, the processing unit of the data lake analysis system supports the conversion between the first input format and the second output format, then the target processing unit is obtained from the multiple processing units of the data lake analysis system. If no, that is, all processing units of the data lake analysis system do not support the conversion between the first input format and the second output format, then the traditional process is used for processing.

[0055] Optionally, in one example, for step 102, the data processing request may further include the number of shards, the number of target processing units is determined according to the number of shards, and the number of target processing units is obtained.

[0056] Step 103, according to the data processing request, obtain first data in a first input format from a data source, and output the first data to a target processing unit, so that the target processing unit converts the first data into second data in a second output format using the first conversion information. This conversion process will not be described in detail.

[0057] Step 104: Acquire the second data from the target processing unit and output the second data. For example, the second data may be output to a computing node so that the computing node uses the second data for processing.

[0058] In one example, the data processing request may further include a service mode. If the service mode is a flow mode, the total amount of data may be obtained, and virtual resource information (such as cost information) may be determined based on the total amount of data, and the virtual resource information may be output. Alternatively, if the service mode is an instance mode, the number of target processing units may be obtained, and virtual resource information may be determined based on the target number of processing units, and the virtual resource information may be output.

[0059] In the above embodiment, the target processing unit includes multiple different conversion information, and different conversion information is used to realize data conversion in different formats; the target processing unit is implemented by a logic chip, and the logic chip may include but is not limited to: FPGA (Field Programmable Gate Array), CPLD (Complex Programmable Logic Device), ASIC (Application Specific Integrated Circuit), etc., without limitation.

[0060] In one example, the above execution order is just an example given for the convenience of description. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0061] Based on the above technical solution, in the embodiment of the present application, the data format conversion is implemented by the target processing unit. The target processing unit is usually implemented by a logic chip and has high processing performance. It can save the computing resources of the data lake analysis system (such as CPU resources, etc.), improve the overall processing performance of the data lake analysis system, improve the overall usage efficiency and experience of the data lake analysis system, accelerate data processing and computing performance, combine hardware acceleration technology to process data docking of the storage cluster, and provide a data interface to the computing cluster.

[0062] Based on the same application concept as the above method, the embodiment of the present application further proposes another data processing method, which can be applied to a data lake analysis system (such as a cloud computing platform in a data lake analysis system). The data lake analysis system is used to provide a serverless data processing service for users. The method includes:

[0063] A data processing request is obtained, the data processing request includes a first input format and a second output format; a target processing unit is obtained from a plurality of processing units in the data lake analysis system, the target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize the conversion between the first input format and the second output format. According to the data processing request, first data in a first input format is obtained from a data source, and the first data is output to the target processing unit, so that the target processing unit converts the first data into second data in a second output format using the first conversion information; the second data is obtained from the target processing unit, and the second data is output; wherein the data source includes a cloud database provided by the data lake analysis system.

[0064] The above data sources may include cloud databases provided by the data lake analysis system, and the cloud databases may be used to provide serverless query analysis services. The data lake analysis system may be a storage-based cloud platform that focuses on data storage, or a computing-based cloud platform that focuses on data processing, or a comprehensive cloud computing platform that combines computing and data storage processing, and there is no restriction on the data lake analysis system.

[0065] The cloud database provided for the data lake analysis system can be used to provide users with serverless query and analysis services. It can analyze and query massive amounts of data in any dimension, and supports high concurrency, low latency (millisecond-level response), real-time online analysis, massive data query and other functions.

[0066] In one example, the data lake analysis system is specifically: a data lake analysis system with separated storage and computing; the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources with different input formats, and the computing cluster includes multiple computing nodes with different output formats. Furthermore, the data lake analysis system may also include a data scanning cluster, the data scanning cluster includes multiple processing units; the data scanning cluster is deployed on the same node as the computing resources of the computing cluster as a built-in module of the computing cluster; or, the data scanning cluster is deployed on a different node as an independent module of the computing cluster; or, the data scanning cluster is an independent cluster different from the computing cluster.

[0067] Based on the same application concept as the above method, a data processing method is also proposed in an embodiment of the present application, which method may include: obtaining a data processing request, which data processing request may include a first input format and a second output format; obtaining first data in a first input format from a data source according to the data processing request, and outputting the first data to a target processing unit, so that the target processing unit converts the first data into second data in a second output format; obtaining second data from the target processing unit, and outputting the second data.

[0068] Based on the same application concept as the above method, an embodiment of the present application further proposes a data processing method, which is applied to a data lake analysis system. The data lake analysis system includes multiple processing units. For each of the multiple processing units, the processing unit includes multiple different conversion information. Different conversion information is used to realize data conversion in different formats. The method includes:

[0069] The processing unit obtains first data in a first input format; if the target conversion information of the processing unit is the first conversion information, and the first conversion information is used to realize the conversion between the first input format and the second output format, the processing unit converts the first data into second data in a second output format using the first conversion information; and the processing unit outputs the second data.

[0070] In one example, before the processing unit converts the first data into second data in a second output format using the first conversion information, if the target conversion information of the processing unit is not the first conversion information, the processing unit modifies the target conversion information of the processing unit to the first conversion information.

[0071] The above data processing method is further explained below in combination with specific application scenarios.

[0072] See also Figure 2 As shown, it is a structural diagram of a Data Lake Analytics system. The Data Lake Analytics system may include a client, a load balancing device, a front-end node (also called a front-end server), a compute node (also called a compute server) and a database. Of course, the Data Lake Analytics system may also include other servers, which is not limited.

[0073] exist Figure 2 In the example, three front-end nodes are used. In actual applications, the number of front-end nodes can be other numbers, and there is no limitation on this. Figure 2 In the embodiment, four computing nodes are taken as an example. In practical applications, the number of computing nodes can be other numbers, and there is no limitation on this. Since the processing flow of each front-end node is the same, the processing flow of each computing node is the same. Therefore, for the convenience of description, in the subsequent embodiments, the processing flow of one front-end node is taken as an example, and the processing flow of one computing node is taken as an example.

[0074] exist Figure 2In the example, five databases are used. In actual applications, the number of databases can be other numbers, and there is no limitation on this. These databases are data sources. In this embodiment, it can be a scenario for heterogeneous data sources, that is, these databases can be databases of the same type or databases of different types. These databases can be relational databases or non-relational databases.

[0075] Furthermore, for each database, the type of this database may also include but is not limited to: OSS (Object Storage Service), Table Store (table storage), HBase (Hadoop Database), HDFS (Hadoop Distributed File System), MySQL (i.e. relational database), RDS (Relational Database Service), DRDS (Distribute Relational Database Service), RDBMS (Relational Database Management System), SQL Server (i.e. relational database), PostgreSQL (i.e. object-relational database), MongoDB (i.e. database based on distributed file storage), etc. Of course, the above are just a few examples of database types, and there is no limitation on the type of database.

[0076] The database is used to store various types of data, and there is no restriction on the data type, such as user data, product data, map data, video data, image data, audio data, etc.

[0077] The client can be an APP (Application) included in a terminal device (such as a PC (Personal Computer), a laptop, a mobile terminal, etc.), or a browser included in the terminal device, without limitation. The load balancing device is used to load balance the data requests of the client, such as after receiving the data request, load balancing the data request to each front-end node.

[0078] In one example, multiple front-end nodes can be used to provide the same function to form a resource pool of front-end nodes. Each front-end node in the resource pool is used to receive a data request sent by a client, parse the data request using SQL (Structured Query Language), generate multiple execution plans based on the parsing results, and process these execution plans. For example, the front-end node can send these execution plans to one or more computing nodes, which process the execution plans.

[0079] In one example, multiple computing nodes are used to provide the same function to form a resource pool of computing nodes. For each computing node in the resource pool, if the computing node receives an execution plan sent by a front-end node, the computing node can process the execution plan and return the processing result to the front-end node.

[0080] To summarize, the data lake analysis system adopts an architecture that separates storage and computing. The computing nodes read data from different data sources (Data Source), which are various types of databases.

[0081] In one example, the data lake analysis system is specifically an architecture in which storage and computing are separated, that is, the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources (i.e., databases) using different input formats, and the computing cluster includes multiple computing nodes using different output formats. On this basis, in an embodiment of the present application, the data lake analysis system may also include a data scanning cluster, which may include multiple processing units, for example, a processing unit implemented by an FPGA, etc.

[0082] See also Figure 3A As shown in the figure, the data scanning cluster can be used as an independent module of the computing cluster and deployed on different nodes from the computing resources (such as CPU resources, etc.) of the computing cluster. That is to say, the processing unit of the data scanning cluster is deployed in the computing cluster, but it is deployed on different nodes from the computing resources (such as CPU resources, etc.) of the computing cluster. Specifically, in the data lake analysis system with separated storage and computing, the data scanning cluster is a module in the computing cluster and is a functional module in the computing cluster that directly faces the storage cluster.

[0083] See also Figure 3BAs shown, the data scanning cluster can be used as a built-in module of the computing cluster and deployed on the same node as the computing resources (such as CPU resources, etc.) of the computing cluster. That is to say, the processing unit of the data scanning cluster is deployed in the computing cluster as a built-in module of the computing node in the computing cluster and is located on the same node as the CPU-based operator. The computing task scheduling determines whether to enable the data scanning cluster for data format conversion. If not enabled, the data format conversion is implemented based on the CPU software module of the computing node.

[0084] See also Figure 3C As shown in the figure, the data scanning cluster can be used as an independent cluster different from the computing cluster. In the data lake analysis system with separate storage and computing, the data scanning cluster is a functional module for the computing cluster, and the data scanning cluster is a functional module for the storage cluster. The data scanning cluster is a completely independent cluster on the cloud. It can concurrently respond to data scanning requests from multiple different computing clusters on the cloud in the form of a service. The data scanning cluster runs completely independently and has its own cluster elastic management and expansion and contraction.

[0085] For the convenience of description, in the subsequent embodiments, a data scanning cluster is taken as an independent cluster as an example.

[0086] In one example, the data lake analysis system may include multiple computing clusters, each of which includes multiple computing nodes. Each computing cluster may be a computing cluster for SQL (Structured Query Language) computing, a computing cluster for machine learning, or a computing cluster for deep learning, without limitation.

[0087] For details, see Figure 3D As shown, these computing clusters may include but are not limited to: a computing cluster based on Presto, a computing cluster based on Spark, a computing cluster based on Hadoop, a computing cluster based on Flink, a computing cluster based on TensorFlow, a computing cluster based on PyTorch, etc.

[0088] For computing clusters based on Presto, a data access interface adapted to Presto is provided, that is, the data output to the computing cluster is data that matches the Presto data format. For computing clusters based on Spark, a data access interface adapted to Spark is provided, that is, the data output to the computing cluster is data that matches the Spark data format. For computing clusters based on Hadoop, a data access interface adapted to Hadoop is provided, that is, the data output to the computing cluster is data that matches the Hadoop data format. For computing clusters based on Flink, a data access interface adapted to Flink is provided, that is, the data output to the computing cluster is data that matches the Flink data format. For computing clusters based on TensorFlow, a data access interface adapted to TensorFlow is provided, that is, the data output to the computing cluster is data that matches the TensorFlow data format. For computing clusters based on PyTorch, a data access interface adapted to PyTorch is provided, that is, the data output to the computing cluster is data that matches the PyTorch data format, and so on.

[0089] In one example, a data lake analysis system may include a storage cluster, which includes multiple data sources. The data source may be a database, such as a cloud database. The cloud database is used to provide users with serverless query and analysis services, which can analyze and query massive data in any dimension, support high concurrency, low latency (millisecond response), real-time online analysis, massive data query, etc.

[0090] In one example, these data sources may include, but are not limited to: OSS-based data sources, TableStore-based data sources, HBase-based data sources, HDFS-based data sources, MySQL-based data sources, RDS-based data sources, DRDS-based data sources, RDBMS-based data sources, PostgreSQL-based data sources, etc. Of course, the above are just examples and are not limited to this.

[0091] See also Figure 3E As shown, due to different types of data sources, the data formats of the data in the data sources are also different. For example, the data formats may include but are not limited to: parquet data format, orc data format, text data format, json data format, kv data format, rcfile data format, avro data format, arrow data format, etc. Of course, the above are just examples, and there may be other data formats, which are not limited to this.

[0092] In summary, since the data format of the data source is different from that of the computing cluster, data format conversion is required so that the computing cluster can process the data correctly. For example, if the data format of the data source is json and the computing cluster is based on Presto, the data in json format needs to be converted to data that matches the Presto data format.

[0093] In the embodiment of the present application, the data format conversion is achieved by providing a data scanning cluster, that is, the data format conversion is achieved through the processing unit (such as FPGA, etc.) in the data scanning cluster.

[0094] In one example, in order to achieve data format conversion, conversion information can be configured in a processing unit (such as FPGA, etc.), and the processing unit can use the conversion information to achieve data format conversion. There is no restriction on the content of the conversion information, as long as the processing unit can use the conversion information to achieve data format conversion.

[0095] For example, conversion information A1 is configured in advance in the processing unit. Based on the conversion information A1, the processing unit can convert data in the json data format into data matching the Presto data format.

[0096] In one example, a plurality of different conversion information may be configured in a processing unit (such as an FPGA, etc.), and different conversion information is used to implement data conversion in different formats. For example, conversion information A1, conversion information A2, conversion information A3, conversion information A4, and so on are configured in the processing unit. Based on conversion information A1, the processing unit can convert data in json data format into data that matches the Presto data format. Based on conversion information A2, the processing unit can convert data in json data format into data that matches the Spark data format. Based on conversion information A3, the processing unit can convert data in text data format into data that matches the Presto data format. Based on conversion information A4, the processing unit can convert data in text data format into data that matches the Spark data format, and so on.

[0097] Of course, the above is only an example of conversion information. In actual applications, more conversion information can be configured in the processing unit to achieve conversion of various data formats. Figure 4 The figure shows a schematic diagram of data format conversion. The first column indicates the data format supported by the data source, and the first row indicates the data format supported by the computing cluster. Figure 4 The "yes" in the table indicates that the conversion between these two data formats is supported. Figure 4The "No" in the "No" indicates that the conversion of the two data formats is not supported. Based on this, multiple conversion information can be configured in the processing unit, so that the processing unit supports the conversion of the two data formats corresponding to the "Yes" through the conversion information.

[0098] In summary, since a plurality of different conversion information are configured in the processing unit, and different conversion information is used to realize data conversion in different formats, the computing power of the processing unit can be fully utilized and the utilization rate of the processing unit can be improved. For example, if the processing unit is configured with conversion information A1, the processing unit is used to convert data in json data format into data matching the Presto data format. When there is no task of "converting data in json data format into data matching the Presto data format", the processing unit is in an idle state, wasting the computing power of the processing unit. If the processing unit is configured with conversion information A1 and conversion information A2, the processing unit is used to convert data in json data format into data matching the Presto data format, and convert data in json data format into data matching the Spark data format. When there is no task of "converting data in json data format into data matching the Presto data format", the processing unit can also convert data in json data format into data matching the Spark data format, thereby avoiding the processing unit being in an idle state and improving the computing power of the processing unit.

[0099] In one example, the use of processing units in the data scanning cluster is relatively fixed, and is used to accelerate data scanning tasks of different computing clusters. Figure 5A As shown, the data scanning cluster may include basic modules such as instruction storage, data storage, constant storage, register group, data storage linked list, instruction execution, etc. For further information, see Figure 5B As shown, the data scanning cluster may also include multiple processing units (such as FPGA, etc.), each processing unit is used to convert different data formats. In addition, the data scanning cluster may also include a scheduling and management module, an input module, an output module, etc.

[0100] In the above application scenario, see Figure 6 As shown, it is a flowchart of the data processing method proposed in the embodiment of the present application, which can be applied to the data scanning cluster of the data lake analysis system. The method may include:

[0101] Step 601, obtaining a data processing request, such as a data scan request.

[0102] Specifically, the client can send a data processing request to the data lake analysis system through a load balancing device, so that the data scanning cluster of the data lake analysis system can obtain the data processing request. For example, the scheduling and management module of the data scanning cluster can obtain the data processing request.

[0103] Step 602: determine whether the data lake analysis system supports data format conversion corresponding to the data processing request. If yes, execute step 603; if no, prompt that the data processing request is not supported.

[0104] Specifically, the data processing request may include an input data format (i.e., the format of the data in the data source. For the sake of distinction, the first input format is taken as an example below, such as the json data format) and an output target format (i.e., the format of the data to be output. For the sake of distinction, the second output format is taken as an example below, such as the Presto data format). Therefore, it can be determined whether the data lake analysis system supports the conversion between the first input format and the second output format. If so, execute step 603; if not, it is prompted that the data processing request is not supported.

[0105] For example, the scheduling and management module of the data scanning cluster can obtain the first input format and the second output format from the data processing request, and query whether the data lake analysis system supports the conversion between the first input format and the second output format. Specifically, it is assumed that the data lake analysis system includes a capability registry, and the capability registry is used to record the conversion of all data formats supported by the data lake analysis system, and the capability registry is referred to as Figure 4 If the capability registry does not contain the first input format and / or the second output format, it is determined that the data lake analysis system does not support the conversion between the first input format and the second output format; if the capability registry contains the first input format and the second output format, and the first input format and the second output format correspond to "No", it is determined that the data lake analysis system does not support the conversion between the first input format and the second output format; if the capability registry contains the first input format and the second output format, and the first input format and the second output format correspond to "Yes", it is determined that the data lake analysis system supports the conversion between the first input format and the second output format.

[0106] Step 603: Select a target processing unit from multiple processing units in the data lake analysis system.

[0107] Among them, for each processing unit among the multiple processing units, the processing unit can be: a processing unit that is not currently working (that is, the processing unit is not currently performing a data conversion operation), or a processing unit that is currently working (that is, the processing unit is currently performing a data conversion operation).

[0108] For example, a target processing unit may be selected from processing unit 1, processing unit 2, and processing unit 3, where processing unit 1 may be a currently inoperative processing unit or a currently operative processing unit, processing unit 2 may be a currently inoperative processing unit or a currently operative processing unit, and so on.

[0109] In one example, the data processing request may also include a service mode. If the service mode is a flow mode, it means that the user is charged by the total amount of data. Based on this, the user can share processing units with other users. Therefore, the multiple processing units of the data lake analysis system can be processing units that are not currently working or processing units that are currently working. In other words, the processing units that are not currently working can be used as target processing units, or the processing units that are currently working can be used as target processing units. If the service mode is an instance mode, it means that the user is charged by the number of processing units. Based on this, the user uses the processing unit alone. Therefore, the multiple processing units of the data lake analysis system can be processing units that are not currently working. In other words, the processing units that are not currently working can be used as target processing units.

[0110] In one example, a processing unit can be arbitrarily selected from multiple processing units in the data lake analysis system, and the selected processing unit is used as the target processing unit. Alternatively, target conversion information of multiple processing units in the data lake analysis system can be obtained, and a processing unit can be selected from the multiple processing units using the target conversion information of each processing unit, and the selected processing unit can be used as the target processing unit.

[0111] For example, assuming that the scheduling and management module of the data scanning cluster needs to select a target processing unit from processing unit 1, processing unit 2 and processing unit 3, the following method is adopted: a processing unit can be randomly selected from these processing units, such as selecting processing unit 1, and using processing unit 1 as the target processing unit; or, a processing unit can be selected from these processing units based on target conversion information of processing unit 1, processing unit 2 and processing unit 3, such as selecting processing unit 2, and using processing unit 2 as the target processing unit.

[0112] Among them, using the target conversion information of each processing unit to select a processing unit from multiple processing units, and using the selected processing unit as the target processing unit may include but is not limited to: if there is a processing unit whose target conversion information is the first conversion information (used to achieve conversion between a first input format and a second output format), then the processing unit whose target conversion information is the first conversion information can be determined as the target processing unit; or, if there is no processing unit whose target conversion information is the first conversion information, then a processing unit can be randomly selected from multiple processing units, and the selected processing unit can be determined as the target processing unit.

[0113] Among them, the target conversion information is the conversion information currently enabled by the processing unit, that is, the conversion information currently being used by the processing unit. For example, the processing unit configures conversion information A1 (used to convert data in json data format to data matching the Presto data format) and conversion information A2 (used to convert data in json data format to data matching the Spark data format). If the target conversion information is conversion information A1, it means that the processing unit is currently used to convert data in json data format to data matching the Presto data format, but not to convert data in json data format to data matching the Spark data format. If the target conversion information is conversion information A2, it means that the processing unit is currently used to convert data in json data format to data matching the Spark data format, and so on.

[0114] Assuming that the first input format is the json data format and the second output format is the Presto data format, the first conversion information is conversion information A1, that is, the first conversion information is used to realize the conversion between the json data format and the Presto data format. If the target conversion information of processing unit 1 is conversion information A1, the target conversion information of processing unit 1 is the first conversion information, and processing unit 1 can be determined as the target processing unit.

[0115] In one example, the data processing request may also include the number of shards, which indicates the number of processing units that the user needs to use. Therefore, the number of target processing units may also be determined based on the number of shards, and then the target processing units may be selected from multiple processing units in the data lake analysis system.

[0116] For example, assuming that the number of shards is 5, that is, the number of processing units is 5, it is necessary to select 5 target processing units from multiple processing units in the data lake analysis system. For the specific selection method, please refer to the above embodiment.

[0117] Step 604: Set the target conversion information of the target processing unit (eg, one or more target processing units, such as 5 target processing units) as the first conversion information according to the first input format and the second output format.

[0118] The first conversion information is used to implement the conversion between the first input format and the second output format, that is, to convert data in the first input format into data in the second output format.

[0119] Specifically, if the target conversion information of the target processing unit is the first conversion information, the target conversion information of the target processing unit is kept unchanged according to the first input format and the second output format; or, if the target conversion information of the target processing unit is the second conversion information (the second conversion information is not used to achieve the conversion between the first input format and the second output format), the target conversion information of the target processing unit is modified from the second conversion information to the first conversion information according to the first input format and the second output format.

[0120] For example, assuming that the first input format is the json data format and the second output format is the Presto data format, the first conversion information is conversion information A1, that is, the first conversion information is used to achieve the conversion between the json data format and the Presto data format. Further, if the target conversion information of the target processing unit is conversion information A1, the target conversion information of the target processing unit can be kept unchanged, that is, the target conversion information is still conversion information A1. If the target conversion information of the target processing unit is conversion information A2 (used to achieve the conversion between the json data format and the Spark data format), the target conversion information of the target processing unit can be modified to conversion information A1, so that the target processing unit is no longer used to achieve the conversion between the json data format and the Spark data format, but is used to achieve the conversion between the json data format and the Presto data format.

[0121] Among them, steps 601 to 604 may be executed by a scheduling and management module of the data scanning cluster.

[0122] Step 605 , obtaining first data in a first input format from a data source (the data in the data source may be referred to as first data) according to the data processing request, and outputting the first data to a target processing unit.

[0123] Specifically, the data processing request may include information of the data source, based on which the first data may be obtained from the data source, and the data format of the first data is the first input format, and the acquisition process is not described in detail. Then, the first data in the first input format may be output to the target processing unit.

[0124] For example, the input module of the data scanning cluster may obtain first data in a first input format from a data source, and output the first data in the first input format to a target processing unit.

[0125] Step 606: The target processing unit converts the first data into second data in a second output format using the first conversion information (the converted data is referred to as second data). The conversion process will not be described in detail.

[0126] Specifically, referring to the above embodiment, the target conversion information of the target processing unit is the first conversion information, such as conversion information A1, and conversion information A1 is used to realize the conversion between the json data format and the Presto data format. Assuming that the first input format is the json data format and the second output format is the Presto data format, based on this, the data format of the first data is the json data format, and the target processing unit can use the conversion information A1 to convert the first data in the json data format into the second data in the Presto data format.

[0127] Step 607: Acquire second data in a second output format from the target processing unit, and output the second data.

[0128] For example, the output module of the data scanning cluster obtains the second data in the second output format, such as the second data in the Presto data format, from the target processing unit, and outputs the second data in the Presto data format to the computing node, such as the computing node in the Presto-based computing cluster. Since the second data output to the computing node is in the Presto data format, the computing node can use the second data for processing.

[0129] In one example, the data processing request may also include a service mode. If the service mode is a flow mode (i.e., a shared service mode), it means that the user is charged by the total amount of data. Based on this, the user can share the processing unit with other users. Therefore, the total amount of data (i.e., the total amount of data read from the data source) can be obtained, and virtual resource information (such as cost information) can be determined based on the total amount of data, and the virtual resource information can be output, such as outputting the virtual resource information to the user. If the service mode is an instance mode (i.e., an exclusive instance mode), it means that the user is charged by the number of processing units. Based on this, the user uses the processing unit alone. Therefore, the target number of processing units can be obtained, and the virtual resource information (such as cost information) can be determined based on the target number of processing units, and the virtual resource information can be output, such as outputting the virtual resource information to the user.

[0130] In one example, the above execution order is just an example given for the convenience of description. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0131] Based on the above technical solution, in the embodiment of the present application, the conversion of data format is realized by the target processing unit, which is usually realized by a logic chip and has high processing performance, saving computing resources (such as CPU resources, etc.) of the data lake analysis system, improving the overall processing performance of the data lake analysis system, improving the overall use efficiency and experience of the data lake analysis system, accelerating data processing and computing performance, combining hardware acceleration technology to process data docking of storage clusters, and providing data interfaces to computing clusters. The data scanning cluster in this embodiment has better versatility and product application capabilities, greatly improving the scope of application of docked and accelerated computing clusters, greatly improving the productization capabilities of cloud products, providing FPGA data scanning acceleration services in multiple modes, and proposing a general FPGA data scanning engine that can have built-in input and output support for multiple data formats, and developing specific FPGA data scanning computing acceleration cores for specific computing engines.

[0132] Based on the same application concept as the above method, the embodiment of the present application also provides a data processing device, such as Figure 7 , which is a structural diagram of the data processing device, the data processing device includes:

[0133] The acquisition module 71 is used to acquire a data processing request, wherein the data processing request includes a first input format and a second output format; acquire a target processing unit, wherein the target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize the conversion between the first input format and the second output format;

[0134] A processing module 72 is configured to obtain first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information;

[0135] The second data is acquired from the target processing unit, and the second data is output.

[0136] When the acquisition module 71 acquires the target processing unit, it is specifically used to: acquire target conversion information of multiple processing units of the data lake analysis system, and select a processing unit from the multiple processing units as the target processing unit using the target conversion information.

[0137] In one example, the processing module 72 is further configured to:

[0138] If the target conversion information of the target processing unit is the first conversion information, keeping the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or,

[0139] If the target conversion information of the target processing unit is the second conversion information, the target conversion information of the target processing unit is modified to the first conversion information according to the first input format and the second output format.

[0140] Based on the same application concept as the above method, the embodiment of the present application further provides a data processing device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores a plurality of computer instructions, and the processor performs the following processing when executing the computer instructions:

[0141] Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format;

[0142] Acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format;

[0143] Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information;

[0144] The second data is acquired from the target processing unit, and the second data is output.

[0145] The present application also provides a machine-readable storage medium, on which a number of computer instructions are stored; when the computer instructions are executed, the following processing is performed:

[0146] Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format;

[0147] Acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format;

[0148] Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information;

[0149] The second data is acquired from the target processing unit, and the second data is output.

[0150] See also Figure 8As shown, it is a structural diagram of the data processing device proposed in the embodiment of the present application, and the data processing device 80 may include: a processor 81, a network interface 82, a bus 83, and a memory 84. The memory 84 may be any electronic, magnetic, optical or other physical storage device, and may contain or store information, such as executable instructions, data, etc. For example, the memory 84 may be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state hard disk, any type of storage disk (such as optical disk, DVD, etc.).

[0151] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0152] For the convenience of description, the above device is described in terms of functions and is described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: Applied to a data lake analysis system, the data lake analysis system is an architecture in which storage and computing are separated, the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources using different input formats, and the computing cluster includes multiple computing nodes using different output formats; the data lake analysis system also includes a data scanning cluster, the data scanning cluster includes multiple processing units implemented by FPGA; the processing units of the data scanning cluster are deployed in the computing cluster as built-in modules of the computing nodes in the computing cluster, and are located in the same node as the CPU-based operators. Whether to enable the data scanning cluster for data format conversion is determined by the computing task scheduling, and if not enabled, the data format conversion is implemented based on the CPU software module of the computing node; the method includes: Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format; Acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format; Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information; Acquire the second data from the target processing unit, and output the second data; Wherein, the target acquisition processing unit includes: Obtain target conversion information of multiple processing units of the data lake analysis system, wherein each processing unit is configured with multiple different conversion information, and the different conversion information is used to realize data conversion in different formats; wherein the target conversion information is the conversion information currently enabled by the processing unit; If there is a processing unit whose target conversion information is the first conversion information, determining the processing unit whose target conversion information is the first conversion information as the target processing unit; If there is no processing unit whose target conversion information is the first conversion information, a processing unit is randomly selected from the plurality of processing units, and the selected processing unit is determined as the target processing unit.

2. The method according to claim 1, characterized in that After acquiring the target processing unit, the method further includes: If the target conversion information of the target processing unit is the first conversion information, keeping the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or, If the target conversion information of the target processing unit is the second conversion information, the target conversion information of the target processing unit is modified to the first conversion information according to the first input format and the second output format.

3. The method according to claim 1, characterized in that The target acquisition processing unit includes: Determine whether the data lake analysis system supports conversion between the first input format and the second output format; If yes, a target processing unit is obtained from the plurality of processing units of the data lake analysis system.

4. The method according to claim 1, characterized in that: The data processing request also includes the number of shards, and the obtaining target processing unit includes: Determine the number of target processing units according to the number of slices; Obtain the number of target processing units.

5. The method according to claim 1, characterized in that The method further comprises: The data processing request also includes a service mode. If the service mode is a flow mode, the total amount of data is acquired, virtual resource information is determined according to the total amount of data, and the virtual resource information is output; If the service mode is the instance mode, the target number of processing units is obtained, virtual resource information is determined according to the target number of processing units, and the virtual resource information is output.

6. A data processing method, characterized in that: Applied to a data lake analysis system, the data lake analysis system is used to provide a serverless data processing service for users, the data lake analysis system is an architecture with storage and computing separated, the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources with different input formats, the computing cluster includes multiple computing nodes with different output formats; the data lake analysis system also includes a data scanning cluster, the data scanning cluster includes multiple processing units implemented by FPGA; the processing unit of the data scanning cluster is deployed in the computing cluster, as a built-in module of the computing node in the computing cluster, and is located in the same node as the CPU-based operator, and the computing task scheduling determines whether to enable the data scanning cluster for data format conversion, and if not, the data format conversion is implemented based on the CPU software module of the computing node; the method includes: Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format; Acquire a target processing unit from a plurality of processing units of the data lake analysis system; wherein the target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize the conversion between the first input format and the second output format; wherein each processing unit of the data lake analysis system is configured with a plurality of different conversion information, and different conversion information is used to realize data conversion in different formats; wherein the target conversion information is the conversion information currently enabled by the processing unit; Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information; Acquire the second data from the target processing unit, and output the second data; Among them, the data source includes a cloud database provided by the data lake analysis system.

7. The method according to claim 6, characterized in that The data lake analysis system is specifically: a data lake analysis system with separated storage and computing; the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources using different input formats, and the computing cluster includes multiple computing nodes using different output formats; The data lake analysis system also includes a data scanning cluster, which includes multiple processing units; the data scanning cluster is deployed on the same node as the computing resources of the computing cluster as a built-in module of the computing cluster; or, the data scanning cluster is deployed on a different node as an independent module of the computing cluster as the computing resources of the computing cluster; or, the data scanning cluster is an independent cluster different from the computing cluster.

8. A data processing method, characterized in that: The method is applied to a data lake analysis system, wherein the data lake analysis system is an architecture in which storage and computing are separated. The data lake analysis system includes a storage cluster and a computing cluster, wherein the storage cluster includes multiple data sources using different input formats, and the computing cluster includes multiple computing nodes using different output formats. The data lake analysis system also includes a data scanning cluster, wherein the data scanning cluster includes multiple processing units implemented by FPGA. The processing units of the data scanning cluster are deployed in the computing cluster as built-in modules of computing nodes in the computing cluster, and are located in the same node as CPU-based operators. Whether to enable the data scanning cluster for data format conversion is determined by computing task scheduling. If not enabled, data format conversion is implemented based on the CPU software module of the computing node. For a processing unit among multiple processing units in the data lake analysis system, the processing unit includes multiple different conversion information, and different conversion information is used to implement data conversion in different formats. The processing unit target conversion information is the conversion information currently enabled by the processing unit. The method includes: The processing unit obtains first data in a first input format; If the target conversion information of the processing unit is first conversion information, and the first conversion information is used to realize the conversion between the first input format and the second output format, the first data is converted into second data in the second output format by using the first conversion information; The processing unit outputs the second data.

9. The method according to claim 8, characterized in that Before converting the first data into second data in a second output format using the first conversion information, the method further includes: If the target conversion information of the processing unit is not the first conversion information, the processing unit modifies the target conversion information of the processing unit to the first conversion information.

10. A data processing device, characterized in that: Applied to a data lake analysis system, the data lake analysis system is an architecture in which storage and computing are separated, the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources using different input formats, and the computing cluster includes multiple computing nodes using different output formats; the data lake analysis system also includes a data scanning cluster, the data scanning cluster includes multiple processing units implemented by FPGA; the processing units of the data scanning cluster are deployed in the computing cluster as built-in modules of the computing nodes in the computing cluster, and are located in the same node as the CPU-based operators. Whether to enable the data scanning cluster for data format conversion is determined by the computing task scheduling, and if not enabled, the data format conversion is implemented based on the CPU software module of the computing node; the device includes: an acquisition module, configured to acquire a data processing request, wherein the data processing request includes a first input format and a second output format; acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format; a processing module, configured to obtain first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information; Acquire the second data from the target processing unit, and output the second data; Wherein, when the acquisition module acquires the target processing unit, it is specifically used to: Obtain target conversion information of multiple processing units of the data lake analysis system, wherein each processing unit is configured with multiple different conversion information, and the different conversion information is used to realize data conversion in different formats; wherein the target conversion information is the conversion information currently enabled by the processing unit; If there is a processing unit whose target conversion information is the first conversion information, determining the processing unit whose target conversion information is the first conversion information as the target processing unit; If there is no processing unit whose target conversion information is the first conversion information, a processing unit is randomly selected from the plurality of processing units, and the selected processing unit is determined as the target processing unit.

11. The device according to claim 10, characterized in that The processing module is also used for: If the target conversion information of the target processing unit is the first conversion information, keeping the target conversion information of the target processing unit unchanged according to the first input format and the second output format; or, If the target conversion information of the target processing unit is the second conversion information, the target conversion information of the target processing unit is modified to the first conversion information according to the first input format and the second output format.

12. A data processing device, characterized in that: Applied to a data lake analysis system, the data lake analysis system is an architecture in which storage and computing are separated, the data lake analysis system includes a storage cluster and a computing cluster, the storage cluster includes multiple data sources using different input formats, and the computing cluster includes multiple computing nodes using different output formats; the data lake analysis system also includes a data scanning cluster, the data scanning cluster includes multiple processing units implemented by FPGA; the processing units of the data scanning cluster are deployed in the computing cluster as built-in modules of the computing nodes in the computing cluster, and are located in the same node as the CPU-based operators. Whether to enable the data scanning cluster for data format conversion is determined by the computing task scheduling. If not enabled, the data format conversion is implemented based on the CPU software module of the computing node; including: A processor and a machine-readable storage medium, wherein the machine-readable storage medium stores a plurality of computer instructions, and when the processor executes the computer instructions, the processor performs the following processing: Obtaining a data processing request, wherein the data processing request includes a first input format and a second output format; Acquire a target processing unit, wherein target conversion information of the target processing unit is first conversion information, and the first conversion information is used to realize conversion between the first input format and the second output format; Acquire first data in a first input format from a data source according to the data processing request, and output the first data to the target processing unit, so that the target processing unit converts the first data into second data in a second output format by using the first conversion information; Acquire the second data from the target processing unit, and output the second data; Wherein, the target acquisition processing unit includes: Obtain target conversion information of multiple processing units of the data lake analysis system, wherein each processing unit is configured with multiple different conversion information, and the different conversion information is used to realize data conversion in different formats; wherein the target conversion information is the conversion information currently enabled by the processing unit; If there is a processing unit whose target conversion information is the first conversion information, determining the processing unit whose target conversion information is the first conversion information as the target processing unit; If there is no processing unit whose target conversion information is the first conversion information, a processing unit is randomly selected from the plurality of processing units, and the selected processing unit is determined as the target processing unit.

Images