Data broadcasting method and device based on DPU, electronic equipment, computer storage medium and program product

Abstract

The embodiment of the invention provides a DPU-based data broadcasting method and device, electronic equipment, a computer storage medium and a program product. The method comprises the steps that in response to a table name of a first data table and a table name of a second data table sent by the CPU, a target data table needing a small storage space when the data tables are stored is determined, and when the table name of the first data table and the table name of the second data table determine that the received SQL comprises broadcast merging operator operation, the table name of the first data table and the table name of the second data table correspond to the table name of the second data table; obtaining the data in the SQL and sending the data to the DPU; if the storage space required for storing the target data table is smaller than or equal to the remaining memory of the DPU, storing the target data table to the memory of the DPU; and broadcasting the target data table to the DPU memory comprising the slave node of the DPU. According to the method, the target data table is directly broadcasted to the DPU memory of the slave node through the DPU of the master node, so that the data broadcasting efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a DPU-based data broadcasting method, device, electronic device, computer storage medium, and program product. Background Art

[0002] Spark is a distributed big data computing platform that is mainly used for offline computing, interactive query, real-time streaming computing, machine learning, graph computing and other scenarios.

[0003] When using Spark for interactive queries, standard SQL is usually used to execute SQL queries and read and write. During the SQL execution process, if there is a join operation in the SQL, and the join operation is a small table to a large table (BroadcastJoin), it is usually executed by the CPU of the master node in Spark.

[0004] The CPU executes the connection of the small table to the large table, and in the process of broadcasting the small table, the CPU will read the small table into the memory of the master node, then serialize the data of the small table and compress it, broadcast the compressed serialized data through the network card of the master node, and in the network cards of other nodes, after receiving the broadcast serialized data through the network cards, the other nodes will deserialize the serialized data into small table data and store it in the memory. However, the serialization and deserialization operations will increase the time required to broadcast the small table, thereby reducing the efficiency of the distributed big data computing platform in broadcasting the small table to the large table. Summary of the invention

[0005] In view of this, embodiments of the present application provide a DPU-based data broadcasting method, device, electronic device, and computer storage medium.

[0006] According to a first aspect of an embodiment of the present application, a DPU-based data broadcast method is provided, which is applied to a DPU of a master node in a distributed computing platform, wherein the master node includes a CPU and the DPU connected to each other, and the method includes:

[0007] In response to the table name of the first data table and the table name of the second data table sent by the CPU, determine a target data table that requires less storage space when storing the data table, wherein the table name of the first data table and the table name of the second data table are obtained in the SQL and sent to the DPU when the CPU determines that the received SQL includes a broadcast merge operator operation;

[0008] If the storage space required to store the target data table is less than or equal to the remaining memory of the DPU, storing the target data table in the memory of the DPU; and

[0009] The target data table is broadcasted to the DPU memory of the slave nodes including the DPU.

[0010] In some embodiments, determining a target data table occupying a smaller storage space in response to the table name of the first data table and the table name of the second data table sent by the CPU includes:

[0011] Determine, according to the received table name of the first data table and the table name of the second data table, a storage space required for storing the first data table and a storage space required for storing the second data table;

[0012] The target data table is determined according to a storage space required for storing the first data table and a storage space required for storing the second data table.

[0013] In some embodiments, the method further comprises:

[0014] After broadcasting the target data table to the DPU memory of the slave node including the DPU, a first instruction is sent to the CPU to instruct the CPU to split the non-target data table into multiple sub-data tables and then broadcast each of the sub-data tables to different slave nodes.

[0015] In some embodiments, the method further comprises:

[0016] In response to the merged data sent by each of the slave nodes, a plurality of the merged data are aggregated to determine the aggregated merged data corresponding to the SQL, wherein the merged data is generated by the slave node according to the received target data table and the sub-data table.

[0017] According to a second aspect of an embodiment of the present application, a DPU-based data broadcast method is provided, which is applied to a CPU of a master node in a distributed computing platform, wherein the master node includes the CPU and the DPU connected to each other, and the method includes:

[0018] Parsing the received SQL to determine the target operation of the SQL, wherein the SQL is configured by the user in the configuration interface;

[0019] If the target operation is a broadcast merge operator operation, obtain the table name of the first data table and the table name of the second data table in the SQL;

[0020] The table name of the first data table and the table name of the second data table are sent to the DPU, wherein the table name of the first data table and the table name of the second data table are used by the DPU to determine a target data table requiring a smaller storage space when storing the data table.

[0021] In some embodiments, the method further comprises:

[0022] In response to a first instruction sent by the DPU, splitting a non-target data table into a plurality of sub-data tables;

[0023] Each of the sub-data tables is broadcasted to different slave nodes.

[0024] According to a third aspect of an embodiment of the present application, a DPU-based data broadcasting device is provided, which is applied to a DPU of a master node in a distributed computing platform, wherein the master node includes a CPU and the DPU connected to each other, and the device includes:

[0025] a determination module, configured to determine, in response to a table name of a first data table and a table name of a second data table sent by the CPU, a target data table requiring a smaller storage space when storing the data table, wherein the table name of the first data table and the table name of the second data table are obtained from the SQL and sent to the DPU when the CPU determines that the received SQL includes a broadcast merge operator operation;

[0026] a storage module, configured to store the target data table in the memory of the DPU if the storage space required to store the target data table is less than or equal to the remaining memory of the DPU; and

[0027] A broadcast module is used to broadcast the target data table to the DPU memory of the slave node including the DPU.

[0028] According to the fourth aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the data broadcasting method described in the first aspect.

[0029] According to a fifth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the data broadcasting method as described in the first aspect is implemented.

[0030] According to a sixth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the DPU-based data broadcasting method.

[0031] According to the DPU-based data broadcasting scheme provided in the embodiment of the present application, first, the DPU of the master node receives the table names of the first data table and the second data table sent by the CPU of the master node, and determines the target data table with a smaller storage space required when storing the data table. Further, if the DPU determines that the storage space required to store the target data table is less than or equal to the remaining memory of the DPU, the target data table is stored in the memory of the DPU; and the target data table is broadcast to the DPU memory of the slave node including the DPU to complete the data broadcast. The embodiment of the present application determines that the storage space required to store the target data table is less than or equal to the size relationship of the remaining memory of the DPU through the DPU. In the case where the target data table can be stored in the DPU memory, the target data table can be directly broadcast to the DPU memory of the slave node through the master node DPU, thereby improving the efficiency of data broadcasting, thereby effectively avoiding the serialization operation required for the CPU to broadcast the data table. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0033] Figure 1 A schematic diagram of the structure of a distributed computing platform applicable to an embodiment of the present application;

[0034] Figure 2 is a flowchart of a method for data broadcasting based on DPU according to an embodiment of the present application;

[0035] Figure 3 is a flowchart of a method for data broadcasting based on DPU according to an embodiment of the present application;

[0036] Figure 4 is a flowchart of a method for data broadcasting based on DPU according to another embodiment of the present application;

[0037] Figure 5 is a structural block diagram of a DPU-based data broadcasting device according to an embodiment of the present application;

[0038] Figure 6 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0040] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0041] The present application embodiment provides a DPU-based data broadcast method, which is applied to a master node including a distributed computing platform. Figure 1 As shown, the distributed computing platform includes a master node and multiple slave nodes, wherein the master node includes a DPU (Data Processing Unit) and a CPU, and the DPU and the CPU are connected. Further, the data broadcasting method of the embodiment of the present application is executed by the DPU, and the specific steps are as follows: Figure 2 As shown, including:

[0042] S201, the master node DPU determines a target data table requiring a smaller storage space when storing the data table in response to the table name of the first data table and the table name of the second data table sent by the master node CPU.

[0043] In an embodiment of the present application, after receiving the SQL, the master node CPU parses the SQL. When it is determined that there is a broadcast merge operator operation in the SQL, the master node CPU obtains the table name of the first data table and the table name of the second data table carried in the SQL, and the master node CPU sends the table name of the first data table and the table name of the second data table to the master node DPU. When the master node DPU receives the table name of the first data table and the table name of the second data table, the master node DPU determines the storage space required to store the first data table and the storage space required to store the second data packet in response to the table name of the first data table and the table name of the second data table. Then, the master node DPU determines a target data table with smaller storage space required to store the data table in the first data table and the second data table.

[0044] In one example, if the master node DPU determines that the storage space required to store the first data packet is smaller than the storage space required to store the second data table, the master node DPU determines the first data table as the data table.

[0045] S202: The master node DPU determines that if the storage space required to store the target data table is less than or equal to the remaining memory of the master node DPU, the target data table is stored in the memory of the master node DPU.

[0046] In an embodiment of the present application, after the master node DPU determines the target data table, the master node DPU obtains the storage space required to store the target data table and obtains the remaining memory of its own memory. Further, the master node DPU compares the storage space required to store the target data table with the remaining memory of the master node DPU. If the master node DPU determines that the storage space required to store the target data table is less than or equal to the remaining memory of the master node DPU, the master node DPU stores the target data table in the memory of the master node DPU.

[0047] In one example, if the target data table is the first data table, the master node DPU obtains the storage space required to store the first data table and obtains the remaining memory of its own memory. Further, the master node DPU compares the storage space required to store the first data table with the remaining memory of the master node DPU. If the master node DPU determines that the storage space required to store the first data table is less than or equal to the remaining memory of the master node DPU, the master node DPU stores the first data table in the memory of the master node DPU.

[0048] S203, the master node DPU broadcasts the target data table to the DPU memory of the slave nodes including the DPU.

[0049] In the embodiment of the present application, after the master node DPU stores the target data table in its own memory, the master node DPU writes the target data table into the DPU memory of the slave node DPU to complete the broadcast of the target data table.

[0050] In one example, if the target data table is the first data table, after the master node DPU stores the first data table in its own memory, the master node DPU writes the first data table into the DPU memory of the slave node DPU to complete the broadcast of the first data table.

[0051] In one example, when the master node DPU broadcasts the target data table to the slave node DPU, the master node DPU establishes a communication connection with the slave node DPU. Furthermore, based on the communication connection, the master node DPU can use remote direct data storage (RDMA) to write the target data packet directly into the memory of the slave node DPU, thereby completing the broadcast of the data table.

[0052] The data broadcasting method provided in the embodiment of the present application is that first, the DPU of the master node receives the table names of the first data table and the second data table sent by the CPU of the master node, determines the target data table with a smaller storage space required when storing the data table, and further, if the DPU determines that the storage space required to store the target data table is less than or equal to the remaining memory of the DPU, stores the target data table in the memory of the DPU; and broadcasts the target data table to the DPU memory of the slave node including the DPU to complete the data broadcast. The embodiment of the present application determines that the storage space required to store the target data table is less than or equal to the size relationship of the remaining memory of the DPU through the DPU. In the case where the target data table can be stored in the DPU memory, the target data table can be directly broadcasted to the DPU memory of the slave node through the master node DPU, thereby improving the efficiency of data broadcasting, thereby effectively avoiding the serialization operation required for the CPU to broadcast the data table.

[0053] Further, the master node DPU of the embodiment of the present application determines a target data table with a smaller storage space required for storing the data table in response to the table name of the first data table and the table name of the second data table sent by the master node CPU, including:

[0054] S2011, the master node DPU determines the storage space required for storing the first data table and the storage space required for storing the second data table according to the received table name of the first data table and the table name of the second data table.

[0055] In an embodiment of the present application, after receiving the table name of the first data table and the table name of the second data table, the master node DPU uses the table name of the first data table to perform data query in the database to determine the data volume of the first data table, and then determines the required storage space to store the first data table based on the data volume of the first data table. Similarly, the master node DPU uses the table name of the second data table to perform data query in the database to determine the data volume of the second data table, and then determines the required storage space to store the second data table based on the data volume of the second data table.

[0056] S2022: The master node DPU determines a target data table according to a storage space required for storing the first data table and a storage space required for storing the second data table.

[0057] In an embodiment of the present application, after the master node DPU determines the required storage space for storing the first data table and the required storage space for storing the second data table, the master node DPU compares the required storage space for storing the first data table with the required storage space for storing the second data table, determines the data table with smaller space required for storing the data table, and then determines the data table with smaller space required for storing the data table as the target data table.

[0058] In the embodiment of the present application, the master node DPU determines the required storage space for storing the first data table through the table name of the first data table, and determines the required storage space for storing the second data table through the table name of the second data table. Then, the master node DPU can determine the target data table through the required storage space for storing the first data table and the required storage space for storing the second data table. Therefore, when the master node DPU determines that the required storage space for storing the target data table is less than or equal to the remaining memory of the DPU, the target data table can be directly broadcast to the DPU memory of the slave node, thereby improving the efficiency of data broadcasting and effectively avoiding the serialization operation required for the CPU to broadcast the data table.

[0059] Furthermore, the DPU-based data broadcasting method of the embodiment of the present application may also include the following steps:

[0060] S204, after broadcasting the target data table to the DPU memory of the slave nodes including the DPU, the master node DPU sends a first instruction to the CPU for instructing the CPU to split the non-target data table into multiple sub-data tables and broadcast each sub-data table to different slave nodes.

[0061] In an embodiment of the present application, after the master node DPU broadcasts the target data table to the DPU memory of the slave node including the DPU, the master node DPU generates a first instruction and sends the first instruction to the master node CPU, wherein the first instruction is used to instruct the CPU to split the non-target data table into multiple sub-data tables, and then broadcast each sub-data table to different slave nodes, so that the slave nodes can use the first data table and the sub-data tables to perform a merge operation.

[0062] In one example, if the master node DPU determines that the target data table is the first data table, after the master node DPU broadcasts the first data table to the slave node DPU, the master node DPU generates a first instruction for instructing the CPU to split the second data table into multiple sub-data tables and then broadcast each sub-data table to different slave nodes, so that the slave nodes can use the first data table and the sub-data tables to perform merge operations.

[0063] After the master node DPU in the embodiment of the present application broadcasts the target data table to the DPU memory of the slave node including the DPU, by sending a first instruction to the master node CPU, the CPU can broadcast the non-target data table in a timely manner, so that the CPU and DPU can collaborate to complete the data broadcast, and the master node DPU can be used to improve the efficiency of data broadcast.

[0064] Furthermore, the DPU-based data broadcasting method of the embodiment of the present application may also include the following steps:

[0065] S205, the master node DPU aggregates the multiple merged data in response to the merged data sent by each slave node, and determines the aggregated merged data corresponding to the SQL.

[0066] In an embodiment of the present application, after the master node DPU sends the first instruction to the CPU, the master node DPU receives the merged data sent by each slave node, and performs aggregation operations on multiple merged data, aggregating the multiple merged data into aggregated merged data to obtain aggregated merged data corresponding to the SQL, thereby completing the merge operation in the SQL.

[0067] Specifically, in the embodiment of the present application, the merged data is determined by the slave node DPU based on the received target data table and sub-data table.

[0068] In the embodiment of the present application, the master node DPU receives the merged data sent by each slave node, aggregates the multiple merged data, and determines the aggregated merged data, which can improve the efficiency of data aggregation.

[0069] Furthermore, the embodiment of the present application provides a data broadcast method based on DPU, which is applied to the CPU of the master node in the distributed computing platform. The specific steps are as follows: Figure 3 As shown, including:

[0070] S301, the master node CPU parses the received SQL and determines the target operation of the SQL.

[0071] In the embodiment of the present application, after receiving the SQL configured by the user in the configuration interface, the master node CPU parses the received SQL and determines the target operation carried in the SQL.

[0072] Specifically, for example, the SQL configured by the user is:

[0073] Se lect name,score from studentjo in examon student.id=exam.id;

[0074] The CPU of the master node parses the SQL and determines that the target operation is a join operation.

[0075] S302: If the master node CPU determines that the target operation is a broadcast merge operator operation, it obtains the table name of the first data table and the table name of the second data table in the SQL.

[0076] In the embodiment of the present application, after determining the target operation, if the target operation is determined to be a broadcast merge operator operation, the master node CPU obtains the table name of the first data table and the table name of the second data table in the SQL.

[0077] Specifically, when the master node CPU determines that the target operation is a join operation, the master node CPU obtains the table name of the first data table and the table name of the second data table carried by the SQL. Taking the above SQL as an example, the table name of the first data table obtained by the master node CPU is: student, and the table name of the second data table is: exam

[0078] S303, the master node CPU sends the table name of the first data table and the table name of the second data table to the master node DPU.

[0079] In an embodiment of the present application, after determining the table name of the first data table and the table name of the second data table, the master node CPU sends the table name of the first data table and the table name of the second data table to the master node DPU, so that the master node DPU can use the table name of the first data table and the table name of the second data table to determine a target data table with a smaller storage space required when storing the data table. Furthermore, if the DPU determines that the required storage space for storing the target data table is less than or equal to the remaining memory of the DPU, the target data table is stored in the memory of the DPU; and the target data table is broadcast to the DPU memory of the slave nodes including the DPU to complete the data broadcast.

[0080] In the embodiment of the present application, the master node CPU parses the SQL, and when the target operation of the SQL is the broadcast merge operator operation, obtains the table name of the first data table and the table name of the second data table, and sends the table name of the first data table and the table name of the second data table to the master node DPU, so that the master node DPU can use the table name of the first data table and the table name of the second data table to determine the target data table with a smaller storage space required when storing the data table. Further, if the DPU determines that the storage space required to store the target data table is less than or equal to the remaining memory of the DPU, the target data table is stored in the memory of the DPU; and the target data table is broadcast to the DPU memory of the slave node including the DPU to complete the data broadcast. Therefore, when the master node DPU can determine that the storage space required to store the target data table is less than or equal to the remaining memory of the DPU, the target data table is directly broadcast to the DPU memory of the slave node, which improves the efficiency of data broadcasting, thereby effectively avoiding the serialization operation required for the CPU to broadcast the data table.

[0081] Furthermore, the DPU-based data broadcasting method of the embodiment of the present application may also include the following steps:

[0082] S304 , in response to the first instruction sent by the master node DPU, split the non-target data table into multiple sub-data tables.

[0083] In the embodiment of the present application, after receiving the first instruction sent by the master node DPU, the master node CPU responds to the first instruction and splits the non-target data table to split the target data table into multiple sub-data tables.

[0084] Specifically, the first instruction in the embodiment of the present application may also include the address of the slave node. The master node CPU may determine the number and address of the slave nodes based on the first instruction, and then the master node CPU may split the non-target data table based on the number of slave nodes to determine multiple sub-data tables, wherein the number of sub-data tables is the same as the number of slave nodes.

[0085] In one example, when splitting the non-target data table, the master node CPU may perform an even split based on the number of slave nodes, or may perform an uneven split based on the number of slave nodes.

[0086] S305, the master node CPU broadcasts each sub-data table to different slave nodes.

[0087] In the embodiment of the present application, after the master node CPU completes the splitting of the non-target data table, the master node CPU broadcasts each sub-data table to different slave nodes respectively, thereby completing the broadcasting of the data table.

[0088] After receiving the first instruction, the master node CPU of the embodiment of the present application splits the non-target data table by responding to the first instruction, so that the CPU and DPU can cooperate to complete the data broadcast, and then the master node DPU can be used to improve the efficiency of data broadcast.

[0089] Furthermore, the embodiment of the present application provides another data broadcasting method based on DPU, the specific steps are as follows: Figure 4 As shown, including:

[0090] The master node CPU receives the SQL configured by the user, parses the SQL, and determines the target operation of the SQL. When the master node CPU determines that the target operation is a broadcast merge operator operation, it obtains the table name of the first data table and the table name of the second data table in the SQL, and sends the table name of the first data table and the table name of the second data table to the master node DPU.

[0091] The master node DPU receives the table name of the first data table and the table name of the second data table, and uses the table name of the first data table to perform data query in the database to determine the data volume of the first data table, and then determine the required storage space to store the first data table based on the data volume of the first data table. At the same time, the table name of the second data table is used to perform data query in the database to determine the data volume of the second data table, and then determine the required storage space to store the second data table based on the data volume of the second data table.

[0092] After determining the required storage space for storing the first data table and the required storage space for storing the second data table, the master node DPU compares the required storage space for storing the first data table with the required storage space for storing the second data table to determine the data table with the smaller space required to store the data table, and then determines the data table with the smaller space required to store the data table as the target data table.

[0093] After the master node DPU determines the target data table, the master node DPU obtains the storage space required to store the target data table and obtains the remaining memory of its own memory. If the master node DPU determines that the required storage space to store the target data table is less than or equal to the remaining memory of the master node DPU, the master node DPU stores the target data table in the memory of the master node DPU.

[0094] After the master node DPU stores the target data table in its own memory, the master node DPU writes the target data table into the DPU memory of the slave node DPU to complete the broadcast of the target data table. After the master node DPU broadcasts the target data table to the DPU memory of the slave nodes including the DPU, the master node DPU generates a first instruction and sends the first instruction to the master node CPU.

[0095] After receiving the first instruction sent by the master node DPU, the master node CPU responds to the first instruction and splits the non-target data table to split the target data table into multiple sub-data tables. After the master node CPU completes the splitting of the non-target data table, the master node CPU broadcasts each sub-data table to different slave nodes respectively, thereby completing the broadcasting of the data table.

[0096] The master node DPU receives the merged data sent by each slave node, and performs aggregation operation on multiple merged data, aggregates the multiple merged data into aggregated merged data to obtain the aggregated merged data corresponding to the SQL, thereby completing the merge operation in the SQL.

[0097] Furthermore, the embodiment of the present application provides a DPU-based data broadcasting device, which is applied to the DPU of the master node in the distributed computing platform. The master node includes a CPU and the DPU connected to each other. Specifically, as shown in 5, the device includes:

[0098] A determination module 501 is used to determine a target data table that requires a smaller storage space when storing the data table in response to the table name of the first data table and the table name of the second data table sent by the CPU, wherein the table name of the first data table and the table name of the second data table are obtained in the SQL and sent to the DPU when the CPU determines that the received SQL includes a broadcast merge operator operation;

[0099] A storage module 502 is configured to store the target data table in the memory of the DPU if the storage space required to store the target data table is less than or equal to the remaining memory of the DPU; and

[0100] The broadcast module 503 is used to broadcast the target data table to the DPU memory of the slave node including the DPU.

[0101] The DPU-based data broadcasting device of this embodiment is used to implement the corresponding DPU-based data broadcasting methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the DPU-based data broadcasting device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0102] Reference Figure 6 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0103] like Figure 6 As shown, the electronic device may include: a processor (processor) 602 , a communication interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0104] in:

[0105] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0106] The communication interface 604 is used to communicate with other electronic devices or servers.

[0107] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps in the above data broadcast method embodiment.

[0108] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0109] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0110] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0111] The program 610 may be specifically configured to enable the processor 602 to perform the following operations:

[0112] In an optional implementation, the program 510 is also used to enable the processor 502 to implement each step in the program 610. The specific implementation can refer to the corresponding description of the corresponding steps and units in the above data broadcast method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0113] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform operations corresponding to any DPU-based data broadcasting method in the above-mentioned multiple method embodiments.

[0114] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0115] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an AS IC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the data broadcast method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the data broadcast method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the data broadcast method shown herein.

[0116] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0117] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A data broadcasting method based on DPU, characterized in that: A DPU applied to a master node in a distributed computing platform, wherein the master node includes a CPU and the DPU connected to each other, and the method includes: In response to the table name of the first data table and the table name of the second data table sent by the CPU, determine a target data table that requires less storage space when storing the data table, wherein the table name of the first data table and the table name of the second data table are obtained in the SQL and sent to the DPU when the CPU determines that the received SQL includes a broadcast merge operator operation; If the storage space required to store the target data table is less than or equal to the remaining memory of the DPU, storing the target data table in the memory of the DPU; and The target data table is broadcasted to the DPU memory of the slave nodes including the DPU.

2. The method according to claim 1, characterized in that The determining of a target data table occupying a smaller storage space in response to the table name of the first data table and the table name of the second data table sent by the CPU comprises: Determine, according to the received table name of the first data table and the table name of the second data table, a storage space required for storing the first data table and a storage space required for storing the second data table; The target data table is determined according to a storage space required for storing the first data table and a storage space required for storing the second data table.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: After broadcasting the target data table to the DPU memory of the slave node including the DPU, a first instruction is sent to the CPU to instruct the CPU to split the non-target data table into multiple sub-data tables and then broadcast each of the sub-data tables to different slave nodes.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to the merged data sent by each of the slave nodes, a plurality of the merged data are aggregated to determine the aggregated merged data corresponding to the SQL, wherein the merged data is generated by the slave node according to the received target data table and the sub-data table.

5. A data broadcasting method based on DPU, characterized in that: The method is applied to a CPU of a master node in a distributed computing platform, wherein the master node includes the CPU and the DPU connected to each other, and comprises: Parsing the received SQL to determine the target operation of the SQL, wherein the SQL is configured by the user in the configuration interface; If the target operation is a broadcast merge operator operation, obtain the table name of the first data table and the table name of the second data table in the SQL; The table name of the first data table and the table name of the second data table are sent to the DPU, wherein the table name of the first data table and the table name of the second data table are used by the DPU to determine a target data table requiring a smaller storage space when storing the data table.

6. The method according to claim 5, characterized in that The method further comprises: In response to a first instruction sent by the DPU, splitting a non-target data table into a plurality of sub-data tables; Each of the sub-data tables is broadcasted to different slave nodes.

7. A data broadcasting device based on DPU, characterized in that: A DPU applied to a master node in a distributed computing platform, wherein the master node includes a CPU and the DPU connected to each other, and the device includes: a determination module, configured to determine, in response to a table name of a first data table and a table name of a second data table sent by the CPU, a target data table requiring a smaller storage space when storing the data table, wherein the table name of the first data table and the table name of the second data table are obtained from the SQL and sent to the DPU when the CPU determines that the received SQL includes a broadcast merge operator operation; a storage module, configured to store the target data table in the memory of the DPU if the storage space required to store the target data table is less than or equal to the remaining memory of the DPU; and A broadcast module is used to broadcast the target data table to the DPU memory of the slave node including the DPU.

8. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the DPU-based data broadcasting method as described in any one of claims 1-6.

9. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the DPU-based data broadcasting method as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the DPU-based data broadcasting method as described in any one of claims 1 to 6.

Images