Data processing method, device, equipment and storage medium

Abstract

The present application discloses a data processing method, device, equipment and storage medium. The data processing method calculates the hash bucket value of each row through the row data of the row number where each column data is located in the column database, and then determines the row number set with the same hash bucket value according to the hash bucket value of each row, and then sends the column data corresponding to each row number in the row number set with the same hash bucket value to the hash bucket corresponding to the row number set. According to the data processing method provided in the embodiment of the present application, the single instruction stream multiple data stream technology can be applied to the Shuffle process of the large data processing system, which greatly improves the execution efficiency of the Shuffle process and improves the data calculation performance of the large data processing system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of Internet technology, the amount of network data has exploded, and large-scale data processing systems have emerged. In order to reduce the time of data processing, in large-scale data processing systems, the operation of some data will involve data redistribution, namely the Shuffle process. The Shuffle process is to collect the same type of data on each node to a certain node for calculation, and collect these data distributed on different nodes together according to certain rules to perform data processing based on these data. However, the time taken by the Shuffle process usually accounts for a large proportion, so how to improve the execution efficiency of the Shuffle process in large-scale data processing systems has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, apparatus, device and storage medium, which can improve the execution efficiency of the Shuffle process in a large data processing system and improve the data computing performance of the large data processing system.

[0004] According to a first aspect of an embodiment of the present application, there is provided a data processing method, including:

[0005] Get at least one column of data in the column database;

[0006] Calculate the hash bucket value of each row according to the row number of each column of data in at least one column of data;

[0007] According to the hash bucket value of each row, determine the row number set with the same hash bucket value;

[0008] The target data is sent to the hash bucket corresponding to the row number set, where the target data is the column data corresponding to each row number in the row number set having the same hash bucket value.

[0009] According to a second aspect of an embodiment of the present application, there is provided a data processing device, including:

[0010] An acquisition module, used for acquiring at least one column of data in a column database;

[0011] A calculation module, used to calculate the hash bucket value of each row according to the row data of the row number of each column of data in at least one column of data;

[0012] A determination module, used for determining a set of row numbers having the same hash bucket value according to the hash bucket value of each row;

[0013] The sending module is used to send target data to the hash bucket corresponding to the row number set, where the target data is the column data corresponding to each row number in the row number set having the same hash bucket value.

[0014] According to a third aspect of an embodiment of the present application, there is provided a computer device, including: a memory and a processor;

[0015] A memory for storing a computer program;

[0016] The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the steps of the data processing method shown in the first aspect.

[0017] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a computer device, the computer device executes the steps of the data processing method shown in the first aspect.

[0018] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program, which, when executed by a computer device, enables the computer device to perform the steps of the data processing method shown in the first aspect.

[0019] According to the data processing method, apparatus, device and storage medium in the embodiments of the present application, the hash bucket value of each row is calculated through the row data of the row number where each column data is located in the column database, and then, according to the hash bucket value of each row, the row number set with the same hash bucket value is determined, and then, the column data corresponding to each row number in the row number set with the same hash bucket value is sent to the hash bucket corresponding to the row number set. In this way, multiple hash buckets are controlled by the big data system, and corresponding operations are performed on the column data corresponding to each row number set in the multiple row number sets at the same time, thereby realizing data parallel processing. Therefore, the data processing method in the embodiments of the present application is to apply the single instruction stream multiple data stream technology to the Shuffle process of the big data processing system, which greatly improves the execution efficiency of the Shuffle process and improves the data computing performance of the large data processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present application can be better understood from the following description of specific embodiments of the present application in conjunction with the accompanying drawings, wherein the same or similar figure numbers represent the same or similar features.

[0021] Figure 1 is a schematic diagram showing a data processing architecture according to an embodiment;

[0022] Figure 2 is a schematic diagram showing a data processing flow according to an embodiment;

[0023] Figure 3 is a flow chart showing a data processing method according to one embodiment;

[0024] Figure 4 is a flow chart showing a data processing method based on first data semantics according to an embodiment;

[0025] Figure 5 is a flow chart showing a data processing method based on second data semantics according to an embodiment;

[0026] Figure 6 is a schematic diagram showing the structure of a data processing device according to an embodiment;

[0027] Figure 7 FIG. 4 is a schematic diagram showing the hardware structure of a computer device according to an embodiment. DETAILED DESCRIPTION

[0028] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0029] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0030] In related technologies, big data processing systems usually use a Massively Parallel Processing (MPP) model or a Batch model in a neural network to process data. However, both the MPP model and the Batch model cannot do without data redistribution, that is, the Shuffle process. For example, some operations in a big data processing system require data on different computing nodes that perform the same task to be aggregated to a specific node for calculation. Here, the process of aggregating these data distributed on different computing nodes that perform the same task according to certain rules is called the Shuffle process. However, in the process of processing data in a big data processing system, the Shuffle process usually accounts for a large proportion of the time, resulting in the execution efficiency of the Shuffle process in a large data processing system, affecting the data computing performance of the large data processing system.

[0031] Based on this, the data processing method provided in the embodiment of the present application, by applying the Single Instruction Multiple Data (SIMD) technology to the Shuffle process, systematically proposes two Shuffle schemes based on the first data semantics, namely SIMD Gather semantics and the second data semantics, namely SIMD Scatter semantics, which greatly improves the execution efficiency of the Shuffle process in the big data processing system, and applies SIMD to the Shuffle process of the big data processing system, thereby improving and enhancing the data computing performance of the large data processing system. Here, SIMD refers to a technology that uses a controller to control multiple processors and simultaneously performs the same operation on each of a set of data (also called "data vectors") to achieve spatial parallelism.

[0032] Based on this, the following Figure 1 , the data processing architecture provided in the embodiment of the present application is described in detail.

[0033] In one or more possible embodiments, Figure 1 As shown, the data processing architecture 10 proposed in the embodiment of the present application may include a database 101, a large data processing system 102 and a bucketing module 103. The data processing architecture is described in detail below.

[0034] The database 101 is a database using column storage corresponding to OLAP, such as a column database, wherein the column database is used to store data based on column-based logical storage units, and the data in a column (referred to as column data in the embodiment of the present application) exists in a continuous storage form in the storage medium. The database 101 is used to store multiple columns of data, and a row number is set for each row corresponding to each column of data in the multiple columns of data. In one example, the data processing architecture 10 can obtain multiple columns of data corresponding to the task instruction from the column database 101 according to the task instruction when the task instruction is obtained. In another example, the column database 101 stores column data of at least two data types, such as column data of null values ​​(null) and non-null values. At this time, the data processing architecture 10 can obtain multiple columns of data corresponding to each data type from the column database 101 according to the task instruction when the task instruction is obtained, so as to process the multiple columns of data of each data type separately.

[0035] The large data processing system 102 is used to receive task instructions, and based on the received task instructions, perform data interaction with the large data processing system 102 and the bucketing module 103. In one example, the large data processing system 102 is used to obtain at least one column of data corresponding to the task instruction from the database 101. In another example, the large data processing system 102 is also used to calculate the hash bucket value of each row according to the row data of the row number of each column of data in at least one column of data, and determine the row number set with the same hash bucket value according to the hash bucket value of each row. Then, the large data processing system 102 is also used to send target data to the bucketing module 103, wherein the target data is the column data corresponding to each row number in the row number set with the same hash bucket value.

[0036] The bucketing module 103 is used to receive target data sent by the large data processing system 102 and process the target data in parallel based on multiple hash buckets in the bucketing module 103 .

[0037] Based on Figure 1 The data processing architecture shown below is combined with the attached Figure 2 Taking three columns of data as an example, the data processing method provided in the embodiment of the present application is described in detail.

[0038] like Figure 2 As shown, the large data processing system 102 receives a task instruction and obtains three columns of data, such as column C1 data, column C2 data, and column C3 data, from a column database according to the task instruction.

[0039] Next, the large data processing system 102 obtains the row data of the row number where each column of data in the C1 column data, C2 column data and C3 column data is located, such as when RowID is 0, C1 column data A, C2 column data 12 and C3 column data Null; similarly, when RowID is 1, C1 column data B, C2 column data 35 and C3 column data 0, as shown in the figure, and no further description is given here. And according to the row data of the row number where each column of data in the C1 column data, C2 column data and C3 column data is located, the hash bucket value of each row is calculated, such as hash bucket value 1 for RowID 0, hash bucket value 0 for RowID 1, and hash bucket value 3 for RowID 2.

[0040] Furthermore, the large data processing system 102 determines the row number set with the same hash bucket value according to the hash bucket value of each row. For example, when RowID is 1 and 6, the hash bucket value is 0, so the row number set with hash bucket value 0 is (1,6). Similarly, when RowID is 0 and 3, the hash bucket value is 1, so the row number set with hash bucket value 1 is (0,3). The same is true for other row number sets, which will not be described in detail here.

[0041] Then, the large data processing system 102 copies the target data and sends the target data to the bucketing module 103, wherein the target data is the column data corresponding to each row number in the row number set having the same hash bucket value. For example, the column data C1 column data B, C2 column data 35, and C3 column data 0 corresponding to row number 1 in the row number set (1,6) and the column data C1 column data G, C2 column data 1, and C3 column data Null corresponding to row number 6 can be sent to the hash bucket 0 corresponding to the hash bucket value 0.

[0042] In this way, by controlling multiple hash buckets through the big data system, the column data corresponding to each row number set in multiple row number sets are dispersed in parallel to multiple hash buckets, so that multiple hash buckets execute corresponding operations at the same time, so that after each hash bucket calculation is completed, the task result corresponding to the task instruction is obtained, thereby realizing parallel processing of data. Therefore, the data processing method in the embodiment of the present application is to apply the single instruction stream multiple data stream technology to the Shuffle process of the big data processing system, which greatly improves the execution efficiency of the Shuffle process and improves the data calculation performance of the large data processing system.

[0043] It should be noted that the data processing method provided in the embodiments of the present application can be applied to large-scale data processing systems such as online analytical processing (OLAP) systems, data warehouse systems, historical data management systems, data marts, and other big data computing or analysis platforms, to improve the execution efficiency of the Shuffle process and improve the data computing performance of large-scale data processing systems.

[0044] According to the above architecture and application scenarios, the following are combined Figure 3 The data processing method provided in the embodiment of the present application is described in detail.

[0045] Figure 3 is a flow chart showing a data processing method according to an embodiment.

[0046] like Figure 3 As shown, the data processing method can be applied to the data processing architecture, which may specifically include:

[0047] Step 310, obtain at least one column of data in the column database; step 320, calculate the hash bucket value of each row according to the row data of the row number of each column of data in at least one column of data; step 330, determine the row number set with the same hash bucket value according to the hash bucket value of each row; step 340, send the target data to the hash bucket corresponding to the row number set, the target data is the column data corresponding to each row number in the row number set with the same hash bucket value.

[0048] The above steps are described in detail below, as shown below.

[0049] Involving step 310, in one or more possible examples, step 310 may specifically include:

[0050] When a task instruction is received, at least one column of data corresponding to the character instruction is acquired from a column database based on the task instruction.

[0051] Involving step 320, in one or more possible examples, step 320 may specifically include:

[0052] According to the row data of the row number where each column of data in at least one column of data is located, a hash value of each row is calculated; based on the hash value of each row, a set of row numbers with the same hash bucket value is determined.

[0053] For example, reference Figure 2 , the row data with RowID 0 is [A,12,NUll], and the hash value of RowID 0 is 1; the row data with RowID 1 is [B,35,0], and the hash value of RowID 1 is 0; the row data with RowID 2 is [C,45,1], and the hash value of RowID 2 is 3, and so on. I will not go into details here.

[0054] It should be noted that the row data of each row in at least one column of data can be hashed, and the hash buckets can be determined by dividing the hash result by the number of buckets for modulo calculation, which ensures that there is data in each hash bucket, but does not limit the number of data items in each bucket.

[0055] Involving step 330, the embodiment of the present application can optimize the process of sending copied data to hash buckets through the first data semantics, namely SIMD Gather semantics and the second data semantics, namely SIMD Scatter semantics. The above two data semantics are described in detail below.

[0056] In one or more possible examples, SIMD optimization is based on the first data semantics, namely SIMD Gather semantics. Based on this, step 330 may specifically include steps 3301 to 3303.

[0057] Step 3301, obtaining column data corresponding to each row number in a row number set from at least one column of data.

[0058] Step 3302, copy the column data to the register according to the dimension of the register storing the data;

[0059] Step 3303: Send the target data in the register to the hash bucket corresponding to the row number set.

[0060] Furthermore, when the register is a single instruction stream multiple data stream SIMD register, the SIMD register includes a first data semantics, and the first data semantics includes a gather semantic function and a toArray semantic function, the above step 3302 can be implemented by the gather semantic function, and the above step 3303 can be implemented by the toArray semantic function, as shown below.

[0061] In one example, step 3302 may specifically include:

[0062] According to the dimension of register storage data, the gather semantic function is used to copy the column data to the register corresponding to the hash bucket value.

[0063] In another example, step 3303 may specifically include:

[0064] Through the toArray semantic function, the target data in the register is sent to the hash bucket corresponding to the row number set, and the hash bucket corresponding to the row number set is instructed to store the target data.

[0065] For example, Figure 4As shown in the figure, according to the row number set with the same hash bucket value, the real column data in at least one column of data is copied to the SIMD register through the gather semantic function. Here, it is assumed that the dimension of the register storage data of the SIMD register is 2, that is, one SIMD register can store 2 column data in each column. Then, the toArray semantic function is used to store the data in the SIMD register into the downstream hash bucket to complete the entire copy process.

[0066] Therefore, by optimizing the Shuffle process with SIMD Gather semantics, the Shuffle process can be made simple, intuitive and easy to implement, reducing the consumption of processing resources, and being highly adaptable to the current big data processing system.

[0067] In another or more possible examples, SIMD optimization is based on the first data semantics, namely SIMD Scatter semantics. Based on this, step 330 may specifically include steps 3304 to 3306.

[0068] Step 3304, copy at least one column of data to the register in the order of row numbers.

[0069] Step 3305, write the data in the register into the temporary memory according to the arrangement order of the hash bucket values.

[0070] Step 3306, copy the target data in the continuous memory, and send the target data to the hash bucket corresponding to the row number set.

[0071] Further, when the register is a single instruction stream multiple data stream SIMD register, the SIMD register includes the second data semantics, and the second data semantics may include a SIMD fromArray semantic function, a SIMD scatter semantic function, and a BatchCopy semantic function, the above step 3304 may be implemented by the SIMD fromArray semantic function, the above step 3305 may be implemented by the SIMD scatter semantic function, and the above step 3306 may be implemented by the BatchCopy semantic function, as shown in the following details.

[0072] Based on this, in one example, step 3304 may specifically include:

[0073] Through the SIMD fromArray semantic function, at least one column of data is copied to the register corresponding to the row number set according to the row number arrangement order and the dimension of the register storing the data.

[0074] In another example, step 3305 may specifically include:

[0075] Through the scatter semantic function, the data in the register is written to the temporary memory according to the arrangement order of the hash bucket values.

[0076] In yet another example, step 3306 may specifically include:

[0077] The target data in the continuous memory is copied through the BatchCopy semantic function, and the target data is sent to the hash bucket corresponding to the row number set.

[0078] For example, Figure 5 As shown, first, the SIMD fromArray semantic function is used to copy the real column data from the database to the SIMD register, then, the scatter semantic function is used to write the column data into a temporary memory, and then, a batch copy is performed again to copy the data in the temporary memory to the downstream hash bucket. Thus, the memory is read sequentially by the SIMD fromArray semantic function, making the memory access friendly. Although the scatter semantic function is a discrete write, the scope of discrete write can be limited by controlling the size of the temporary memory in the embodiment of the present application. If the temporary memory is small enough to be cached, the memory access can also be made friendly. Thus, by optimizing the Shuffle process through SIMD Gather semantics, the Shuffle process can be made memory-friendly, the execution efficiency of the Shuffle process is improved, and the data computing performance of large data processing systems is improved.

[0079] In summary, the data processing method provided by the embodiment of the present application can apply SIMD technology to the Shuffle process of the big data processing system, greatly improving the performance of the Shuffle process. In addition, by optimizing the Shuffle process through the first data semantics, namely SIMDGather semantics, the implementation is simple, and the implementation is simple in the big data processing system, which can be applied to the scenario with a small number of hash buckets; or, by optimizing the Shuffle process through the second data semantics, namely SIMD Scatter semantics, the Shuffle process can be made memory-friendly, thereby solving the problem of low memory access efficiency of the big data processing system, greatly improving the execution efficiency of the Shuffle process of the big data processing system, and improving the data computing performance of the large data processing system.

[0080] It should be noted that the dimension of the register storage data involved above may be determined in the following manner. Based on this, the data provided in the embodiment of the present application may also include:

[0081] According to the dimension factor, the dimension of the register storage data is calculated; where,

[0082] The dimension factor includes at least one of the following: the data type of the column data corresponding to each row number in the row number set, the number of row numbers in the row number set, and the number of different hash bucket values.

[0083] It should be clear that the present application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0084] Based on the same inventive concept, the present application provides a data processing device corresponding to the above-mentioned data processing method. Figure 6 Provide detailed explanation.

[0085] Figure 6 It is a schematic diagram showing the structure of a data processing device according to an embodiment.

[0086] like Figure 6 As shown, the data processing device 60 is used as follows Figure 1 In the data processing architecture shown, the data processing device 60 may specifically include:

[0087] An acquisition module 601 is used to acquire at least one column of data in a column database;

[0088] A calculation module 602 is used to calculate the hash bucket value of each row according to the row data of the row number of each column of data in at least one column of data;

[0089] A determination module 603 is used to determine a set of row numbers having the same hash bucket value according to the hash bucket value of each row;

[0090] The sending module 604 is used to send target data to the hash bucket corresponding to the row number set, where the target data is the column data corresponding to each row number in the row number set having the same hash bucket value.

[0091] Based on this, the data processing device 60 provided in the embodiment of the present application is described in detail below.

[0092] In one or more possible embodiments, the data processing device 60 provided in the embodiment of the present application may further include a first copy module; wherein,

[0093] The acquisition module 601 may also be used to acquire, from at least one column of data, column data corresponding to each row number in the row number set;

[0094] A first copy module, used for copying column data to the register according to the dimension of the register storing data;

[0095] The sending module 604 may also be configured to send target data in the register to the hash bucket corresponding to the row number set.

[0096] In another or more possible embodiments, the first copy module can be specifically used to copy column data to the register corresponding to the hash bucket value through the gather semantic function according to the dimension of the register storing data when the register is a single instruction stream multiple data stream SIMD register, the SIMD register includes the first data semantics, and the first data semantics includes a gather semantic function.

[0097] In yet another one or more possible embodiments, the sending module 604 can be specifically used to, when the first data semantics also includes a toArray semantic function, send the target data in the register to the hash bucket corresponding to the row number set through the toArray semantics, and instruct the hash bucket corresponding to the row number set to store the target data.

[0098] In one or more possible embodiments, the data processing device 60 provided in the embodiment of the present application may further include a second copy module and a writing module; wherein,

[0099] A second copy module, used for copying at least one column of data to a register in the order of row numbers;

[0100] The writing module is used to write the data in the register into the temporary memory according to the arrangement order of the hash bucket values;

[0101] The sending module 604 can be specifically used to copy the target data in the continuous memory and send the target data to the hash bucket corresponding to the row number set.

[0102] In yet another one or more possible embodiments, the second copy module can be specifically used to, when the register is a single instruction stream multiple data stream SIMD register, the SIMD register includes second data semantics, and the second data semantics includes a SIMDfromArray semantic function, copy at least one column of data to a register corresponding to a row number set according to the row number arrangement order and the dimension of the register storing data through the SIMD fromArray semantic function.

[0103] In yet another one or more possible embodiments, the write module can be specifically used to, when the second data semantics also includes a SIMD scatter semantic function, write the data in the register into a temporary memory through a scatter semantic function in the order of arrangement of the hash bucket values.

[0104] In yet another one or more possible embodiments, the sending module 604 can be specifically used to copy the target data in the continuous memory through the BatchCopy semantic function, and send the target data to the hash bucket corresponding to the row number set when the second data semantics also includes the BatchCopy semantic function.

[0105] In one or more possible embodiments, the data processing device 60 provided in the embodiment of the present application may further include a calculation module for calculating the dimension of the register storage data according to the dimension factor; wherein,

[0106] The dimension factor includes at least one of the following: the data type of the column data corresponding to each row number in the row number set, the number of row numbers in the row number set, and the number of different hash bucket values.

[0107] In this way, the data processing device provided in the embodiment of the present application can calculate the hash bucket value of each row through the row data of the row number of each column data in the column database, and then determine the row number set with the same hash bucket value according to the hash bucket value of each row, and then send the column data corresponding to each row number in the row number set with the same hash bucket value to the hash bucket corresponding to the row number set. In this way, multiple hash buckets are controlled by the big data system, and corresponding operations are performed on the column data corresponding to each row number set in multiple row number sets at the same time, thereby realizing data parallel processing. Therefore, the data processing method in the embodiment of the present application is to apply the single instruction stream multiple data stream technology to the Shuffle process of the big data processing system, which greatly improves the execution efficiency of the Shuffle process and improves the data calculation performance of the large data processing system.

[0108] Figure 7 FIG. 4 is a schematic diagram showing the hardware structure of a computer device according to an embodiment.

[0109] like Figure 7 As shown, the computer device 700 includes an input device 701 , an input interface 702 , a processor 703 , a memory 704 , an output interface 705 , and an output device 706 .

[0110] The input interface 702, the processor 703, the memory 704, and the output interface 705 are interconnected via the bus 710, and the input device 701 and the output device 706 are connected to the bus 710 via the input interface 702 and the output interface 705, respectively, and are further connected to other components of the computer device 700. Specifically, the input device 701 receives input information from the outside, and transmits the input information to the processor 703 via the input interface 702; the processor 703 processes the input information based on the computer executable instructions stored in the memory 704 to generate output information, stores the output information temporarily or permanently in the memory 704, and then transmits the output information to the output device 706 via the output interface 705; the output device 706 outputs the output information to the outside of the computer device 700 for use by the user.

[0111] In one embodiment, Figure 7 The computer device 700 shown may be implemented as a data processing device, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the data processing method described in the above embodiment.

[0112] In one embodiment, the memory can also be used for at least one column of data, a hash bucket value, a set of row numbers having the same hash bucket value, and a combination of the above Figures 1 to 5 Describe the computational results of each step in the data processing process.

[0113] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer-readable storage medium. For example, an embodiment of the present application includes a computer-readable storage medium, which includes storing a program or instruction on the computer-readable storage medium, and when the program or instruction is executed by a computer device, the computer device performs the steps of the above method.

[0114] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0115] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions, and when it is run on a computer, the computer executes the method described in each of the above embodiments. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid-state drives), etc.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, comprising: Get at least one column of data in the column database; Calculate the hash bucket value of each row according to the row data of the row number where each column of data in the at least one column of data is located; Determine a row number set having the same hash bucket value according to the hash bucket value of each row; From the at least one column of data, obtain the column data corresponding to each row number in the row number set; According to the dimension of the register storing the data, copy the column data to the register; Sending target data in the register to the hash bucket corresponding to the row number set, the target data being column data corresponding to each row number in the row number set having the same hash bucket value; The register is a single instruction stream multiple data stream SIMD register, the SIMD register includes a first data semantics, and the first data semantics includes a gather semantic function; the copying of the column data to the register according to the dimension of the register storing data includes: According to the dimension of the register storing the data, the column data is copied to the register corresponding to the hash bucket value through the gather semantic function.

2. The method according to claim 1, wherein: The first data semantics further includes a toArray semantic function; the sending the target data in the register to the hash bucket corresponding to the row number set includes: The target data in the register is sent to the hash bucket corresponding to the row number set through the toArray semantic function, and the hash bucket corresponding to the row number set is instructed to store the target data.

3. The method according to claim 1, wherein: The method further comprises: According to the dimension factor, the dimension of the register storage data is calculated; wherein, The dimension factor includes at least one of the following: the data type of the column data corresponding to each row number in the row number set, the number of row numbers in the row number set, and the number of different hash bucket values.

4. A data processing method, comprising: Get at least one column of data in the column database; Calculate the hash bucket value of each row according to the row data of the row number where each column of data in the at least one column of data is located; Determine a row number set having the same hash bucket value according to the hash bucket value of each row; Copying the at least one column of data to a register in the order of row numbers; Writing the data in the register into a temporary memory according to the arrangement order of the hash bucket values; Copy the target data in the continuous memory, and send the target data to the hash bucket corresponding to the row number set, wherein the target data is the column data corresponding to each row number in the row number set having the same hash bucket value; The register is a single instruction stream multiple data stream SIMD register, the SIMD register includes a second data semantics, and the second data semantics includes a SIMD fromArray semantic function; the copying of the at least one column of data to the register in the order of row numbers includes: By means of the SIMD fromArray semantic function, the at least one column of data is copied to a register corresponding to the row number set according to the row number arrangement order and the dimension of the register storing the data.

5. The method according to claim 4, wherein: The second data semantics also includes a SIMD scatter semantic function; the writing of the data in the register into a temporary memory according to the arrangement order of the hash bucket values ​​includes: Through the scatter semantic function, the data in the register is written into the temporary memory according to the arrangement order of the hash bucket values.

6. The method according to claim 4 or 5, wherein: The second data semantics also includes a BatchCopy semantic function; the copying of the target data in the continuous memory and sending the target data to the hash bucket corresponding to the row number set includes: The target data in the continuous memory is copied through the BatchCopy semantic function, and the target data is sent to the hash bucket corresponding to the row number set.

7. The method according to claim 4, wherein: The method further comprises: According to the dimension factor, the dimension of the register storage data is calculated; wherein, The dimension factor includes at least one of the following: the data type of the column data corresponding to each row number in the row number set, the number of row numbers in the row number set, and the number of different hash bucket values.

8. A data processing device, comprising: An acquisition module, used for acquiring at least one column of data in a column database; A calculation module, used to calculate the hash bucket value of each row according to the row data of the row number where each column of data in the at least one column of data is located; A determination module, configured to determine a set of row numbers having the same hash bucket value according to the hash bucket value of each row; A sending module, configured to send target data to the hash bucket corresponding to the row number set, wherein the target data is column data corresponding to each row number in the row number set having the same hash bucket value; The acquisition module is further used to acquire, from the at least one column of data, column data corresponding to each row number in the row number set; A first copy module, used for copying the column data to the register according to the dimension of the register storing the data; The sending module is further used to send the target data in the register to the hash bucket corresponding to the row number set; The first copy module can be specifically used to copy the column data to the register corresponding to the hash bucket value through the gather semantic function according to the dimension of the register storing data when the register is a single instruction stream multiple data stream SIMD register, the SIMD register includes the first data semantics, and the first data semantics includes a gather semantic function.

9. A data processing device, comprising: An acquisition module, used for acquiring at least one column of data in a column database; A calculation module, used to calculate the hash bucket value of each row according to the row data of the row number where each column of data in the at least one column of data is located; A determination module, configured to determine a set of row numbers having the same hash bucket value according to the hash bucket value of each row; A sending module, configured to send target data to the hash bucket corresponding to the row number set, wherein the target data is column data corresponding to each row number in the row number set having the same hash bucket value; A second copy module, used for copying the at least one column of data to a register according to the row number arrangement order; A writing module, used for writing the data in the register into a temporary memory according to the arrangement order of the hash bucket values; The sending module is specifically used to copy the target data in the continuous memory and send the target data to the hash bucket corresponding to the row number set; The second copy module is specifically used to, when the register is a single instruction stream multiple data stream SIMD register, the SIMD register includes second data semantics, and the second data semantics includes a SIMD fromArray semantic function, copy at least one column of data to a register corresponding to the row number set according to the row number arrangement order and the dimension of the register storing data through the SIMDfromArray semantic function.

10. A computer device comprising: Memory and processor, The memory is used to store a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the steps of the data processing method described in any one of claims 1 to 3 or any one of claims 4 to 7.

11. A computer-readable storage medium storing a program or instruction, wherein when the program or instruction is executed by a computer device, the computer device executes the steps of the data processing method as claimed in any one of claims 1 to 3 or any one of claims 4 to 7.

12. A computer program product, comprising a computer program, which enables a computer device to execute the steps of the data processing method according to any one of claims 1 to 3 or any one of claims 4 to 7 when the computer program is executed by the computer device.

Images