Data computation method and apparatus, and electronic device

Abstract

The present invention provides a data computation method and apparatus and an electronic device, where a hardware device is connected to a service device and is provided with a plurality of storage access ports. The method comprises: the service device acquiring a plurality of pieces of computation column data and the byte counts respectively corresponding to the plurality of pieces of computation column data, and for any piece of computation column data, sending the computation column data and the byte count corresponding to the computation column data to the hardware device by means of a target storage access port; and for any piece of computation column data, the hardware device receiving the computation column data and the byte count corresponding to the computation column data by means of the target storage access port, and determining computation results on the basis of the plurality of pieces of computation column data and the byte counts respectively corresponding to the plurality of pieces of computation column data. On the basis of the plurality of storage access ports, the hardware device can perform parallel reading of the data sent by the service device to obtain corresponding computation results, thereby effectively improving computational efficiency during subsequent data computation while effectively improving data reading efficiency.

Description

Data calculation method, device and electronic equipment Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data calculation method, device and electronic equipment. Background Art

[0002] Databases, particularly relational databases, are the primary technology for storing, accessing, and processing large amounts of data across various industries today. With the rapid development of emerging technologies such as the Internet of Things, data is growing at an exponential rate. This results in low efficiency when hardware devices read data from relational databases, which in turn leads to low computational efficiency when performing subsequent computations on the data. Consequently, traditional data computation methods are increasingly unable to meet application requirements.

[0003] Therefore, how to improve data reading efficiency has become an urgent problem to be solved. Summary of the Invention

[0004] The present invention provides a data calculation method, device and electronic device, wherein the hardware device can read data sent by a service device in parallel based on multiple storage access ports to obtain corresponding calculation results. This effectively improves the data reading efficiency while also effectively improving the calculation efficiency of subsequent calculations performed on the data.

[0005] In a first aspect, the present invention provides a data calculation method applied to a hardware device, the hardware device having multiple storage access ports, the hardware device being connected to a service device, the method comprising:

[0006] In the process of receiving the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data issued by the service device, for any calculated column data, the calculated column data and the number of bytes corresponding to the calculated column data are received through a target storage access port; wherein the target storage access port is a port among the plurality of storage access ports, and any two calculated column data respectively correspond to different target storage access ports;

[0007] A calculation result is determined according to the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data.

[0008] According to a data calculation method provided by the present invention, a calculation result is determined based on the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data, including: determining the number of data rows corresponding to all the calculation column data based on the number of bytes corresponding to each of the multiple calculation column data; for any calculation column data, reading the calculation column data according to the number of data rows through the target storage access port to obtain target row data; and calculating the target row data to obtain a calculation result.

[0009] According to a data calculation method provided by the present invention, the calculation column data is read according to the data row number through the target storage access port to obtain the target row data; and the target row data is calculated to obtain a calculation result, including: reading the calculation column data according to the data row number through the target storage access port to obtain the first target row data; S1, reading the calculation column data according to the data row number to obtain the second target row data, and calculating the first target row data to obtain a first calculation result, wherein the first row data of the second target row data is adjacent to the last row data of the first target row data; determining the second target row data as the new first target row data; repeating step S1 until the target row data is not read, and determining the calculation result corresponding to the last target row data.

[0010] According to a data calculation method provided by the present invention, the number of data rows corresponding to all the calculation column data is determined based on the number of bytes corresponding to each of the multiple calculation column data, including: obtaining the width of any storage access port among the multiple storage access ports; determining the maximum number of bytes from the number of bytes corresponding to each of the multiple calculation column data; and determining the number of data rows corresponding to all the calculation column data based on the width and the maximum number of bytes.

[0011] According to a data calculation method provided by the present invention, the number of data rows corresponding to all the calculated column data is determined based on the width and the maximum number of bytes, including: when the width is greater than or equal to the maximum number of bytes, the integer division result of the width and the maximum number of bytes is determined as the number of data rows corresponding to all the calculated column data; when the width is less than the maximum number of bytes, 1 is determined as the number of data rows.

[0012] In a second aspect, the present invention provides a data calculation method, applied to a service device, the service device being connected to a hardware device, the hardware device being provided with a plurality of storage access ports, the method comprising:

[0013] Obtain multiple calculated column data and the number of bytes corresponding to each of the multiple calculated column data;

[0014] For any calculated column data, the calculated column data and the number of bytes corresponding to the calculated column data are sent to the hardware device through the target storage access port. The calculated column data and the number of bytes are used by the hardware device to determine the calculation result.

[0015] The target storage access port is a port among the multiple storage access ports, and any two calculated column data items correspond to different target storage access ports.

[0016] According to a data calculation method provided by the present invention, the acquisition of multiple calculation column data includes: converting the format of all stored row data to obtain column data; and determining the multiple calculation column data required to be calculated by the hardware device from the column data.

[0017] In a third aspect, the present invention further provides a data computing device, which is applied to a hardware device, wherein the hardware device is provided with a plurality of storage access ports, and the hardware device is connected to a service device, and the device comprises:

[0018] a transceiver module configured to, during a process of receiving a plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data sent by the service device, receive the calculated column data and the number of bytes corresponding to each of the plurality of calculated column data through a target storage access port for any calculated column data; wherein the target storage access port is a port among the plurality of storage access ports, and any two calculated column data may correspond to different target storage access ports;

[0019] The processing module is configured to determine a calculation result according to the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data.

[0020] In a fourth aspect, the present invention further provides a data computing device, applied to a service device, wherein the service device is connected to a hardware device, and the hardware device is provided with multiple storage access ports, the device comprising:

[0021] An acquisition module, configured to acquire a plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data;

[0022] a transceiver module, configured to send, for any calculated column data, the calculated column data and the number of bytes corresponding to the calculated column data to the hardware device through a target storage access port, wherein the calculated column data and the number of bytes are used by the hardware device to determine a calculation result;

[0023] The target storage access port is a port among the multiple storage access ports, and any two calculated column data items correspond to different target storage access ports.

[0024] The present invention also provides an electronic device, which is a service device or a hardware device. The electronic device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements the data calculation method described in the first aspect or the second aspect above.

[0025] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the data calculation method described in the first or second aspect above is implemented.

[0026] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the data calculation method as described in the first aspect or the second aspect is implemented.

[0027] The present invention provides a data calculation method, device, and electronic device, wherein a hardware device is connected to a service device, and the hardware device is provided with multiple storage access ports. The method obtains multiple calculation column data and the byte numbers corresponding to each of the multiple calculation column data through the service device; for any calculation column data, the calculation column data and the byte number corresponding to the calculation column data are sent to the hardware device through the target storage access port, and the calculation column data and the byte number are used by the hardware device to determine the calculation result; in the process of receiving the multiple calculation column data and the byte number corresponding to each of the multiple calculation column data sent by the service device, the hardware device receives the calculation column data and the byte number corresponding to each of the calculation column data through the target storage access port for any calculation column data; wherein the target storage access port is a port in the multiple storage access ports, and any two calculation column data correspond to different target storage access ports; and the calculation result is determined based on the multiple calculation column data and the byte number corresponding to each of the multiple calculation column data. In this method, the hardware device can read the data sent by the service device in parallel based on the multiple storage access ports to obtain the corresponding calculation result, so that while effectively improving the data reading efficiency, it can also effectively improve the calculation efficiency when performing subsequent calculations on the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] FIG1 is a flow chart of a data calculation method based on row-type storage data provided by the prior art;

[0030] FIG2 is a schematic diagram of a timing diagram of data processing by a hardware device provided by the prior art;

[0031] FIG3 is a second flow chart of a data calculation method based on row-type storage data provided by the prior art;

[0032] FIG4 is a second schematic diagram of a timing sequence of data processing by a hardware device provided by the prior art;

[0033] FIG5 is a flow chart of a data calculation method provided by the present invention;

[0034] FIG6 is a schematic diagram of one scenario of the data calculation method provided by the present invention;

[0035] FIG7 is a second schematic diagram of a scenario of the data calculation method provided by the present invention;

[0036] FIG8 is a timing diagram of the hardware device provided by the present invention processing target row data;

[0037] FIG9 is a schematic diagram of a structure of a data computing device provided by the present invention;

[0038] FIG10 is a second structural diagram of the data computing device provided by the present invention;

[0039] FIG11 is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0041] It should be noted that Figure 1 is a flow chart of a data calculation method based on row-based storage data, as provided by the prior art. As can be seen from Figure 1, data is transmitted between a host and a hardware device via a bus (Peripheral Component Interconnect Express 0, PCIe). The data stored in the host may include: A1, A2, A3, B1, B2, B3, C1, C2, C3, D1, D2, and D3. This data is stored in a row-based format on the host's storage medium (e.g., a hard drive / memory). This data is referred to as row-based storage data (abbreviated as row-based data).

[0042] It should be noted that, in combination with Figure 1, from the perspective of column data: A1, A2, A3 are one column of data; ..., D1, D2, D3 are one column of data, that is, there are 4 columns of data in Figure 1; from the perspective of row data: A1, B1, C1, D1 are one row of data; ..., A3, B1, C3, D3 are one column of data, that is, there are 3 rows of data in Figure 1.

[0043] The service device then reads the row-type storage data containing all fields and sends it as a parameter (argv) to the hardware device. The hardware device then receives and reads the parameter serially (i.e., through a storage access port (Interface)) to determine the corresponding row-type storage data. The hardware device then extracts the row-type storage data and calculates the extracted calculation fields to obtain the corresponding calculation results. Finally, the hardware device returns the calculation results to the service device, which then performs subsequent processing on them. The calculation fields may include: A1, A2, A3, D1, D2, and D3.

[0044] It should be noted that the hardware device reads row-stored data in a serial manner, and the subsequent process of performing calculations on calculated fields is as follows:

[0045] Step 1: The hardware device reads all data {A1, B1, C1, D1} in the first row of all row-stored data through the storage access port. Then, the hardware device parses the data {A1, B1, C1, D1} according to the row storage format to obtain the calculated field {A1, D1}. Next, the hardware device performs a calculation on the calculated field {A1, D1} to obtain the corresponding calculation result.

[0046] Step 2: The hardware device reads all the data {A2, B2, C2, D2} in the second row of all the row-stored data through the above-mentioned storage access port; then, parses the data {A2, B2, C2, D2} according to the row storage format to obtain the calculated field {A2, D2}; then, performs calculation on the calculated field {A2, D2} to obtain the corresponding calculation result.

[0047] Step 3: The hardware device reads all the data {A3, B3, C3, D3} in the third row of all the row-stored data through the above-mentioned storage access port; then, the hardware device parses the data {A3, B3, C3, D3} according to the row storage format to obtain the calculated field {A3, D3}; then, the hardware device performs a calculation on the calculated field {A3, D3} to obtain the corresponding calculation result.

[0048] Based on the above steps 1 to 3, as shown in Figure 2, it is a timing diagram of the hardware device processing data provided by the prior art. As can be seen from Figure 2, the hardware device reads, parses and calculates the row-type stored data line by line.

[0049] Combining Figures 1 and 2, we can conclude that the existing data calculation method based on row-based storage data has the following shortcomings: 1. There is redundant data (such as {B1, C1, B2, C2, B3, C3}) in the data transmitted from the service device to the hardware device, that is, the transmitted data contains column data that is not required for calculation, resulting in a waste of transmission bandwidth; 2. The data received by the hardware device contains data that is not required for calculation, resulting in a waste of storage space resources on the device side; 3. The hardware device only reads data serially from one storage access port, resulting in low data reading efficiency; 4. The hardware device is not suitable for parsing data in row storage format, resulting in the hardware device's performance not being fully utilized.

[0050] Figure 3 is a flow chart of a data calculation method based on row-based data storage, as provided by the prior art. As can be seen from Figure 3, the service device and the hardware device transmit data via a bus. The data stored in the service device may include: A1, A2, A3, B1, B2, B3, C1, C2, C3, D1, D2, and D3. This data is stored in a row-based format on the storage medium of the service device.

[0051] The service device can then use a format conversion tool to convert row-based data into column-based data (referred to as columnar data). The service device then determines the required calculated column data from all the column-based data and sends all the calculated column data as a parameter (e.g., the first calculated column data corresponds to parameter argv0, and the second calculated column data corresponds to parameter argv1) to the hardware device. The hardware device then serially receives and reads these parameters to determine the corresponding calculated column data. The hardware device then performs calculations on each of these calculated column data to obtain the corresponding calculation results. Finally, the hardware device returns all the calculation results to the service device, which then performs subsequent processing on all of them. Parameters can also be referred to as arrays. The calculated column data can include: first calculated column data (column A data) as {A1, A2, A3}, and second calculated column data (column D data) as {D1, D2, D3}.

[0052] The hardware device serially reads computed column data and subsequently executes the corresponding pipeline calculations as follows: First, the hardware device uses the Advanced eXtensible Interface (AXI) as the storage access port. AXI has a fixed width. Assuming this width is 32 bytes, the first computed column data field width is 8 bytes, and the second computed column data field width is 16 bytes. In this case, the number of data rows read by the hardware device in a single read is N = 32 / max (8, 16) = 2.

[0053] Based on this, step 1: the hardware device reads the data {A1, A2} in the first calculation column data according to the number of data rows through the storage access port.

[0054] Step 2: The hardware device reads the data {D1, D2} in the second calculation column data according to the data row number through the above storage access port.

[0055] Step 3: The hardware device performs calculations on the read data {A1, A2} and data {D1, D2}. Simultaneously, it continues to read unread data in the calculated column data through the aforementioned storage access port, streamlining the calculation and reading operations. Calculation refers to operations such as expression calculation or filtering on fields in the database record (i.e., the read data), performed in rows. Specifically, the calculation is as follows:

[0056] (1) Read the data {A3} in the first calculation column data through the above storage access port;

[0057] (2) Calculate the data {A1, D1} to obtain the first calculation result;

[0058] (3) Perform calculation on the data {A2, D2} to obtain the second calculation result;

[0059] (4) Read the data {D3} in the second calculation column data through the above storage access port;

[0060] (5) Perform calculations on the third row of data, that is, data {A3} and data {D3}, to obtain the third calculation result.

[0061] Based on steps 1 through 3 above, Figure 4 shows a timing diagram of data processing by a hardware device based on columnar data storage, as provided by prior art. As can be seen from Figure 4, the hardware device serially reads the calculated column data and subsequently performs calculations on the corresponding pipelines for the calculated column data.

[0062] Combining Figures 3 and 4, we can conclude that the existing data calculation methods based on columnar storage data have the following shortcomings: 1. The hardware device reads data from only one storage access port, resulting in low storage access port utilization and low data reading efficiency; 2. The hardware device reads the calculation column data serially, resulting in the hardware device being unable to fully perform pipeline calculations, which in turn makes the calculation efficiency of the calculation column data low.

[0063] In summary, both the data calculation method shown in FIG1 and the data calculation method shown in FIG2 have the disadvantages of low data reading efficiency and low data calculation efficiency.

[0064] It should be noted that the service device and the hardware device involved in the embodiment of the present invention transmit data via a bus. The data stored in the service device is stored in a storage medium of the service device in a row storage format.

[0065] Among them, the service device can also be called a server host, which refers to a computer system that manages and transmits data information.

[0066] The hardware device may also be referred to as heterogeneous computing hardware. The hardware device is provided with multiple storage access ports, which can effectively improve the low data reading efficiency.

[0067] Optionally, the hardware device may include at least one of the following: a graphics processing unit (GPU), a field-programmable gate array (FPGA), and an application-specific integrated circuit (ASIC).

[0068] It should be noted that the application of hardware devices in relational databases can offload complex calculations to the hardware devices, which can effectively improve the query performance of the relational database.

[0069] It should be noted that the execution subject involved in the embodiment of the present invention can be a data computing device, or a service device and a hardware device.

[0070] The following further illustrates the embodiments of the present invention by taking a service device and a hardware device as examples.

[0071] FIG5 is a flow chart of the data calculation method provided by the present invention, which may include:

[0072] 501. The service device obtains multiple calculated column data and the number of bytes corresponding to each of the multiple calculated column data.

[0073] Calculated column data refers to the data used by hardware devices for calculations.

[0074] The number of bytes corresponding to the calculated column data refers to the length / size of the data type corresponding to the calculated column data. The definition of this data type in the data structure is a set of values ​​with the same properties and a set of operations defined on this value set.

[0075] Optionally, the number of bytes corresponding to different calculated column data may be the same or different, and is not specifically limited here.

[0076] In some embodiments, the service device obtains multiple calculated column data, which may include: the service device converts the format of all stored row data to obtain column data; and the service device determines the multiple calculated column data required to be calculated by the hardware device from the column data.

[0077] Since the data in the service device are all stored in the storage medium of the service device in a row storage format, which is not conducive to subsequent calculations of the hardware device, the service device can first convert the format of all stored row data to obtain the column data corresponding to all the row data; and since the column data may contain data that the hardware device does not need to calculate, in order to improve the data calculation efficiency of the hardware device, the service device can determine the calculated column data from the column data, and the number of the calculated column data is multiple.

[0078] 502. For any calculated column data, the service device sends the calculated column data and the number of bytes corresponding to the calculated column data to the hardware device through the target storage access port.

[0079] In the process of receiving multiple calculated column data and the byte numbers corresponding to the multiple calculated column data sent by the service device, for any calculated column data, the hardware device receives the calculated column data and the byte number corresponding to the calculated column data through the target storage access port.

[0080] The target storage access port is a port among the multiple storage access ports, and any two calculated column data items correspond to different target storage access ports.

[0081] To address the issue of low data reading efficiency caused by hardware devices reading data from only one storage access port, multiple storage access ports can be provided in the hardware device. This allows the service device to send multiple computed column data and the corresponding byte counts to the hardware device through as many storage access ports as there are computed column data. In other words, one computed column data item corresponds to one storage access port. In this case, the storage access port that performs the data transmission can be referred to as the target storage access port. This allows multiple target storage access ports in the hardware device to receive multiple computed column data items and the corresponding byte counts sent by the service device in parallel, effectively improving data reading efficiency.

[0082] In addition, since there is no redundant data in the multiple calculation column data sent by the service device, the waste of transmission bandwidth can be effectively avoided; and since the calculation column data received by the hardware device is the calculation data required by the hardware device, this can effectively avoid the waste of storage space resources on the device side.

[0083] 503. The hardware device determines a calculation result according to the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data.

[0084] Since the calculation column data received by the hardware device is the calculation data required by the hardware device, after receiving the multiple calculation column data and the byte numbers corresponding to each of the multiple calculation column data, the hardware device can determine the calculation results corresponding to each of the multiple calculation column data based on the multiple calculation column data and all the byte numbers.

[0085] In some embodiments, the hardware device determines the calculation result based on multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data, which may include: the hardware device determines the number of data rows corresponding to all the calculation column data based on the number of bytes corresponding to each of the multiple calculation column data; the hardware device reads the calculation column data according to the number of data rows through the target storage access port for any calculation column data to obtain the target row data; and calculates the target row data to obtain the calculation result.

[0086] The number of data rows refers to the number of rows read by the hardware device for the calculated column data, which can be represented by N.

[0087] After obtaining the byte counts corresponding to each of the multiple calculated column data, the hardware device may first determine the data row counts corresponding to all of the calculated column data based on all of the byte counts. Then, for any of the multiple calculated column data, the hardware device may read the calculated column data according to the data row counts through the corresponding storage access port, i.e., the target storage access port, to obtain the target row data and thereby determine the calculation result. Based on this, the hardware device may obtain the target row data corresponding to each of the multiple calculated column data and thereby obtain the corresponding calculation result.

[0088] In some embodiments, the hardware device reads the calculation column data according to the number of data rows through the target storage access port to obtain the target row data; and calculates the target row data to obtain the calculation result, which may include: the hardware device reads the calculation column data according to the number of data rows through the target storage access port to obtain the first target row data; the hardware device S1 reads the calculation column data according to the number of data rows to obtain the second target row data, and calculates the first target row data to obtain the first calculation result, wherein the first row data of the second target row data is adjacent to the last row data of the first target row data; the hardware device determines the second target row data as the new first target row data; repeats step S1 until the target row data is not read, and determines the calculation result corresponding to the last target row data.

[0089] When the hardware device reads the calculation column data according to the number of data rows through the target storage access port to obtain the target row data, it can first read the row data according to the calculation column data corresponding to the previously obtained number of data rows to obtain the first target row data; then, in step S1, the calculation column data is continued to be read according to the same number of data rows to obtain the second target row data, that is, the first row data in the second target row data is adjacent to the last row data in the first target row data. At this time, the first target row data can also be calculated to obtain the corresponding first calculation result; further, the hardware device uses the second target row data as the new first target row data, and repeats the above step S1 until the target storage access port no longer reads any data. At this time, the hardware device only needs to calculate the target row data read for the last time, and then determine the corresponding calculation result, which can effectively improve data calculation efficiency.

[0090] In some embodiments, the hardware device determines the number of data rows corresponding to all the calculation column data based on the number of bytes corresponding to each of the multiple calculation column data, which may include: the hardware device obtains the width of any storage access port among multiple storage access ports; the hardware device determines the maximum number of bytes from the number of bytes corresponding to each of the multiple calculation column data; the hardware device determines the number of data rows corresponding to all the calculation column data based on the width and the maximum number of bytes.

[0091] Since the hardware device is provided with multiple storage access ports and the width of each storage access port is the same, the hardware device can first obtain the width of any storage access port; then, the hardware device compares all the obtained byte numbers in pairs until the maximum byte number is determined. At this time, the hardware device can drive the number of data rows corresponding to all calculated column data based on the maximum byte number and the width.

[0092] It should be noted that there is no limitation on the timing of the hardware device obtaining the width of the storage access port and the hardware device determining the maximum number of bytes.

[0093] In some embodiments, the hardware device determines the number of data rows corresponding to all calculated column data based on the width and the maximum number of bytes, including: when the width is greater than or equal to the maximum number of bytes, the hardware device determines the integer division result of the width and the maximum number of bytes as the number of data rows corresponding to all calculated column data; when the width is less than the maximum number of bytes, the hardware device determines 1 as the number of data rows.

[0094] When the hardware device determines the number of data rows corresponding to all calculated column data based on the width and the maximum number of bytes, it can first compare the width with the maximum number of bytes: if the width is greater than or equal to the maximum number of bytes, then the width can be divided by the maximum number of bytes to obtain an integer divisibility result, and the integer divisibility result is determined as the number of data rows corresponding to all calculated column data; otherwise, only 1 needs to be determined as the number of data rows.

[0095] Assume that a hardware device receives two computed column data, first computed column data and second computed column data, and the storage access port width is 32 bytes. In Example 1, the field width of the first computed column data is 8 bytes, and the field width of the second computed column data is 16 bytes. Since both field widths are smaller than the width, and 16 bytes is the maximum number of bytes between the two field widths, the integer divisibility result N = 32 / 16 = 2 is obtained, and 2 is determined as the number of data rows.

[0096] Example 2: The field width of the first calculated column data is 8 bytes, and the field width of the second calculated column data is 24 bytes. It can be seen that since both field widths are smaller than the width, and 24 bytes is the maximum number of bytes in the two field widths, the integer division result N=32 / 24=1 can be obtained, and 1 is determined as the number of data rows.

[0097] Example 3: The field width of the first calculated column data is 8 bytes, and the field width of the second calculated column data is 48 bytes. It can be seen that 48 bytes is the maximum number of bytes between the two field widths. The 48 bytes are larger than the field width of the storage access port. At this time, 1 can be determined as the number of data rows.

[0098] In summary, as shown in Figures 6 and 7, which are exemplary scenarios of the data calculation method provided by the present invention, it can be seen from Figures 5, 6, and 7 that the service device can scan the storage medium to obtain row-type storage data, then convert the row-type storage data into column-type storage data, and then save it in a column-type storage file; then, the service device creates a column-type storage table and imports the column-type storage data in the column-type storage file into the column-type storage table; further, the service device reads and extracts the calculated column data in the column-type storage data from the column-type storage table, and sends the calculated column data to the hardware device.

[0099] When the hardware device reads the computed column data according to the number of rows through the target storage access port to obtain the target row data, it uses the Advanced Extensible Interface (AXI) as the storage access port. This AXI has a fixed width. Assuming this width is 32 bytes, the hardware device receives two computed column data items: the first computed column data item and the second computed column data item. The first computed column data item has a field width of 8 bytes, and the second computed column data item has a field width of 16 bytes. Since both field widths are smaller than the width, and 16 bytes is the maximum number of bytes between the two field widths, the number of rows read by the hardware device in a single pass is determined to be N = 32 / max (8, 16) = 2. This means that the hardware device reads two rows of data from the computed column data each time.

[0100] Based on this, step 1: the hardware device reads the first calculated column data and the second calculated column data in parallel through the target storage access port a1 and the target storage access port a2, as follows:

[0101] (1) Read the two rows of data {A1, A2} of the first calculated column data through the target storage access port a1;

[0102] (2) Read the two rows of data {D1, D2} of the second calculation column data through the target storage access port a2

[0103] Step 2: The hardware device performs calculations on the read data {A1, A2} and data {D1, D2}, respectively. Simultaneously, the hardware device reads the unread data in the first calculation column and the second calculation column in parallel through the target storage access port a1 and the target storage access port a2, so that the calculation and reading operations are pipelined, as follows:

[0104] (1) Read the data {A3} in the first calculation column data through the target storage access port a1;

[0105] (2) Read the data {D3} in the first calculation column data through the target storage access port a2;

[0106] (3) Perform calculation on the data {A1, D1} to obtain the first calculation result;

[0107] (4) Perform calculation on the data {A2, D2} to obtain the second calculation result;

[0108] Step 3: Calculate the third row of data, that is, data {A3} and data {D3}, to obtain a third calculation result.

[0109] Finally, the hardware device sends the three calculation results to the service device, so that the service device can perform subsequent processing on the three calculation results.

[0110] Based on steps 1 through 3 above, Figure 8 shows a timing diagram of data processing by the hardware device provided by the present invention. Figure 8 illustrates the parallel processing of computed column data by the hardware device and the subsequent pipeline execution of computations on the corresponding computed column data. Compared to Figures 2 and 4, this significantly improves data computation efficiency, thereby enhancing overall query performance.

[0111] In an embodiment of the present invention, a service device obtains multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data; for any calculation column data, the service device sends the calculation column data and the number of bytes corresponding to the calculation column data to the hardware device through the target storage access port; in the process of receiving the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data sent by the service device, for any calculation column data, the hardware device receives the calculation column data and the number of bytes corresponding to the calculation column data through the target storage access port; the hardware device determines the calculation result based on the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data. In this method, the hardware device can read the data sent by the service device in parallel based on multiple storage access ports to obtain the corresponding calculation result. This effectively improves the data reading efficiency while also effectively improving the calculation efficiency when performing subsequent calculations on the data.

[0112] The data calculation device provided by the present invention is described below. The data calculation device described below and the data calculation method described above can be referenced to each other.

[0113] FIG9 is a schematic diagram of the structure of a data computing device provided by the present invention. The data computing device is applied to a hardware device having multiple storage access ports. The hardware device is connected to a service device. The device includes:

[0114] The transceiver module 901 is configured to, during a process of receiving multiple calculated column data and the byte counts corresponding to the multiple calculated column data sent by the service device, receive the calculated column data and the byte counts corresponding to any calculated column data through a target storage access port; wherein the target storage access port is a port among the multiple storage access ports, and any two calculated column data may correspond to different target storage access ports;

[0115] The processing module 902 is configured to determine a calculation result according to the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data.

[0116] Optionally, the processing module 902 is specifically used to determine the number of data rows corresponding to all the calculation column data based on the number of bytes corresponding to each of the multiple calculation column data; for any calculation column data, read the calculation column data according to the number of data rows through the target storage access port to obtain the target row data; and calculate the target row data to obtain the calculation result.

[0117] Optionally, the processing module 902 is specifically used to read the calculation column data according to the data row number through the target storage access port to obtain the first target row data; S1, read the calculation column data according to the data row number to obtain the second target row data, and calculate the first target row data to obtain a first calculation result, wherein the first row data of the second target row data is adjacent to the last row data of the first target row data; determine the second target row data as the new first target row data; repeat step S1 until the target row data is not read, and determine the calculation result corresponding to the last target row data.

[0118] Optionally, the processing module 902 is specifically used to obtain the width of any storage access port among the multiple storage access ports; determine the maximum number of bytes from the number of bytes corresponding to each of the multiple calculation column data; and determine the number of data rows corresponding to all the calculation column data based on the width and the maximum number of bytes.

[0119] Optionally, the processing module 902 is specifically used to determine the integer division result of the width and the maximum number of bytes as the number of data rows corresponding to all the calculated column data when the width is greater than or equal to the maximum number of bytes; and to determine 1 as the number of data rows when the width is less than the maximum number of bytes.

[0120] FIG10 is a schematic diagram of the structure of a data computing device provided by the present invention. The data computing device is applied to a service device, which is connected to a hardware device having multiple storage access ports. The device includes:

[0121] An acquisition module 1001 is configured to acquire a plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data;

[0122] The transceiver module 1002 is configured to send any calculated column data and the number of bytes corresponding to the calculated column data to the hardware device through the target storage access port. The calculated column data and the number of bytes are used by the hardware device to determine a calculation result.

[0123] The target storage access port is a port among the multiple storage access ports, and any two calculated column data items correspond to different target storage access ports.

[0124] Optionally, the acquisition module 1001 is specifically configured to convert the format of all stored row data to obtain column data; and determine the plurality of calculated column data required to be calculated by the hardware device from the column data.

[0125] As shown in Figure 11, it is a structural diagram of the electronic device provided by the present invention. The electronic device is a service device or a hardware device. The electronic device may include: a processor (processor) 1110, a communication interface (Communications Interface) 1120, a memory (memory) 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 can call the logic instructions in the memory 1130 to execute the data calculation method, wherein the hardware device is connected to the service device, and the hardware device is provided with multiple storage access ports, and the method includes: the service device obtains multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data; for any calculation column data, the calculation column data and the number of bytes corresponding to the calculation column data are sent to the hardware device through the target storage access port, and the calculation column data and the number of bytes are used by the hardware device to determine the calculation result; in the process of receiving the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data sent by the service device, the hardware device receives the calculation column data and the number of bytes corresponding to the calculation column data for any calculation column data through the target storage access port; wherein the target storage access port is a port among the multiple storage access ports, and the target storage access ports corresponding to any two calculation column data are different; and the calculation result is determined according to the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data.

[0126] Furthermore, the logic instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0127] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data calculation method provided by the above methods, wherein a hardware device is connected to a service device, and the hardware device is provided with multiple storage access ports. The method includes: the service device obtains multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data; for any calculation column data, the hardware device sends the calculation column data and the number of bytes corresponding to the calculation column data to the hardware device through a target storage access port, and the calculation column data and the number of bytes are used by the hardware device to determine a calculation result; in the process of receiving the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data sent by the service device, the hardware device receives the calculation column data and the number of bytes corresponding to each of the calculation column data through a target storage access port for any calculation column data; wherein the target storage access port is a port among the multiple storage access ports, and any two calculation column data correspond to different target storage access ports; and the calculation result is determined based on the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data.

[0128] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data calculation method provided by the above-mentioned methods, wherein a hardware device is connected to a service device, and the hardware device is provided with multiple storage access ports, and the method includes: the service device obtains multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data; for any calculation column data, the calculation column data and the number of bytes corresponding to the calculation column data are sent to the hardware device through a target storage access port, and the calculation column data and the number of bytes are used by the hardware device to determine the calculation result; in the process of receiving the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data sent by the service device, the hardware device receives the calculation column data and the number of bytes corresponding to the calculation column data for any calculation column data through a target storage access port; wherein the target storage access port is a port among the multiple storage access ports, and the target storage access ports corresponding to any two calculation column data are different; and the calculation result is determined based on the multiple calculation column data and the number of bytes corresponding to each of the multiple calculation column data.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0130] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data calculation method, characterized in that: Applied to a hardware device, the hardware device is provided with a plurality of storage access ports, the hardware device is connected to a service device, and the method comprises: In the process of receiving the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data sent by the service device, for any calculated column data, the calculated column data and the number of bytes corresponding to the calculated column data are received through a target storage access port; wherein the target storage access port is a port among the plurality of storage access ports, and the target storage access ports corresponding to any two calculated column data are different; A calculation result is determined according to the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data.

2. The method according to claim 1, characterized in that The determining of the calculation result according to the plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data includes: Determine the number of data rows corresponding to all the calculated column data according to the number of bytes corresponding to each of the plurality of calculated column data; For any calculated column data, the calculated column data is read according to the data row number through the target storage access port to obtain target row data; and the target row data is calculated to obtain a calculation result.

3. The method according to claim 2, characterized in that The calculation column data is read according to the number of data rows through the target storage access port to obtain target row data; The target row data is calculated to obtain calculation results, including: The calculated column data is read according to the number of data rows through the target storage access port to obtain first target row data; S1. Read the calculation column data according to the number of data rows to obtain second target row data, and calculate the first target row data to obtain a first calculation result, wherein the first row data of the second target row data is adjacent to the last row data of the first target row data; The second target row data is determined as the new first target row data; step S1 is repeatedly executed until no target row data is read, and the calculation result corresponding to the last target row data is determined.

4. The method according to claim 2 or 3, characterized in that: The determining the number of data rows corresponding to all the calculated column data according to the number of bytes corresponding to each of the plurality of calculated column data includes: Obtaining a width of any storage access port among the plurality of storage access ports; Determine a maximum number of bytes from the number of bytes corresponding to each of the plurality of calculated column data; The number of data rows corresponding to all the calculated column data is determined according to the width and the maximum number of bytes.

5. The method according to claim 4, characterized in that The determining, according to the width and the maximum number of bytes, the number of data rows corresponding to all the calculated column data includes: In the case where the width is greater than or equal to the maximum number of bytes, a result of dividing the width by the maximum number of bytes is determined as the number of data rows corresponding to all the calculated column data; In the case where the width is smaller than the maximum number of bytes, 1 is determined as the number of data rows.

6. A data calculation method, characterized in that: Applied to a service device, the service device is connected to a hardware device, the hardware device is provided with a plurality of storage access ports, the method comprises: Obtaining multiple calculated column data and the number of bytes corresponding to each of the multiple calculated column data; For any calculated column data, the calculated column data and the number of bytes corresponding to the calculated column data are sent to the hardware device through the target storage access port, and the calculated column data and the number of bytes are used by the hardware device to determine the calculation result; The target storage access port is a port among the multiple storage access ports, and any two calculated column data respectively correspond to different target storage access ports.

7. The method according to claim 6, characterized in that The obtaining of multiple calculated column data includes: Convert the format of all stored row data to obtain column data; The plurality of calculated column data required to be calculated by the hardware device are determined from the column data.

8. A data computing device, characterized in that: Applied to a hardware device, the hardware device is provided with a plurality of storage access ports, the hardware device is connected to a service device, and the apparatus comprises: a transceiver module, configured to receive the calculated column data and the number of bytes corresponding to the calculated column data for any calculated column data through a target storage access port during the process of receiving the plurality of calculated column data and the number of bytes corresponding to the plurality of calculated column data respectively sent by the service device; wherein the target storage access port is a port among the plurality of storage access ports, and the target storage access ports corresponding to any two calculated column data are different; The processing module is used to determine the calculation result according to the plurality of calculation column data and the number of bytes corresponding to each of the plurality of calculation column data.

9. A data computing device, characterized in that: Applied to a service device, the service device is connected to a hardware device, the hardware device is provided with a plurality of storage access ports, and the device comprises: An acquisition module, used to acquire a plurality of calculated column data and the number of bytes corresponding to each of the plurality of calculated column data; a transceiver module, configured to send, for any calculated column data, the calculated column data and the number of bytes corresponding to the calculated column data to the hardware device through a target storage access port, wherein the calculated column data and the number of bytes are used by the hardware device to determine a calculation result; The target storage access port is a port among the multiple storage access ports, and any two calculated column data respectively correspond to different target storage access ports.

10. An electronic device, the electronic device being a service device or a hardware device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data calculation method according to any one of claims 1 to 7 is implemented.

Images