Data aggregation processing method, device, equipment and storage medium
By transferring data to the accelerator card for aggregation processing, the problem of high central processor occupancy rate in the database processing big data is solved, and the data aggregation function is uninstalled and processing speed is improved.
Patent Information
- Application Number
- CN202311659918.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-12-05
AI Technical Summary
When a database processes big data, the execution of the aggregate function by the central processor (CPU) leads to high CPU occupancy, affects the system response time, and may even cause system crashes.
By transferring the pending data set to the accelerator card for aggregation processing and writing the aggregation results to the memory of the target host, the data aggregation function is uninstalled, the utilization rate of the central processor is released, and the aggregation processing speed is improved.
Effectively reduces the occupancy of the central processor, provides space for it to handle other tasks, and improves the overall speed of aggregation processing.
Smart Images

Figure CN117807118B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a data aggregation processing method, device, equipment and storage medium. Background Art
[0002] With the development of computer technology, the continuous improvement of disk reading and writing and network bandwidth performance, the performance bottleneck in the database field has shifted to computing power. At present, the aggregation function of the database relies on the execution of the aggregation function by the central processing unit (CPU) architecture. When the amount of data is large, the CPU will generate a high CPU occupancy rate when aggregating the data, which will lead to a long response time for the computer system to perform other tasks, or even the inability to process other tasks. In severe cases, it will cause the system to crash. Summary of the invention
[0003] In order to solve the above technical problems, the present disclosure provides a data aggregation processing method, device, equipment and storage medium.
[0004] A first aspect of an embodiment of the present disclosure provides a data aggregation processing method, which is applicable to an accelerator card. The method includes:
[0005] Get the data set to be processed and the processing type of the aggregation processing to be performed by the target;
[0006] Performing aggregation processing on the data in the to-be-processed data set based on the processing type to obtain an aggregation result;
[0007] The aggregation result is written to the memory of the target host.
[0008] A second aspect of an embodiment of the present disclosure provides a data aggregation processing device, which is applicable to an accelerator card, and includes:
[0009] A data acquisition module is used to obtain the data set to be processed and the processing type of the aggregation processing performed by the target;
[0010] An aggregation module, used for performing aggregation processing on the data in the to-be-processed data set based on the processing type to obtain an aggregation result;
[0011] A writing module is used to write the aggregation result into the host memory connected to the acceleration card.
[0012] A third aspect of an embodiment of the present disclosure provides a computer device, including a memory and a processor, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the data aggregation processing method as described in the first aspect above is implemented.
[0013] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the data aggregation processing method as described in the first aspect above is implemented.
[0014] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0015] In the data aggregation processing method, apparatus, device and storage medium provided in the embodiments of the present disclosure, by obtaining a data set to be processed and a processing type of aggregation processing to be performed by a target, aggregation processing is performed on the data in the data set to be processed based on the processing type to obtain an aggregation result, and the aggregation result is written into the memory of the target host. The data set to be processed can be transferred to an acceleration card, and aggregation processing is performed on the data in the data set to be processed on the acceleration card, and the aggregation result is written into the memory of the target host, thereby realizing the unloading of the data aggregation function. Compared with the traditional method of performing data aggregation through a central processing unit, it not only releases the occupancy rate of the central processing unit and provides space for it to process other tasks, but also improves the overall speed of the aggregation processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0018] Figure 1 is a flow chart of a data aggregation processing method provided by an embodiment of the present disclosure;
[0019] Figure 2 is a flowchart of another data aggregation processing method provided by an embodiment of the present disclosure;
[0020] Figure 3 is a flowchart of another data aggregation processing method provided by an embodiment of the present disclosure;
[0021] Figure 4 is a flow chart of another data aggregation processing method provided by an embodiment of the present disclosure;
[0022] Figure 5 is a structural schematic diagram of a data aggregation processing device provided by an embodiment of the present disclosure;
[0023] Figure 6It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0025] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0027] Figure 1 1 is a flow chart of a data aggregation processing method provided by an embodiment of the present disclosure. The method can be executed by a data aggregation processing device, and the data aggregation processing device can be set in an acceleration card. Figure 1 As shown, the data aggregation processing method provided in this embodiment includes the following steps:
[0028] S101, obtaining a data set to be processed and a processing type of aggregation processing to be performed by a target.
[0029] The data set to be processed in the embodiment of the present disclosure can be understood as a set of data that needs to be aggregated.
[0030] The aggregation processing in the embodiments of the present disclosure can be understood as summary data derived from a large amount of basic data or factual data, which characterizes the overall characteristics of all data. By way of example, the processing types of aggregation processing may include summation processing, averaging processing, maximum value calculation, minimum value calculation, etc., which are not limited here.
[0031] In the embodiment of the present disclosure, the data aggregation processing device can obtain a data set to be processed consisting of data that needs to be aggregated, and at the same time obtain the processing type of the aggregation processing performed on these data targets.
[0032] In an exemplary implementation of the disclosed embodiment, the data aggregation processing device can obtain the data set to be processed and the processing type of the aggregation processing performed by the target from the memory of the target host. Specifically, the data aggregation processing device receives the data set to be processed and the processing type read and sent from the memory by the target host, and can also actively send a request to the target host to read the data set to be processed and the processing type from the memory. The target host is a host connected to the acceleration card.
[0033] In another exemplary implementation of the disclosed embodiment, the data aggregation processing device can obtain the data set to be processed and the processing type from the Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM) of the acceleration card. When the host has a data aggregation processing requirement, the data set to be processed and the processing type can be sent to the DDR SDRAM in the acceleration card. The data aggregation processing device can monitor the DDR SDRAM and obtain the newly written data therefrom when it is detected that there is newly written data in the DDR SDRAM.
[0034] S102: Aggregate the data in the data set to be processed based on the processing type to obtain an aggregation result.
[0035] In the embodiment of the present disclosure, the data aggregation processing device can obtain the processing type of the aggregation processing to be performed on the data set to be processed and the target, and then perform aggregation processing on the data in the data set to be processed according to the processing type, and obtain the aggregation result after the processing is completed.
[0036] In an exemplary implementation of an embodiment of the present disclosure, after obtaining the processing type, the data aggregation processing device can select a program code corresponding to the processing type from preset program codes, bring in the data contained in the data set to be processed to execute the program code, obtain the execution result, and determine the execution result as the aggregation result.
[0037] S103: Write the aggregation result into the memory of the target host.
[0038] The target host in the embodiment of the present disclosure may be understood as a host connected to the acceleration card. For example, the acceleration card may be plugged into the target host.
[0039] In the disclosed embodiment, after obtaining the aggregation result, the data aggregation processing device may send the aggregation result to the memory of the target host and write the aggregation result into the memory.
[0040] In an exemplary implementation of the present disclosure, the data aggregation processing device can write the aggregation result into the DDR SDRAM of the acceleration card after obtaining the aggregation result, and send a signal to the target host so that the target host obtains the aggregation result from the DDR SDRAM of the acceleration card and writes it into the memory.
[0041] The disclosed embodiment obtains a data set to be processed and a processing type of aggregation processing to be executed by a target, performs aggregation processing on the data in the data set to be processed based on the processing type, obtains an aggregation result, and writes the aggregation result into the memory of the target host. The data set to be processed can be transferred to an acceleration card, and aggregation processing is performed on the data in the data set to be processed on the acceleration card, and the aggregation result is written into the memory of the target host, thereby realizing the unloading of the data aggregation function. Compared with the traditional method of performing data aggregation through a central processing unit, it not only releases the occupancy rate of the central processing unit and provides space for it to process other tasks, but also improves the overall speed of the aggregation processing.
[0042] Figure 2 is a flowchart of another data aggregation processing method provided by an embodiment of the present disclosure. Figure 2 As shown, based on the above embodiment, data aggregation processing can be performed by the following method.
[0043] S201. Obtain the data type of the data contained in the data set to be processed.
[0044] The data types in the embodiments of the present disclosure may include INT32, INT64, UINT32, UINT64, DECIMAL32, DECIMAL64, DATE, etc., which are not limited here.
[0045] In the embodiment of the present disclosure, the data aggregation processing device can obtain the data type of the data contained in the data set to be processed while obtaining the data set to be processed, wherein the data type of all data contained in the data set to be processed is the same.
[0046] S202: Determine width information corresponding to the data type.
[0047] In the disclosed embodiment, the data aggregation processing device can further determine the width information of the data according to the data type after obtaining the data type of the data in the data set to be processed. For example, for data of types INT32, UINT32, DECIMAL32, and DATE, the corresponding bit width information is 32 bits (or 4 bytes); for data of types INT64, UINT64, and DECIMAL64, the corresponding bit width information is 64 bits (or 8 bytes).
[0048] S203: Determine the number of data to be processed at one time by the accelerator card based on the pre-acquired processing bandwidth and width information of the accelerator card.
[0049] The processing bandwidth in the embodiments of the present disclosure can be understood as the data width that the acceleration card can process at one time. For example, when the data aggregation processing device obtains the data set to be processed, the processing type and the data type from the DDR SDRAM of the acceleration card, it can be determined based on the configuration information of the DDR SDRAM, such as 2.4GHz main frequency, 64bit data width, and 300MHz read and write frequency. The processing bandwidth of the acceleration card is determined to be 2.4GHz×64bit / 300MHz=512bit.
[0050] In the disclosed embodiment, after obtaining the width information of the data in the data set to be processed, the data aggregation processing device can combine the pre-acquired processing bandwidth of the acceleration card to calculate the number of data processed at a single time by the acceleration card. Specifically, the quotient of the processing bandwidth and the width information can be calculated to obtain the number of data processed at a single time by the acceleration card. For example, for 32-bit data, the number of data processed at a single time by the acceleration card is 512bit / 32bit=16, and for 64-bit data, the number of data processed at a single time by the acceleration card is 512bit / 64bit=8.
[0051] S204: Group the data in the to-be-processed data set based on the number of data.
[0052] In the disclosed embodiment, the data aggregation processing device can calculate the number of data processed by the acceleration card at a single time, and then group the data in the order of the data in the data set to be processed according to the number of data, so that all groups except the last group contain data of this number of data.
[0053] S205: For each group, perform aggregation processing on the data in the group based on the processing type to obtain an aggregation result.
[0054] In the embodiment of the present disclosure, the data aggregation processing device can perform aggregation processing corresponding to the processing type according to the grouping after the grouping is completed to obtain the final aggregation result. The specific aggregation processing method is similar to S102 and will not be repeated here.
[0055] In an exemplary implementation of the disclosed embodiment, the data aggregation processing device can obtain the total number of data contained in the data set to be processed while obtaining the data set to be processed, and calculate the rounds to be processed by the acceleration card based on the total number of data and the number of data processed by the acceleration card at a single time. For example, for n 32-bit data, the number of data processed by the acceleration card each time is 16, and the round-down result of (n+15) / 16 can be calculated to obtain the processing round of the acceleration card, and based on the round, the grouped data is obtained, and aggregation processing is performed. After reaching the corresponding processing round, it is determined that the aggregation process is ended.
[0056] The disclosed embodiment obtains the data type of the data contained in the data set to be processed, determines the width information corresponding to the data type, determines the number of data to be processed by the accelerator card at a single time based on the processing bandwidth and width information of the accelerator card obtained in advance, groups the data in the data set to be processed based on the number of data, and aggregates the data in the group based on the processing type for each group to obtain an aggregated result. The data in the data set to be processed can be grouped and processed according to the actual processing capability of the accelerator card, and multiple data in a group can be processed at one time, thereby speeding up the overall processing speed.
[0057] Figure 3 is a flowchart of another method for data aggregation processing provided by an embodiment of the present disclosure, such as Figure 3 As shown, based on the above embodiment, data aggregation processing can be performed by the following method.
[0058] S301. Determine a target aggregate processing block corresponding to the data type and the processing type based on a first correspondence between the data type, the processing type and the aggregate processing block.
[0059] The aggregation processing block in the embodiment of the present disclosure may be understood as a pre-packaged execution unit for performing aggregation processing.
[0060] In an embodiment of the present disclosure, after determining the data type of the data in the data set to be processed and the processing type of the aggregation processing to be performed by the target, the data aggregation processing device can determine the target aggregation processing block corresponding to the data type of the data in the data set to be processed and the processing type of the aggregation processing to be performed by the target based on the first correspondence between the pre-acquired data type, the processing type and the aggregation processing block.
[0061] In an exemplary implementation of the disclosed embodiment, the first correspondence relationship may store the correspondence between the data type, the processing type and the identification information or the calling interface of the aggregation processing block, and the data aggregation processing device may determine the target identification information or the target calling interface of the target aggregation processing block according to the data type and the processing type.
[0062] S302: For each group, call the target aggregation processing block to perform aggregation processing on the data in the group to obtain a group aggregation result.
[0063] In the disclosed embodiment, the data aggregation processing device may call the target aggregation processing block to perform aggregation processing on the data in each group when performing aggregation processing on the data in the group, so as to obtain the group aggregation result corresponding to each group.
[0064] In an exemplary implementation of the disclosed embodiment, the data aggregation processing device may locate the target aggregation processing block through target identification information or a target call interface, and then send the data in the group to the target aggregation processing block to obtain an output group aggregation result.
[0065] S303: Determine the aggregation result corresponding to the data set to be processed based on the grouping aggregation result and the processing type.
[0066] In the embodiment of the present disclosure, after obtaining the group aggregation result corresponding to each group, the data aggregation processing device can determine the method of summarizing the group aggregation results based on the processing type, and then summarize the group aggregation results in this way to obtain the aggregation result corresponding to the data set to be processed.
[0067] In an exemplary implementation of an embodiment of the present disclosure, when the processing types of aggregation processing include summation processing, averaging processing, calculation of maximum value and calculation of minimum value, the number of aggregation processing blocks is 8, namely, summation processing for all data types, averaging processing for all data types, calculation of maximum value for signed 32-bit data (INT32 and DECIMAL32), calculation of minimum value for signed 32-bit data, calculation of maximum value for signed 64-bit data (INT64 and DECIMAL64), calculation of minimum value for signed 64-bit data, calculation of maximum value for unsigned data (UINT32, UINT64 and DATE), and calculation of minimum value for unsigned data.
[0068] The disclosed embodiment determines the target aggregation processing block corresponding to the data type and the processing type based on the first correspondence between the data type, the processing type and the aggregation processing block, and for each group, calls the target aggregation processing block to perform aggregation processing on the data in the group to obtain a group aggregation result, and determines the aggregation result corresponding to the data set to be processed based on the group aggregation result and the processing type, thereby realizing the unloading of the aggregation function for different data types and different aggregation types, releasing the occupancy rate of the central processing unit, providing space for it to process other tasks, and improving the overall speed of aggregation processing.
[0069] Figure 4FIG. 1 is a flowchart of another method for data aggregation processing provided by an embodiment of the present disclosure. Figure 4 As shown, based on the above embodiment, data aggregation processing can be performed by the following method.
[0070] S401 : Obtain identification information corresponding to each data in the data set to be processed, where the identification information is used to indicate whether the corresponding data is valid.
[0071] The identification information in the embodiments of the present disclosure can be understood as corresponding one-to-one with the data in the data set to be processed, and is used to characterize whether the corresponding data is valid, that is, whether it participates in aggregation processing. For example, the identification information can be Boolean data, and the values can be 1 and 0. When the value of the identification information is 0, the corresponding data is invalid and does not participate in aggregation processing. When the value of the identification information is 1, the corresponding data is valid and participates in aggregation processing.
[0072] In the embodiment of the present disclosure, the data aggregation processing device can obtain identification information corresponding to each data in the data set to be processed, indicating whether the data is valid, while obtaining the data set to be processed.
[0073] In an exemplary implementation of the embodiments of the present disclosure, the data aggregation processing device can obtain identification data corresponding to the data set to be processed while obtaining the data set to be processed, and then extract identification information corresponding to each data from the identification data. Specifically, the identification data is composed of 0 and 1, and the number of bits is the same as the total number of data contained in the data set to be processed. The 1st, 2nd, 3rd... data in the data set to be processed respectively correspond to the 1st, 2nd, 3rd... bit data in the identification data, and the identification information corresponding to each data can be extracted bit by bit from the identification data.
[0074] S402: Determine target filling data corresponding to the data type and the processing type based on a second corresponding relationship between the data type, the processing type and the filling data.
[0075] The fill data in the embodiments of the present disclosure can be understood as preset data used to replace invalid data and set to eliminate the impact of invalid data on the final aggregation result. Optionally, for summing and averaging processing, the fill data corresponding to data of all data types can be set to 0. For calculating the maximum value and calculating the minimum value, the fill data corresponding to each data type is the minimum value and maximum value within the value range of the current data type, respectively. For example, when performing the aggregation processing of calculating the maximum value for INT32 type data, the corresponding target fill data is the minimum value of the INT32 type data within the value range: -2 31 =-2147483648.
[0076] In an embodiment of the present disclosure, after determining the data type of the data in the data set to be processed and the processing type of the aggregation processing to be performed by the target, the data aggregation processing device can determine the target filling data corresponding to the data type of the data in the data set to be processed and the processing type of the aggregation processing to be performed by the target based on the second correspondence between the pre-acquired data type, processing type and filling data.
[0077] S403: For each group, replace the data in the group whose corresponding identification information is the first preset value with the target filling data.
[0078] The first preset value in the embodiment of the present disclosure may be understood as a value of identification information indicating that the corresponding data is invalid, such as 0.
[0079] In the disclosed embodiment, the data aggregation processing device may, after determining the target filling data corresponding to the data type and the processing type, replace the data in each group whose corresponding identification information is a first preset value with the target filling data.
[0080] S404: Call the target aggregation processing block to perform aggregation processing on the data in the replaced group.
[0081] In the disclosed embodiment, the data aggregation processing device may call a target aggregation processing block corresponding to the processing type to perform aggregation processing on the data in the group after replacing the data in the group whose corresponding identification information is a first preset value with the target filling data.
[0082] In an exemplary implementation of the disclosed embodiment, the data aggregation processing device may use a pipeline loop to execute the replacement steps of S403 and the calling steps of S404 for multiple groups. Specifically, the data aggregation processing device may execute the replacement step in S403 for the first group at a first moment, and after the invalid data in the first group is replaced, at a second moment, execute the step of calling the target aggregation processing block to perform aggregation processing on the data in the replaced first group, and at the same time, execute the replacement step in S403 for the second group at a second moment, and after the invalid data in the second group is replaced, at a third moment, execute the aggregation processing in S404 for the data in the replaced second group, and so on, until the replacement step in S403 is executed for the Nth group at the Nth moment, and at the same time, the aggregation processing in S404 is executed for the replaced data of the N-1th group at the Nth moment, and after the invalid data in the Nth group is replaced, at the N+1th moment, execute the aggregation processing in S404 for the replaced data of the Nth group.
[0083] The disclosed embodiment obtains identification information corresponding to each data in the data set to be processed, the identification information is used to characterize whether the corresponding data is valid, and determines the target filling data corresponding to the data type and the processing type based on the second corresponding relationship between the data type, the processing type and the filling data. For each group, the data in the group corresponding to the identification information with a first preset value is replaced with the target filling data, and the target aggregation processing block is called to perform aggregation processing on the data in the replaced group. When invalid data exists, the influence of the invalid data can be eliminated during the aggregation processing without adding additional screening steps, thereby further improving the speed of the aggregation processing.
[0084] In some embodiments of the present disclosure, the processing type of the aggregation processing includes at least one of summing processing, averaging processing, calculating a maximum value, and calculating a minimum value.
[0085] Specifically, for summation processing, the data aggregation processing device can convert all data into 64-bit data for calculation. For averaging processing, the average value can be calculated based on the total amount of data on the basis of the summation processing. When invalid data is included in the data set to be processed, the number of valid data can be counted, and the quotient of the summation result and the number of valid data can be calculated to obtain the average value. For aggregation processing of calculating the maximum value and calculating the minimum value, the new maximum / minimum value can be determined by comparing the maximum / minimum value of all previous data with the current data until the comparison with the last data is completed to obtain the final maximum / minimum value.
[0086] In some other embodiments of the present disclosure, the data aggregation processing device may traverse the identification information corresponding to each data in the data set to be processed, and count the number of data in the data set to be processed whose corresponding identification information is a second preset value.
[0087] Specifically, the second preset value can be understood as the value of the identification information that indicates that the corresponding data is valid, such as 1. Optionally, when the identification information corresponding to the invalid data is 0 and the identification information corresponding to the valid data is 1, the cumulative value of all the identification information can be calculated to obtain the valid data, that is, the number of data whose corresponding identification information is the second preset value; when the identification information corresponding to each data is extracted from the identification data, the number of valid data can be obtained by accumulating the identification information bit by bit.
[0088] In some other embodiments of the present disclosure, the acceleration card is connected to the target host via a high-speed serial computer expansion bus.
[0089] Specifically, the accelerator card has a standard high-speed serial computer expansion bus (Peripheral Component Interconnect express, PCIe) interface, which is inserted into the host device through a PCIe slot. The accelerator card and the host can transmit data through PCIe.
[0090] Figure 5 Schematic diagram of a data aggregation processing device provided by an embodiment of the present disclosure. Figure 5 As shown, the data aggregation processing device 500 includes: a data acquisition module 510, an aggregation module 520, and a writing module 530, wherein the data acquisition module 510 is used to obtain a data set to be processed and a processing type of aggregation processing to be performed by a target; the aggregation module 520 is used to perform aggregation processing on the data in the data set to be processed based on the processing type to obtain an aggregation result; and the writing module 530 is used to write the aggregation result into the host memory connected to the acceleration card.
[0091] Optionally, the aggregation processing device 500 also includes: a type acquisition module, used to obtain the data type of the data contained in the data set to be processed; the aggregation module 520 includes: a first determination unit, used to determine the width information corresponding to the data type; a second determination unit, used to determine the number of data processed by the acceleration card at a single time based on the pre-acquired processing bandwidth of the acceleration card and the width information; a grouping unit, used to group the data in the data set to be processed based on the number of data; and an aggregation unit, used to perform aggregation processing on the data in the group based on the processing type for each group to obtain the aggregation result.
[0092] Optionally, the aggregation unit includes: a first determination subunit, used to determine the target aggregation processing block corresponding to the data type and the processing type based on a first correspondence between the data type, the processing type and the aggregation processing block; an aggregation subunit, used to call the target aggregation processing block for each group to perform aggregation processing on the data in the group to obtain a group aggregation result; and a second determination subunit, used to determine the aggregation result corresponding to the data set to be processed based on the group aggregation result and the processing type.
[0093] Optionally, the aggregation unit also includes: an acquisition subunit, used to acquire identification information corresponding to each data in the data set to be processed, wherein the identification information is used to indicate whether the corresponding data is valid; a third determination subunit, used to determine the target filling data corresponding to the data type and the processing type based on a second correspondence between the data type, the processing type and the filling data; a replacement subunit, used to replace the data in the group whose corresponding identification information is a first preset value with the target filling data for each of the groups; and the aggregation subunit, specifically used to call the target aggregation processing block to perform aggregation processing on the data in the replaced group.
[0094] Optionally, the processing type of the aggregation processing includes at least one of summing processing, averaging processing, calculating a maximum value, and calculating a minimum value.
[0095] Optionally, the data aggregation processing device 500 further includes: a statistical module, configured to traverse the identification information corresponding to each data in the data set to be processed, and count the number of data in the data set to be processed whose corresponding identification information is a second preset value.
[0096] Optionally, the acceleration card is connected to the target host via a high-speed serial computer expansion bus.
[0097] The data aggregation processing device provided in this embodiment can execute the method described in any of the above embodiments. Its execution method and beneficial effects are similar and will not be repeated here.
[0098] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.
[0099] like Figure 6 As shown, the computer device may include a processor 610 and a memory 620 storing computer program instructions.
[0100] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0101] The memory 620 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 620 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 620 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 620 is a non-volatile solid-state memory. In a particular embodiment, the memory 620 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.
[0102] The processor 610 reads and executes the computer program instructions stored in the memory 620 to perform the steps of the data aggregation processing method provided in the embodiment of the present disclosure.
[0103] In one example, the computer device may further include a transceiver 630 and a bus 640. Figure 6 As shown, the processor 610, the memory 620 and the transceiver 630 are connected via a bus 640 and communicate with each other.
[0104] The bus 640 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 640 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
[0105] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the data aggregation processing method provided by the embodiments of the present disclosure.
[0106] The above-mentioned storage medium may, for example, include a memory 620 of computer program instructions, and the above-mentioned instructions may be executed by the processor 610 of the data aggregation processing device to complete the data aggregation processing method provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact DiscROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc. The above-mentioned computer program may be written in any combination of one or more programming languages to perform the program code of the operation of the embodiment of the present disclosure, and the programming language includes an object-oriented programming language, such as Java, C++, etc., and also includes a conventional procedural programming language, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device, partially on the remote computing device, or entirely on the remote computing device or server.
[0107] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0108] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data aggregation processing method, characterized in that: The method is applicable to an accelerator card, and the method comprises: Get the data set to be processed and the processing type of the aggregation processing to be performed by the target; Obtaining the data type of the data contained in the data set to be processed; Performing aggregation processing on the data in the to-be-processed data set based on the processing type to obtain an aggregation result; Writing the aggregation result into the memory of the target host; The aggregating the data in the to-be-processed data set based on the processing type to obtain an aggregation result includes: Determine width information corresponding to the data type; Determining the number of data to be processed at a single time by the accelerator card based on the pre-acquired processing bandwidth of the accelerator card and the width information; Grouping the data in the to-be-processed data set based on the number of data; For each group, aggregation processing is performed on the data in the group based on the processing type to obtain the aggregation result.
2. The method according to claim 1, characterized in that The step of performing aggregation processing on the data in each group based on the processing type to obtain the aggregation result includes: Based on a first correspondence between a data type, a processing type and an aggregate processing block, determining a target aggregate processing block corresponding to the data type and the processing type; For each group, calling the target aggregation processing block to perform aggregation processing on the data in the group to obtain a group aggregation result; Based on the grouping aggregation result and the processing type, the aggregation result corresponding to the data set to be processed is determined.
3. The method according to claim 2, characterized in that Before calling the target aggregation processing block for each group to perform aggregation processing on the data in the group to obtain the group aggregation result, the method further includes: Obtaining identification information corresponding to each data in the data set to be processed, wherein the identification information is used to indicate whether the corresponding data is valid; Based on a second correspondence between the data type, the processing type and the filling data, determining target filling data corresponding to the data type and the processing type; For each of the groups, replacing data in the group whose corresponding identification information is a first preset value with the target filling data; The calling of the target aggregation processing block to perform aggregation processing on the data in the group includes: The target aggregation processing block is called to perform aggregation processing on the data in the replaced group.
4. The method according to any one of claims 1 to 3, characterized in that The processing type of the aggregation processing includes at least one of summing processing, averaging processing, calculating a maximum value, and calculating a minimum value.
5. The method according to claim 3, characterized in that: The method further comprises: The identification information corresponding to each data in the data set to be processed is traversed, and the number of data in the data set to be processed whose corresponding identification information is a second preset value is counted.
6. The method according to claim 1, characterized in that The acceleration card is connected to the target host via a high-speed serial computer expansion bus.
7. A data aggregation processing device, characterized in that: The device is applicable to an acceleration card, and comprises: A data acquisition module is used to obtain the data set to be processed and the processing type of the aggregation processing performed by the target; A type acquisition module, used to acquire the data type of the data contained in the data set to be processed; An aggregation module, used for performing aggregation processing on the data in the to-be-processed data set based on the processing type to obtain an aggregation result; A writing module, used for writing the aggregation result into the host memory connected to the acceleration card; The aggregation module comprises: A first determining unit, used to determine width information corresponding to the data type; A second determining unit, configured to determine the number of data to be processed at a single time by the accelerator card based on the pre-acquired processing bandwidth of the accelerator card and the width information; A grouping unit, used for grouping the data in the to-be-processed data set based on the number of data; The aggregation unit is used to perform aggregation processing on the data in each group based on the processing type to obtain the aggregation result.
8. A computer device, characterized in that: include: Memory; processor; and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the data aggregation processing method according to any one of claims 1 to 6 is implemented.
