Redundant data parallel preprocessing optimization method based on heterogeneous computing platform

Abstract

The invention relates to a redundant data parallel preprocessing optimization method based on a heterogeneous computing platform, which belongs to the technical field of electronic information and comprises the following steps of: firstly, determining redundant data in a data preprocessing stage; secondly, effective data positions are marked in real time, and the number of effective data is counted; secondly, determining a threshold value according to the number, dimension and type of original storage data, and proposing an intelligent data structure selection strategy according to the number of effective data; and double-precision calculation is only performed on the identified redundant data once and the result is cached, and the result is directly reused in subsequent iteration. According to the method, the problems of low computing resource utilization rate, over-high cross-device communication bandwidth occupation, storage I / O bottleneck and the like caused by data redundancy in heterogeneous computing are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a redundant data parallel preprocessing optimization method based on a heterogeneous computing platform. Background Art

[0002] With the ever-increasing demand for computing, traditional single-processor architectures are increasingly unable to meet the requirements for efficiently handling large-scale computing tasks. In recent years, heterogeneous computing platforms have emerged as an important approach to addressing this problem. Heterogeneous computing refers to a computing approach that utilizes different types of processing units (such as CPUs, GPUs, and FPGAs) to collaborate and complete computing tasks. Compared to traditional homogeneous computing (which uses only a single type of processor), heterogeneous computing leverages the strengths and advantages of each processor type to improve computing performance and efficiency. Because CPUs are suited to complex logic and control tasks, while GPUs excel at large-scale parallel computing, heterogeneous CPU+GPU platforms have become a mainstream platform. The CPU handles logic control, while the GPU handles simple floating-point calculations. The two exchange data via the PCLE bus and have independent memory. Heterogeneous computing is also commonly used in supercomputer research. The "Sunway Blue Light II" uses China's new-generation high-performance heterogeneous many-core processor (SW26010Pro). Each processor contains six core groups, each with 65 cores, for a total of 390 cores. Each core group consists of a master core and a slave core array. The master core is generally used for data communication and I / O operations, and a large number of computationally intensive tasks are assigned to the slave cores for parallel processing.

[0003] In heterogeneous computing, because different computing units have their own independent storage space, data needs to be frequently exchanged between them. In complex numerical calculations, many tasks involve large amounts of redundant data, most of which is in double-precision format. This data is significantly wasted when transferred between different devices. Furthermore, the computation and I / O output of redundant data can become performance bottlenecks. Traditional methods for handling redundant data often struggle to select the optimal compression strategy when faced with multiple iterations and different scenarios, thus failing to fully improve overall performance. Summary of the Invention

[0004] The core goal of heterogeneous platform optimization is to migrate large-scale, parallelizable computational tasks within a program to a suitable platform for parallel processing. After the computation is complete, data must be transferred to another processor, which selectively outputs the results to files based on the task's output requirements. With the parallelization of computational tasks, computation speed decreases, and the performance bottleneck shifts to data transmission and file writing. When a computational task contains a large amount of redundant data, directly computing and transferring all the data not only reduces computational efficiency but also increases storage and transmission overhead, resulting in reduced algorithm performance.

[0005] Existing traditional deduplication methods use fixed precision processing (e.g., FP64 throughout), resulting in low computational efficiency. Furthermore, they lack adaptive perception capabilities. When a particular piece of data is repeated too many times, the amount of redundant data can change. When using the same data structure, if there is too little redundant data, the time required to transmit valid data and each data location can exceed the time required to transmit the entire array.

[0006] To effectively address these issues, this paper proposes a novel parallel preprocessing method for redundant data. Through single-precision preprocessing, radix sorting, dynamic data structure selection, and double-precision correction, this method compresses redundant data and enables efficient storage, thereby improving computational efficiency. This method effectively addresses issues such as low computing resource utilization, excessive cross-device communication bandwidth usage, and storage I / O bottlenecks caused by data redundancy in heterogeneous computing.

[0007] First, during the data preprocessing phase, single-precision (FP32) data copies are created and sorted using an optimized radix sort before the first iteration to identify redundant data. Second, during the iterative computation, a dynamic monitoring mechanism is established based on the single-precision copies and the identified redundant data. During the computation, a parallel comparison algorithm (optimized using CUDA kernels) is used to mark valid data locations and count the number of valid data in real time. Then, an intelligent data structure selection strategy is proposed based on the number of valid data. When valid data is single-valued, a coordinate offset + difference storage format (SingleDiff) is used. For small numbers of values, a hybrid sparse storage format combining coordinate lists with common value optimization is used. The original data format is retained only for multi-valued cases, and a decision formula is proposed for the critical cases of small and multi-valued values. Finally, a double-precision (FP64) selective recomputation mechanism is implemented during the accuracy assurance phase. Full-precision computation is performed on marked valid data to ensure the numerical stability of critical data. For identified redundant data, double-precision computation is performed only once, and the result is cached for direct reuse in subsequent iterations. This innovative solution significantly improves computational efficiency and storage utilization while maintaining computational accuracy.

[0008] The technical solution of the present invention is: A redundant data parallel preprocessing optimization method based on a heterogeneous computing platform, comprising: First, in the data preprocessing stage, single-precision (FP32) copies are created and sorted using radix sort before the first iteration to determine redundant data; Secondly, during the iterative calculation process, a dynamic monitoring mechanism is established based on single-precision replicas and determined redundant data. During the calculation process, a parallel comparison algorithm is used to mark the location of valid data in real time and count the number of valid data. Then, the threshold is determined based on the number, dimension, and type of the original stored data, and an intelligent data structure selection strategy is proposed based on the amount of valid data: When the number of valid data exceeds the threshold, it is determined to be dense mode and the original data format is retained; otherwise, it continues to determine whether the valid data is single-valued. When the valid data is single-valued, it is determined to be single-valid mode, and the coordinate offset + difference storage format (SingleDiff) is used. When the valid data is not single-valued, it is determined to be multi-valid sparse mode, and a hybrid sparse storage format combining coordinate list and common value optimization is used; Finally, in the accuracy assurance phase, a double-precision (FP64) selective recalculation mechanism is used: full-precision calculations are performed on marked valid data, while double-precision calculations are performed only once on identified redundant data and the results are cached and directly reused in subsequent iterations. After the calculation is completed, the result of the recalculation and storage on the device is obtained; after obtaining the result, each feature value of the result is traversed and output to text.

[0009] Preferably, according to the present invention, in the data preprocessing stage, single-precision copies are created, and the single-precision copies are sorted using radix sort before the first iteration to determine redundant data; including: First, on the host side, a single-precision buffer of the same size as the original data is pre-allocated, the original double-precision data (FP64) is converted to a single-precision (FP32) copy, and the converted single-precision copy is moved to the device-side video memory; Secondly, determine the value of redundant data for the single-precision copy; in the case of unordered data, sort the floating-point data first, and compare the sorted data with the adjacent data. If the data is the same as the adjacent data, the data is redundant data. If it is different, continue to judge the next data until the redundant data is found.

[0010] Further preferably, the floating-point data is first sorted by combining bitmap conversion with radix sorting; including: During the bitmap conversion process, the data is judged to be positive or negative. If it is positive, the original order is maintained, but the sign bit is flipped; if it is negative, the order is reversed and the data is sorted correctly by absolute value. During the radix sort process, GPU calculation is performed using a 4-bit bucketing method, that is, 4 bits of data are processed each round, containing 16 buckets. At the same time, CUDA shared memory counting is used. During the radix sort process, four bits of data are processed each round, and a total of 8 rounds are performed. Before the radix sort is executed, the data to be processed is written to the shared memory. After the radix sort is executed, the data in the shared memory is read back into the global memory.

[0011] According to the preferred embodiment of the present invention, a single-precision copy of all data involved in numerical calculation is created, and the single-precision copy that needs to be processed generated in the data preprocessing stage is used to perform numerical calculation according to the logic executed by the original code; assuming that the number of valid data is ,During the calculation process, the calculation result is compared with the redundant data. If it is the same as the redundant data, Perform self-addition operation to determine the number of valid data; At the same time, a mask array is established to mark the position of valid data. The values of the mask array are initialized to 0. If the data at the current position is different from the redundant data, the value of the mask array at the current position, that is, the mask value, is marked as 1; therefore, when the mask value of a certain position is 0, it means that the position is redundant data; when the mask value is 1, it means that the position is valid data.

[0012] Preferably, according to the present invention, the specific implementation process of the intelligent data structure selection strategy includes: According to the calculated Make judgments; There are three situations: When the number of valid data When it is a single value, it is determined to be a single valid mode, using the coordinate offset + difference storage format. Suppose the structure using the coordinate offset + difference storage format is , structure Set three parameters in the , including: redundant data value (FP64), valid data value (FP64), valid data position (Integer); When the number of valid data When it is not a single value, determine whether the data storage mode is a multi-valid sparse mode or a dense mode; the specific storage mode determination is obtained by the following formula: When it is determined to be a multi-valid sparse mode, a hybrid sparse storage format combining coordinate list and public value optimization is used. Let the structure be , while creating a valid data array Used to store valid data, assuming that the hybrid sparse storage format includes redundant data (FP64), valid data array (FP64), and valid data location (Integer). Among them, double precision occupies 8 bytes, integer occupies 4 bytes; assuming that the valid data array The dimension is d, then the total amount of data in the mixed sparse storage format Expressed as:

[0013] (1); When it is judged as dense mode, the original storage format is used, that is, the data is stored in the array in order. If N data are stored, the original array Total amount of data transferred Expressed as: (2).

[0014] Further preferably, determining whether the storage mode of the data is a multi-effective sparse mode or a dense mode includes: Determine the judgment threshold k The threshold is: (3); Where N refers to the number of stored data, and d refers to the dimension of the valid data array; when > k When threshold is set, dense mode is selected, that is, original data storage is used; otherwise, more effective sparse mode is selected.

[0015] According to a preferred embodiment of the present invention, the specific implementation process of the double-precision selective recalculation mechanism includes: The calculation value of redundant data uses the calculation result of the first data, and the calculation of valid data is processed according to different methods; In single valid mode, during the calculation process, the redundant data, valid data, and valid data location are stored in In the middle; let sum be the reduced sum, n be Array size, is a valid value, and the calculation formula is: (4); In the case of a multi-valid sparse mode, the calculation needs to re-traverse the original array If the value in the mask array corresponding to the current position is 1, the numerical calculation is performed according to the logic in the original code, and the numerical calculation result and the position of the valid data are stored in the structure In the process of performing the reduced accumulation operation, the valid data is accumulated during the calculation process. After the calculation is completed, the accumulated sum is added to the multiple redundant data, as shown in formula (5): (5).

[0016] An automated interface SRDO (Solving Redundant Data Optimization) is called to implement the redundant data parallel preprocessing optimization method based on the heterogeneous computing platform.

[0017] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the redundant data parallel preprocessing optimization method based on a heterogeneous computing platform are implemented.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned redundant data parallel preprocessing optimization method based on a heterogeneous computing platform.

[0019] The beneficial effects of the present invention are: 1. The present invention reduces the impact of full-process double-precision on time and memory through single-precision preprocessing and double-precision selective recalculation mechanisms. Different precisions are determined for different computing scenarios, which not only guarantees the calculation results but also saves time and improves computing performance.

[0020] 2. By establishing different storage structures, the present invention can determine the most suitable storage type according to the amount of valid data, thereby ensuring that the minimum amount of data is transmitted during the data transmission phase, saving data transmission time and maximizing parallel computing.

[0021] 3. By preprocessing redundant data, the present invention effectively solves problems such as low computing resource utilization, excessive cross-device communication bandwidth occupancy, and storage I / O bottlenecks caused by data redundancy in heterogeneous computing. It significantly improves computing efficiency and storage utilization while ensuring computing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of a redundant data parallel preprocessing optimization method based on a heterogeneous computing platform according to the present invention; Figure 2 This is a schematic diagram of single-precision and double-precision floating points in the present invention; Figure 3 Identifying redundant data flow graphs for the present invention; Figure 4 Diagram of strategy selection for smart data structures. DETAILED DESCRIPTION

[0023] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0024] Example 1 When optimizing programs on heterogeneous systems, data transmission between systems often becomes a performance bottleneck as optimization progresses. The presence of large amounts of redundant data exacerbates the repeated transmission of multiple data points, further exacerbating data transmission issues. Furthermore, the computation of redundant data, I / O operations, and other such processes waste time and memory. To address this issue, the present invention proposes an innovative optimization method that uses mixed precision and radix sort deduplication techniques to parallelize preprocessing of redundant data generated during the computation process, thereby improving the efficiency of redundant data computation, transmission, and storage. The present invention also designs an automated interface, SRDO (Solving Redundant Data Optimization), for easy direct access by programmers.

[0025] A redundant data parallel preprocessing optimization method based on heterogeneous computing platform, such as Figure 1 Shown, including: First, during the data preprocessing phase, single-precision (FP32) copies are created and sorted using an optimized radix sort before the first iteration to identify redundant data. Secondly, during the iterative calculation process, a dynamic monitoring mechanism is established based on single-precision copies and determined redundant data. During the calculation process, a parallel comparison algorithm (optimized using CUDA kernel functions) is used to mark the location of valid data in real time and count the number of valid data. Then, the threshold is determined based on the number, dimension, and type of the original stored data, and an intelligent data structure selection strategy is proposed based on the amount of valid data: When the number of valid data exceeds the threshold, it is determined to be dense mode and the original data format is retained; otherwise, it continues to determine whether the valid data is single-valued. When the valid data is single-valued, it is determined to be single-valid mode, and the coordinate offset + difference storage format (SingleDiff) is used. When the valid data is not single-valued, it is determined to be multi-valid sparse mode, and a hybrid sparse storage format combining coordinate list and common value optimization is used; Finally, during the accuracy assurance phase, a double-precision (FP64) selective recalculation mechanism is employed: full-precision calculations are performed on marked valid data to ensure the numerical stability of critical data; while double-precision calculations are performed only once on identified redundant data, and the results are cached and directly reused in subsequent iterations. After the calculation is complete, the host needs to call a data transfer function, such as cudaMemcpy(), to obtain the result recalculated and stored on the device. After obtaining the result, each eigenvalue of the result is traversed and output to text. This invention significantly improves computing efficiency and storage utilization while ensuring calculation accuracy.

[0026] Example 2 The redundant data parallel preprocessing optimization method based on a heterogeneous computing platform described in Example 1 is different in that: In the data preprocessing stage, single-precision copies are created and sorted using radix sort before the first iteration to determine redundant data; Figure 3 Shown, including: When performing numerical calculations, in order to avoid the involvement of large amounts of redundant data, it is necessary to effectively identify and determine the redundant data before the calculation. In floating-point calculations, especially in fields such as scientific computing and numerical simulation, double-precision calculations are usually used to ensure the accuracy of the calculation results. However, in some scenarios where precision is not a high requirement, using double-precision calculations may lead to unnecessary waste of time. In contrast, single-precision uses 32 bits to represent data, and each floating-point number only occupies 4 bytes, which greatly saves memory compared to the 8 bytes of double-precision. Especially in GPUs, single-precision calculations can be accelerated by dedicated hardware.

[0027] Therefore, to address this problem, single precision is used for preprocessing before calculation to identify and determine the value, quantity, and location of redundant data, thereby effectively reducing unnecessary calculations and resource waste.

[0028] First, on the host side, a single-precision buffer of the same size as the original data is pre-allocated, the original double-precision data (FP64) is converted to a single-precision (FP32) copy, and the converted single-precision copy is moved to the device-side video memory; Secondly, determine the value of redundant data for the single-precision copy; in the case of unordered data, sort the floating-point data first, and compare the sorted data with the adjacent data. If the data is the same as the adjacent data, the data is redundant data. If it is different, continue to judge the next data until the redundant data is found.

[0029] In the sorting phase, given the powerful parallel computing capabilities and large amounts of floating-point calculations on heterogeneous platforms, a combination of bitmap conversion and radix sorting is used to sort floating-point data. This includes: Before performing radix sort, floating-point numbers must be converted to integers. However, if this is done according to the IEEE 754 standard, negative numbers may be incorrectly ordered when compared bitwise. Therefore, a bitmap conversion is proposed before radix sort. During this conversion, the data is determined to be positive or negative. If positive, the order is maintained, but the sign bit is flipped. If negative, the order is reversed and the data is correctly sorted by absolute value. This operation allows the same comparison logic to be used for both positive and negative numbers. During radix sort, GPU computation is performed using a 4-bit bucketing approach, processing 4 bits of data per round, encompassing 16 buckets. CUDA shared memory is also used to reduce conflicts in global atomic operations. Therefore, each round processes four bits of data, for a total of eight rounds. Before the radix sort is executed, the data to be processed is written to shared memory and then read back into global memory after the radix sort is executed.

[0030] During multiple iterative computations, the amount of redundant data changes, and a single data structure often fails to achieve optimal storage efficiency. Furthermore, when data values are disordered, common redundant data compression methods such as differential encoding and run-length encoding are inapplicable. Therefore, a classification method based on the amount of valid data is proposed, which categorizes data into three types: single-valid pattern, multi-valid sparse pattern, and dense pattern. The most appropriate data structure is then selected for storage based on each pattern.

[0031] Create single-precision copies of all data involved in numerical calculations, and perform numerical calculations according to the logic of the original code execution together with the single-precision copies generated in the data preprocessing stage. Suppose the number of valid data is ,During the calculation process, the calculation result is compared with the redundant data. If it is the same as the redundant data, Perform self-addition operation to determine the number of valid data; At the same time, a mask array is established to mark the position of valid data. The values of the mask array are initialized to 0. If the data at the current position is different from the redundant data, the value of the mask array at the current position, that is, the mask value, is marked as 1; therefore, when the mask value of a certain position is 0, it means that the position is redundant data; when the mask value is 1, it means that the position is valid data.

[0032] like Figure 4 As shown in Figure 2, the specific implementation process of the intelligent data structure selection strategy includes: According to the calculated Make judgments; There are three situations: When the number of valid data When it is a single value, it is determined to be a single valid mode, using the coordinate offset + difference storage format. Suppose the structure using the coordinate offset + difference storage format is , structure Set three parameters in the , including: redundant data value (FP64), valid data value (FP64), valid data position (Integer); Since it is a single outlier, there is no need to loop through the data during I / O operations and subsequent numerical calculations.

[0033] When the number of valid data When it is not a single value, determine whether the data storage mode is a multi-valid sparse mode or a dense mode; the specific storage mode determination is obtained by the following formula: When it is determined to be a multi-valid sparse mode, a hybrid sparse storage format combining coordinate list and public value optimization is used. Let the structure be , while creating a valid data array Used to store valid data, suppose the hybrid sparse storage format includes redundant data (FP64), valid data array (FP64), and valid data location (Integer). Among them, double precision occupies 8 bytes, integer occupies 4 bytes; suppose valid data array The dimension is d, then the total amount of data in the mixed sparse storage format Expressed as:

[0034] (1); When it is judged as dense mode, the original storage format is used, that is, the data is stored in the array in order. If N data are stored, the original array (Stores raw double-precision data) Total amount of data transferred Expressed as: (2); By calculating the amount of data transmitted by the hybrid sparse storage format and comparing it with the amount of data transmitted by the original array, a threshold is determined, thereby determining the optimal storage format.

[0035] Determine whether the data storage mode is a sparse mode or a dense mode; including: Determine the judgment threshold k The threshold is: (3); Where N refers to the number of stored data, and d refers to the dimension of the valid data array; when > k When threshold is set, dense mode is selected, that is, original data storage is used; otherwise, more effective sparse mode is selected.

[0036] The specific implementation process of the double-precision selective recalculation mechanism includes: After determining the appropriate data structure for storage, the initial calculations using single precision resulted in errors in the calculation results. Therefore, after selecting a new data structure, the calculations need to be re-performed using double precision to correct the errors.

[0037] Regardless of the data structure chosen to store data, the calculation value of redundant data uses the calculation result of the first data, and the calculation of valid data is processed in different ways; In single valid mode, the calculation process is very simple, only the valid data of the marked position needs to be recalculated, and no loop or parallel operation is required. During the calculation process, the redundant data, valid data, and valid data position are stored in In the reduction operation, taking the reduction sum as an example, let sum be the reduction sum (calculate the sum of all data), n is Array size, is a valid value, and the calculation formula is:

[0038] (4); In the case of a multi-valid sparse mode, the calculation needs to re-traverse the original array (This array is the original array, the array originally used to perform calculations in the code, and is not processed). If the value in the mask array corresponding to the current position is 1, numerical calculations (such as addition, subtraction, multiplication, and division) are performed according to the logic in the original code, and the numerical calculation results and the location of valid data are stored in the structure. In the process of performing the reduced accumulation operation, the valid data is accumulated during the calculation process. After the calculation is completed, the accumulated sum is added to the multiple redundant data, as shown in formula (5): (5).

[0039] In both of these modes, heterogeneous device transmission and I / O operations require only the structure to be processed. The structure contains only valid data, its location, and a single redundant data item. This significantly reduces the amount of data transmitted, conserves memory space, and improves execution efficiency.

[0040] Through the implementation of the above method, the present invention was tested in a CPU-GPU heterogeneous system. Two test procedures were used, and the test time was obtained by taking the average of three times.

[0041] When parallelizing Fortran codes on traditional heterogeneous platforms, the numerical calculation part is mainly ported to the GPU using CUDA for calculation. After the calculation is completed, the calculation results are transferred back to the CPU to perform the remaining operations.

[0042] Test 1 is a basic test program used to verify the efficiency difference between the inventive method and the original method that does not process redundant data. During execution, the program first reads input data from a specified text file, then performs calculations, parallelizing the calculation phase, and finally outputs the results to a file. The runtime of this entire phase is measured, along with the runtime after the inventive method processes redundant data, for subsequent comparison and analysis.

[0043] Test 2 was based on the time before and after processing redundant data in an algorithm for the dynamic evolution of polymer nanomaterials. This algorithm, developed by a laboratory at the Beijing University of Chemical Technology, uses different external field conditions to calculate density changes at each stage of the crystallization process, and then uses these changes to calculate changes in chemical potential.

[0044] The testing and verification of the present invention is divided into two parts: Test 1 uses a basic test program to verify the performance acceleration effect of the present invention method at different data sizes by comparing the execution time of the original GPU parallel program with that of the present invention method. The test is conducted using 100,000, 500,000, and 1 million data sizes, corresponding to small scale, medium scale, and large scale, respectively. Test 2 uses a polymer nanomaterial dynamic property evolution algorithm program to verify the acceleration effect of the present invention method at different particle numbers by comparing the execution time of the original GPU parallel program with that of the present invention method. The test is conducted using 130,000 particles, 520,000 particles, and 1 million particles, corresponding to small scale, medium scale, and large scale, respectively. The specific experimental results and comparison are shown in the table below.

[0045] Table 1 shows the time of the original GPU parallel computing and the method of the present invention under different data sizes in the test 1 program, as well as the speedup ratio of the method of the present invention compared with the original GPU parallel computing.

[0046] Table 1 Test 1 time comparison of different data amounts (unit: ms);

[0047] Table 2 shows the time of original GPU parallel computing and the method of the present invention in the crystallization process for different numbers of particles in the test 2 program, as well as the speedup ratio of the method of the present invention compared to the original GPU parallel computing.

[0048] Table 2. Time comparison of different particle numbers in test 2 (unit: ms);

[0049] Experimental results show that the method of the present invention exhibits significant advantages in acceleration performance. By comparing multiple groups of comparative tests with different data volumes and different numbers of particles, the method of the present invention shows performance improvement compared with the original GPU parallel program solution.

[0050] Example 3 An automated interface SRDO (Solving Redundant Data Optimization) is called to implement the redundant data parallel preprocessing optimization method based on a heterogeneous computing platform described in embodiment 1 or 2.

[0051] This interface is designed to solve the data redundancy problem in heterogeneous systems during converged computing and improve execution efficiency. The implementation of this interface reduces programming difficulty, and programmers can directly call this interface to solve problems. The interface description is shown in Table 3:

[0052] Table 3 SRDO interface description;

[0053] Figure 2 This is a schematic diagram of single-precision and double-precision floating points in the present invention; When using the interface, the programmer needs to first reference the SRDO module and then implement the interface functionality by calling the corresponding function. An example of using the interface is shown below:

[0054] / / Call the SRDO interface program main use SRDO ... / / Omit other operations / / Initialize the interface call SRDO_Init () / / Parameter assignment SINGLE_OUTLIER = 1 / / single effective mode MULTI_OUTLIER = 2 / / Multiple effective sparse modes DENSE_DATA = 3 / / Dense mode N = 100,000 array = [(i, i = 1, N) ] array_f32=(real(array(:),kind=4) / / Call interface function call SRDO_Sort (array_f32, N, common_value) call SRDO _GPU_Analyse(array_f32, N, common_value, unique_count,mask) if(unique_count==1) sparse_mode = 1 else if(unique_count <kthreshold) sparse_mode = 2 else sparse_mode = 3 call SRDO_GPU_Compute (array, N, mask, sparse_mode, sparse_data) / / Close the interface call SRDO_Finalize() ... / / Omit other operations end program main Example 4 A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the redundant data parallel preprocessing optimization method based on a heterogeneous computing platform described in embodiment 1 or 2 are implemented.

[0055] Example 5 A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the redundant data parallel preprocessing optimization method based on a heterogeneous computing platform described in Example 1 or 2.

Claims

1. A redundant data parallel preprocessing optimization method based on a heterogeneous computing platform, characterized in that: include: First, in the data preprocessing stage, single-precision copies are created and sorted using radix sort before the first iteration to determine redundant data; Secondly, during the iterative calculation process, a dynamic monitoring mechanism is established based on single-precision replicas and determined redundant data. During the calculation process, a parallel comparison algorithm is used to mark the location of valid data in real time and count the number of valid data. Then, the threshold is determined based on the number, dimension, and type of the original stored data, and an intelligent data structure selection strategy is proposed based on the amount of valid data: When the number of valid data exceeds the threshold, it is determined to be dense mode and the original data format is retained; otherwise, it is further determined whether the valid data is single-valued. When the valid data is single-valued, it is determined to be single-valid mode, and the coordinate offset + difference storage format is used. When the valid data is not single-valued, it is determined to be multi-valid sparse mode, and a hybrid sparse storage format combining coordinate list and common value optimization is used. Finally, in the accuracy assurance phase, a double-precision selective recalculation mechanism is used: full-precision calculations are performed on marked valid data, while double-precision calculations are performed only once on identified redundant data and the results are cached and directly reused in subsequent iterations. After the calculation is completed, the result of the recalculation and storage on the device is obtained; After obtaining the results, each feature value of the results is traversed and output to text.

2. The method for parallel preprocessing and optimization of redundant data based on a heterogeneous computing platform according to claim 1, characterized in that: During the data preprocessing phase, single-precision copies are created and sorted using radix sort before the first iteration to determine redundant data. This includes: First, on the host side, a single-precision buffer of the same size as the original data is pre-allocated, the original double-precision data is converted into a single-precision copy, and the converted single-precision copy is moved to the device-side video memory; Secondly, determine the value of redundant data for the single-precision copy; in the case of unordered data, sort the floating-point data first, and compare the sorted data with the adjacent data. If the data is the same as the adjacent data, the data is redundant data. If it is different, continue to judge the next data until the redundant data is found.

3. The method for parallel preprocessing and optimization of redundant data based on a heterogeneous computing platform according to claim 2, characterized in that: The floating-point data is first sorted by combining bitmap conversion and radix sorting; including: During the bitmap conversion process, the data is judged to be positive or negative. If it is positive, the original order is maintained, but the sign bit is flipped; if it is negative, the order is reversed and the data is sorted correctly by absolute value. During the radix sort process, GPU calculation is performed using a 4-bit bucketing method, that is, 4 bits of data are processed each round, containing 16 buckets. At the same time, CUDA shared memory counting is used. During the radix sort process, four bits of data are processed each round, and a total of 8 rounds are performed. Before the radix sort is executed, the data to be processed is written to the shared memory. After the radix sort is executed, the data in the shared memory is read back into the global memory.

4. The method for parallel preprocessing and optimization of redundant data based on a heterogeneous computing platform according to claim 1, characterized in that: Create single-precision copies of all data involved in numerical calculations, and perform numerical calculations according to the logic of the original code execution together with the single-precision copies that need to be processed generated in the data preprocessing stage; Assume the number of valid data is ,During the calculation process, the calculation result is compared with the redundant data. If it is the same as the redundant data, Perform self-addition operation to determine the number of valid data; At the same time, a mask array is established to mark the position of valid data. The values of the mask array are initialized to 0. If the data at the current position is different from the redundant data, the value of the mask array at the current position, that is, the mask value, is marked as 1. Therefore, when the mask value of a certain position is 0, it means that the position is redundant data; when the mask value is 1, it means that the position is valid data.

5. The method for parallel preprocessing and optimization of redundant data based on a heterogeneous computing platform according to claim 1, characterized in that: The specific implementation process of the intelligent data structure selection strategy includes: According to the calculated Make judgments; There are three situations: When the number of valid data When it is a single value, it is determined to be a single valid mode, using the coordinate offset + difference storage format. Suppose the structure using the coordinate offset + difference storage format is , structure Set three parameters in the , including: redundant data value, valid data value, and valid data position; When the number of valid data When it is not a single value, determine whether the data storage mode is a multi-valid sparse mode or a dense mode; the specific storage mode determination is obtained by the following formula: When it is determined to be a multi-valid sparse mode, a hybrid sparse storage format combining coordinate list and public value optimization is used. Let the structure be , while creating a valid data array Used to store valid data; suppose the mixed sparse storage format includes redundant data, valid data array, and valid data position; among them, double precision occupies 8 bytes and integer occupies 4 bytes; suppose the valid data array The dimension is d, then the total amount of data in the mixed sparse storage format Expressed as: （1）； When it is judged as dense mode, the original storage format is used, that is, the data is stored in the array in order. If N data are stored, the original array Total amount of data transferred Expressed as: （2）。 6. The method for parallel preprocessing and optimization of redundant data based on a heterogeneous computing platform according to claim 5, characterized in that: Determine whether the data storage mode is a sparse mode or a dense mode; including: Determine the judgment threshold k The threshold is: （3）； Where N refers to the number of stored data, and d refers to the dimension of the valid data array; when > k When threshold is set, dense mode is selected, that is, original data storage is used; otherwise, more effective sparse mode is selected.

7. A redundant data parallel preprocessing optimization method based on a heterogeneous computing platform according to any one of claims 1 to 6, characterized in that: The specific implementation process of the double-precision selective recalculation mechanism includes: The calculation value of redundant data uses the calculation result of the first data, and the calculation of valid data is processed according to different methods; In single valid mode, during the calculation process, the redundant data, valid data, and valid data location are stored in In the middle; let sum be the reduced sum, n be Array size, is a valid value, and the calculation formula is: （4）； In the case of a multi-valid sparse mode, the calculation needs to re-traverse the original array If the value in the mask array corresponding to the current position is 1, the numerical calculation is performed according to the logic in the original code, and the numerical calculation result and the position of the valid data are stored in the structure In the process of performing the reduced accumulation operation, the valid data is accumulated during the calculation process. After the calculation is completed, the accumulated sum is added to the multiple redundant data, as shown in formula (5): （5）。 8. An automated interface SRDO, characterized in that: The redundant data parallel preprocessing optimization method based on a heterogeneous computing platform described in any one of claims 1 to 7 is implemented by calling the automation interface SRDO.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the redundant data parallel preprocessing optimization method based on a heterogeneous computing platform according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the redundant data parallel preprocessing optimization method based on a heterogeneous computing platform according to any one of claims 1 to 7 are implemented.

Images