Method and system for optimizing cpu overhead in a cluster search algorithm run

Abstract

The application belongs to the technical field of computers, and particularly relates to a method and system for optimizing CPU overhead in the operation of a bundle search algorithm. The method for optimizing CPU overhead in the operation of a bundle search algorithm comprises the following steps: performing slice processing on the probability distribution set of each modeling unit; using an algorithm on a GPU to solve the top k candidate solutions with high probability; and splicing the slices of the top k candidate solution data set to output a complete probability distribution data set. After optimization of the bundle search algorithm, the CPU overhead is significantly reduced, and the occupation time of the GPU is also reduced. After optimization, the average time consumption of the entire model training process is improved by about 36%, and the optimization effect is obvious.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically a method and system for optimizing CPU overhead during the operation of a beam search algorithm. Background Technology

[0002] Beam Search is a general-purpose search algorithm applied to various deep learning models such as Seq2Seq, Transformer, GPT, BERT, and CRNN. ​​Its main task is to generate the optimal output sequence during the decoding process. Beam Search can be viewed as a heuristic search strategy that generates multiple alternatives at each time step, retains a certain number of optimal alternatives, and discards the rest, thereby finding the optimal solution in the search space and ultimately enabling the model to generate a more accurate and fluent output sequence.

[0003] The basic idea of ​​Beam Search is that during the decoding process, it maintains a candidate sequence set for the beam, containing k candidate solutions that could potentially be the optimal solution. These k candidate solutions are selected from all u solutions, choosing the top k highest-scoring candidate sequences. Let the optimal k candidate solutions be denoted as topk. During the search process, the Beam Search algorithm generates all possible next states and calculates the score for each state, sorting and selecting them to ultimately choose the k candidate solutions for the next state. By continuously updating the candidate solutions in the beam set, the Beam Search algorithm can gradually converge until it finds the final optimal solution.

[0004] Assuming the modeling unit length is n, the number of optimal solutions is k, and the total number of possible solutions is u, the complexity of the BeamSearch algorithm is O(n*k*u) according to the above principles. It can be seen that the algorithm's complexity is positively correlated with the number of optimal solutions k and the total number of possible solutions u for each modeling unit. However, in actual industrial production, such as training a Chinese language model, the total number of possible solutions u is extremely large, significantly slowing down the entire training process.

[0005] When the size of the probability distribution set output by the trained model is large and the probability distribution of the model's output dataset is relatively concentrated, the traditional Beam Search algorithm will consume a lot of time in practical applications. Therefore, a method is needed to optimize the computational speed of Beam Search. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a method for optimizing CPU overhead during the operation of the beam search algorithm. This method proposes an optimization scheme to address the problem that the beam search algorithm consumes too much time in the process of finding the optimal k candidate solutions.

[0007] The technical solution adopted by this invention to solve its technical problem is: a method for optimizing CPU overhead during the operation of a beam search algorithm, comprising:

[0008] The probability distribution set of each modeling unit is sliced;

[0009] The probability distribution dataset after slice processing is used to solve for the top k candidate solutions with high probabilities using an algorithm on the GPU.

[0010] The first k candidate solution datasets are sliced ​​and concatenated to output the complete probability distribution dataset.

[0011] Preferably, the slicing process involves separating the whitespace characters at the end of the probability distribution dataset, while retaining the main body of the probability distribution dataset.

[0012] Preferably, after completing the solution operation, the indices of the original dataset corresponding to the first k candidate solutions are stored to obtain the index dataset.

[0013] Preferably, the index dataset is converted from Integer to float.

[0014] Preferably, the first k candidate solution datasets, the index dataset, and the whitespace fragments are concatenated in sequence to obtain the complete probability distribution dataset.

[0015] The present invention also provides a system for optimizing CPU overhead during the operation of a beam search algorithm. The system includes: a data receiving module for receiving a probability distribution dataset from the training result set of the previous model.

[0016] The data processing module includes a slicing layer, a solution layer, and a type conversion layer. The slicing layer is used to slice the probability distribution dataset and separate the trailing whitespace characters. The solution layer is an algorithm on the GPU used to solve for the top k candidate solutions with high output probabilities from the sliced ​​probability distribution dataset. The type conversion layer is used to convert the indices corresponding to the top k candidate solutions in the original dataset into character types, specifically FLA.

[0017] The concatenation module is used to concatenate the first k candidate solution datasets, the index dataset, and the whitespace fragments in sequence into a complete probability distribution dataset; and output it to the N-gram model to obtain the optimal solution.

[0018] Compared with existing technologies, this invention has the following beneficial effects: After optimization, the beam search algorithm significantly reduces CPU overhead and GPU usage time. After optimization, the average training time of the entire model is reduced by approximately 36%, demonstrating a significant optimization effect. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for optimizing CPU overhead during the execution of a beam search algorithm according to the present invention.

[0020] Figure 2 This is a system model diagram for optimizing CPU overhead during the execution of the beam search algorithm according to the present invention;

[0021] Figure 3 To optimize CPU utilization;

[0022] Figure 4 To optimize CPU utilization;

[0023] Figure 5 Overall process improvement after optimization. Detailed Implementation

[0024] To facilitate understanding of this research, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. However, this research can be implemented in many different forms and is not limited to the embodiments described in this specification. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this research.

[0025] This invention provides a method for optimizing CPU overhead during the execution of a beam search algorithm, the process of which is as follows: Figure 1 As shown, the specific steps are as follows:

[0026] 1. First, receive the probability distribution dataset from the training result set of the previous model.

[0027] The received probability distribution dataset is sliced. Specifically, the trailing whitespace character is separated from the probability distribution dataset of each modeling unit, resulting in three parts: a whitespace-margined dataset, a topk dataset, and an index dataset. Since each modeling unit's probability distribution set ends with a whitespace character, and this trailing whitespace character is not needed for sorting during the actual topk solution calculation, this invention slices the probability distribution set of each modeling unit and only processes the main part of the probability distribution set, namely the topk dataset. `topk` represents the top k candidate solutions.

[0028] 2. Preprocess the k candidate solutions of n modeling units.

[0029] Traditional BeamSearch algorithms require maintaining a candidate set sequence, but GPU memory is relatively small compared to CPU memory. Therefore, when processing large datasets or maintaining numerous candidate sequences, memory overflow may occur. Consequently, BeamSearch typically runs on the CPU. When the size of the training model's probability distribution set is large and the probability distribution of the model's output dataset is relatively concentrated, some possible solutions correspond to very low model output probabilities. Therefore, the size of the candidate solution set can be pruned. This invention selects the top k candidate solutions with the highest output probabilities for each of the n modeling units. After optimization by this invention, the process of solving for the top k candidate solutions can be separated, and a corresponding algorithm exists on the GPU to perform this calculation. Therefore, the top k solution part of the BeamSearch algorithm can be separated and accelerated separately, enabling preprocessing.

[0030] 3. Since the index in the original dataset is of type Integer, while the subsequent model processing requires a data type of float, it is necessary to convert the data type of the index dataset from Integer to float.

[0031] 4. Perform slice stitching and output the complete probability distribution dataset.

[0032] The topk dataset, index dataset, and whitespace fragments processed by the above steps are concatenated in sequence to obtain the complete probability distribution dataset.

[0033] The concatenated probability distribution dataset is input into the N-gram model, which evaluates and corrects the probability distribution dataset generated by Beam Search. By generating a fully connected graph of the probability distribution, the model obtains the path with the highest probability and outputs the optimal solution.

[0034] The present invention also provides a system, such as Figure 2As shown, the system includes: a data receiving module, used to receive the probability distribution dataset from the training result set of the previous-level model; and a data processing module, including a slicing layer, a solution layer, and a type conversion layer. The slicing layer is used to slice the probability distribution dataset and separate the trailing whitespace characters from the dataset; the solution layer is an algorithm on the GPU (nth_element operator), used to solve for the top k candidate solutions with high output probabilities from the sliced ​​probability distribution dataset; the type conversion layer is used to convert the indices corresponding to the top k candidate solutions in the original dataset to FLOAD; and the concatenation module is used to concatenate the solved top k candidate solution dataset, the index dataset, and the whitespace fragments in sequence into a complete probability distribution dataset; and output it to the N-gram model to obtain the optimal solution.

[0035] To evaluate the performance of the method of this invention, the CRNN-CTC model optimized by this method was tested. CTC-CRNN is a deep learning model that combines the CTC (Connection-Time Classification) algorithm and CRNN (Convolutional Recurrent Neural Network) for text recognition tasks. The CTC algorithm is a method for training sequence classifiers without aligning labels. It allows the model to classify any subsequence in the input sequence, thus enabling the model to handle variable-length sequence data.

[0036] The CTC-CRNN model typically consists of convolutional layers, recurrent layers, fully connected layers, and a CTC layer. Convolutional layers extract features from the input data, recurrent layers process the sequence data, fully connected layers convert the output into predictions, and the CTC layer calculates the loss function to guide model training. During training, the CTC-CRNN model feeds the input sequence into convolutional and recurrent layers to extract features, and then the fully connected layer converts the output into a probability distribution of characters or words. Next, the CTC layer calculates the difference between the model's predictions and the actual labels, and uses this difference as the loss function to backpropagate to the model for optimization.

[0037] During the prediction process, the CTC-CRNN model decodes the model output using the Beam Search algorithm to obtain the final text result.

[0038] This invention optimizes the CPU overhead of the Beam Search algorithm in the CTC-RNN model. The optimized GPU time is shown in Table 1. Since the calculation of the top k solutions is moved from the CPU to the GPU, the GPU computation time is reduced by 8% compared to before optimization. However, the time to export the trained weights and parameters from the deep learning framework (D2H) is optimized by 1643%. This is because the original algorithm needs to export all u candidate solutions, which consumes a significant amount of time. After optimization, the model only needs to export the results of the first k candidate solutions, greatly saving computation time. Therefore, although the GPU computation time increases after optimization using the method of this invention, the overall GPU time utilization is optimized by 5%, as shown in Table 1. Figure 3 As shown, the Beam Search algorithm before optimization had a CPU utilization rate as high as 61% during computation. However, by transferring the top-k solution process to the GPU, as shown... Figure 4 As shown, its CPU utilization dropped to 4%. This experiment tested the optimized CTC-RNN model, as shown... Figure 5 As shown, the average training time of the entire model was reduced by about 36%, demonstrating a significant optimization effect.

[0039] Table 1 GPU usage time

[0040]

Claims

1. A method for optimizing CPU overhead in a running of a beam search algorithm, characterized in that, The system comprises: The receiving module is configured to receive the probability distribution data set from the training result set of the previous model. The data processing module comprises a slicing layer, a solving layer, and a type conversion layer. The slicing layer is configured to slice the probability distribution data set and separate the blank symbols at the end of the probability distribution data set. The solving layer is an algorithm on the GPU, configured to solve the top k candidate solutions with high output probability from the sliced probability distribution data set. The type conversion layer is configured to convert the indexes corresponding to the top k candidate solutions in the original data set from Intenger to float. The splicing module is configured to splice the top k candidate solution data set, the index data set, and the blank symbol slice in sequence to form a complete probability distribution data set, and output the complete probability distribution data set to the N-gram model to obtain the optimal solution. The system comprises:

2. A system for implementing the method of claim 1, characterized in that, The receiving module is configured to receive the probability distribution data set from the training result set of the previous model. The data processing module comprises a slicing layer, a solving layer, and a type conversion layer. The slicing layer is configured to slice the probability distribution data set and separate the blank symbols at the end of the probability distribution data set. The solving layer is an algorithm on the GPU, configured to solve the top k candidate solutions with high output probability from the sliced probability distribution data set. The type conversion layer is configured to convert the indexes corresponding to the top k candidate solutions in the original data set from Intenger to float. The splicing module is configured to splice the top k candidate solution data set, the index data set, and the blank symbol slice in sequence to form a complete probability distribution data set, and output the complete probability distribution data set to the N-gram model to obtain the optimal solution. ​

Images