Model reasoning method and device, electronic equipment and storage medium
By splitting the input data into sub-data in distributed model inference and deploying computing and communication nodes on different hardware resources, the parallelism of communication and computing is achieved, and the inference delay and low computing resource utilization in tensor parallelism are solved. The problem improves the efficiency of model reasoning.
Patent Information
- Application Number
- CN202411207298.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-22
AI Technical Summary
In distributed model inference, frequent communication operations caused by tensor parallelism increase in inference delay, and lead to idle computing resources of graphics card, reducing model inference efficiency.
By dividing the input data into multiple sub-data and deploying computing nodes and communication nodes on different hardware resources, the parallelism of communication and computing is realized, ensuring that the computing node immediately performs the next round of computing after receiving the communication results, avoiding the computing nodes from Idle.
It effectively hides most of the time-consuming communication operations, reduces the idle situation of computing nodes, improves model inference efficiency, and reduces model inference delay.
Smart Images

Figure CN119005338A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model reasoning method, device, electronic device and storage medium. Background Art
[0002] The rapid development of artificial intelligence technology, especially the popularity of large language models (LLM), has led to a large increase in model parameters, which requires high computing power support. Therefore, models are often deployed on multiple graphics cards or multiple servers for distributed processing to improve speed and real-time performance.
[0003] In distributed processing, model segmentation is a common technique. Take Tensor Parallelism (TP) as an example. It allows different parts of the model to work in parallel on multiple graphics cards to improve efficiency. However, the large number of communication operations brought by TP has become a performance bottleneck for model reasoning, resulting in an increase in reasoning latency. In addition, since the graphics card needs to wait for the communication to complete before the next round of calculations, frequent communication operations will also cause the graphics card's computing resources to be idle during model reasoning and cannot be effectively utilized, further exacerbating the delay problem of model reasoning. Summary of the invention
[0004] The present invention provides a model reasoning method, device, electronic device and storage medium, which are used to solve the defect of low efficiency of model reasoning in related technologies.
[0005] The present invention provides a model reasoning method, comprising: Split the input data into multiple sub-data; Based on the first computing node used in the model reasoning, the plurality of sub-data are calculated to obtain a first calculation result of each sub-data; Based on the first communication node used in the model reasoning, a first calculation result of a sub-data is communicated and processed, and transmitted to the second calculation node used in the model reasoning for calculation, and while the first communication result of the received sub-data is calculated based on the second calculation node, a first calculation result of the next sub-data is communicated and processed based on the first communication node; the first communication result of the sub-data is obtained by the first communication node communicating and processing the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
[0006] According to a model reasoning method provided by the present invention, the first computing node applied in the model reasoning is used to calculate the multiple sub-data, including: Based on the first computing node used in the model reasoning, sequentially computing the plurality of sub-data; The first communication node applied in the model reasoning performs communication processing on a first calculation result of a sub-data, including: While calculating one sub-data based on the first computing node, performing communication processing on a first calculation result of the previous sub-data based on the first communication node; The first computing node and the first communication node are deployed on different hardware resources.
[0007] According to a model inference method provided by the present invention, the step of dividing the input data into a plurality of sub-data comprises: The input data is divided into multiple sub-data in batches, and the model reasoning corresponding to each sub-data is independent of each other.
[0008] According to a model inference method provided by the present invention, the step of dividing the input data into a plurality of sub-data comprises: The input data is divided into a plurality of sub-data according to the data volume, and the word units contained in each sub-data are complete word units.
[0009] A model reasoning method provided by the present invention also includes: While calculating the first communication result of the next sub-data based on the second computing node, the second computing result of one sub-data is communicated and processed based on the second communication node used in the model reasoning, and transmitted to the third computing node used in the model reasoning for calculation, wherein the second computing result of the one sub-data is obtained by calculating the first communication result of the one sub-data by the second computing node; and, while calculating the second communication result of the received one sub-data based on the third computing node, performing communication processing on the second calculation result of the next sub-data based on the second communication node, the second communication result of the one sub-data being obtained by the second communication node performing communication processing on the second calculation result of the one sub-data; The second computing node and the second communication node are deployed on different hardware resources, and the third computing node and the second communication node are deployed on different hardware resources.
[0010] According to a model inference method provided by the present invention, the first computing node and the second computing node are deployed in a stream processor cluster, and the first communication node is deployed in a direct memory read module.
[0011] The present invention also provides a model reasoning device, comprising: A segmentation unit, used for segmenting the input data into a plurality of sub-data; A first reasoning unit, configured to calculate the plurality of sub-data based on a first computing node used in model reasoning to obtain a first calculation result for each sub-data; A second reasoning unit is used to perform communication processing on a first calculation result of a sub-data based on the first communication node used in the model reasoning, and transmit it to the second calculation node used in the model reasoning for calculation, and perform communication processing on the first calculation result of the next sub-data based on the first communication node while calculating the first communication result of the received sub-data based on the second calculation node; the first communication result of the sub-data is obtained by the first communication node performing communication processing on the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model inference method described above is implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the model reasoning methods described above.
[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned model reasoning methods.
[0015] The model reasoning method, device, electronic device and storage medium provided by the present invention divide the input data into multiple sub-data, so that while the first communication result of a received sub-data is calculated based on the second computing node, the first calculation result of the next sub-data is communicated and processed based on the first communication node, thereby realizing the parallelism of communication and calculation. As a result, under distributed model reasoning, most of the time consumed by communication operations can be hidden by the calculation time, and the situation where the computing nodes are idle is minimized, thereby effectively improving the efficiency of model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1It is a flowchart of model reasoning in related technologies.
[0018] Figure 2 It is a schematic diagram of the distribution of model inference time in related technologies.
[0019] Figure 3 It is one of the flow charts of the model reasoning method provided by the present invention.
[0020] Figure 4 This is the second flow chart of the model reasoning method provided by the present invention.
[0021] Figure 5 This is one of the distribution diagrams of the time consumption of model reasoning provided by the present invention.
[0022] Figure 6 It is one of the schematic diagrams for segmenting input data provided by the present invention.
[0023] Figure 7 This is the third flow chart of the model reasoning method provided by the present invention.
[0024] Figure 8 This is the second schematic diagram of segmentation of input data provided by the present invention.
[0025] Fig. 9 This is the fourth flow chart of the model reasoning method provided by the present invention.
[0026] Fig.10 This is the fifth flow chart of the model reasoning method provided by the present invention.
[0027] Fig.11 It is a structural schematic diagram of the model reasoning device provided by the present invention.
[0028] Fig.12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] With the rapid development of artificial intelligence technology, especially the rise of large-scale language models, the number of model parameters has increased dramatically to support more complex and accurate natural language processing tasks.
[0031] The huge number of parameters also means high demand for computing resources. It is usually necessary to deploy the model to multiple graphics cards or even multiple servers for distributed reasoning to speed up the processing process and meet real-time requirements.
[0032] In distributed reasoning, model partitioning is a common technical means, which accelerates the reasoning process by allocating different parts of the model to different computing nodes and using parallel computing resources. Among them, tensor parallelism is an efficient way to partition the model, which allows different layers or sub-modules of the model to process data in parallel between multiple graphics cards, thereby significantly improving computing efficiency.
[0033] However, although tensor parallelism performs well in accelerating model reasoning, the large number of communication operations it introduces has become a performance bottleneck that cannot be ignored. For example, in a typical distributed reasoning scenario of a large-scale 80-layer language model, the use of tensor parallelism may require up to 160 AllReduce communications, which involve a large amount of data transmission and synchronization waiting, resulting in a significant increase in the overall reasoning latency. Especially in environments with limited network bandwidth or high latency, the impact of communication overhead on reasoning performance is more significant, limiting the real-time performance and efficiency of the model in practical applications.
[0034] In addition, frequent communication operations may also cause the graphics card computing resources to be idle, because the graphics card needs to wait for the communication to be completed before continuing to perform the next round of calculations.
[0035] For example, Figure 1 This is a flow chart of model reasoning in related technologies. Figure 1 As shown, taking the application of node 1, node 2, and node 3 for model reasoning as an example, node 1 and node 3 are both computing nodes, that is, node 1 and node 3 are both nodes that use the computing resources on a single graphics card to run, and node 2 is a communication node, that is, node 2 is specifically a node that uses the communication resources between multiple graphics cards to communicate. In the model reasoning process, after node 1 completes the calculation, the calculation result must be transmitted to node 3 through node 2 before node 3 can start the next round of calculation. While node 2 is transmitting, node 3 is idle, resulting in a decrease in the overall reasoning efficiency. Moreover, in this case, the duration of model reasoning is approximately equal to the sum of the execution times of each node.
[0036] Figure 2 This is a schematic diagram of the distribution of model reasoning time in related technologies. Figure 2 As shown in the distribution of model inference time, it can be seen that the computing time and communication time are distributed in different time periods, that is, new computing can only be performed after the communication is completed, and there is no parallel computing and communication. This imbalance between computing and communication further exacerbates the problem of inference latency and reduces resource utilization.
[0037] For example, when running the prefill scenario on some vendors' multi-card servers, the communication time accounts for about 18% of the overall time, while when running the prefill scenario on other vendors' multi-card servers, the communication time accounts for more than 30% of the overall time.
[0038] In view of the above problems, the present invention provides a model reasoning method. Figure 3 This is one of the flow charts of the model reasoning method provided by the present invention, such as Figure 3 As shown, the method includes: Step 310, dividing the input data into a plurality of sub-data.
[0039] Here, input data refers to data input into the model for model reasoning. The model here can be a large language model, or other models with a large parameter scale that require distributed reasoning, such as a Transformer model.
[0040] For different types of models, the type of input data may also be different. For example, for speech models, such as large language models, the input data may be text; for speech models, such as speech recognition models, speech translation models, etc., the input data may be speech; for image models, such as target detection models, image classification models, and target tracking models, the input data may be images or videos, and the embodiments of the present invention do not specifically limit this.
[0041] In order to improve the efficiency of model reasoning, after obtaining the input data, the input data can be segmented to obtain multiple sub-data. It can be understood that each sub-data here is part of the input data. For example, in the case where the input data is text, the text can be segmented, such as segmenting by sentence, segmenting by paragraph, etc.; for another example, in the case where the input data is speech, the speech can be segmented, such as segmenting by speaker, segmenting by duration, etc.; for another example, in the case where the input data is video, the video can be segmented, such as segmenting by multiple frames of images, and the embodiment of the present invention does not specifically limit this.
[0042] Step 320: Calculate the plurality of sub-data based on the first calculation node used in the model reasoning to obtain a first calculation result for each sub-data.
[0043] Step 330, based on the first communication node used in the model reasoning, perform communication processing on the first calculation result of a sub-data, and transmit it to the second computing node used in the model reasoning for calculation, and while calculating the first communication result of a received sub-data based on the second computing node, perform communication processing on the first calculation result of the next sub-data based on the first communication node; the first communication result of the sub-data is obtained by the first communication node performing communication processing on the first calculation result of the sub-data; the second computing node and the first communication node are deployed on different hardware resources.
[0044] Specifically, in the distributed model reasoning process, it can be applied to computing nodes and communication nodes, and in the embodiment of the present invention, the number of computing nodes and communication nodes is not limited.
[0045] Among them, the computing node is the node used to implement data computing in model reasoning, and different computing nodes can implement different data computing tasks. Each computing node can correspond to an artificial intelligence chip, such as a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit), NPU (Neural network Processing Unit), DPU (Deeplearning Processing Unit), APU (Accelerated Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Unit).
[0046] A communication node is a node used to implement data transmission and communication between different computing nodes in model reasoning. For example, a communication node can perform communication processing on the calculation result of a computing node and transmit the result of the communication processing to another computing node so that the other computing node can perform calculations based on the received communication processing result.
[0047] In the model reasoning process, the first running computing node is recorded as the first computing node, and the communication node that performs communication processing on the computing result of the first computing node and transmits the communication result obtained by the communication processing to the next computing node is recorded as the first communication node. In addition, the computing node that receives the communication result transmitted by the first communication node and performs calculation based on the communication result is recorded as the second computing node.
[0048] After the input data is segmented and multiple sub-data are obtained, the first computing node can be controlled to sequentially calculate the multiple sub-data. In this process, the first computing node can obtain the calculation result of each sub-data, which is recorded as the first calculation result. It can be understood that each sub-data corresponds to a first calculation result.
[0049] In order to improve the model reasoning effect, when the first calculation result is obtained, the first communication node can be controlled to perform communication processing on the first calculation result of each sub-data calculated by the first calculation node one by one, and transmit the first communication result of each sub-data obtained by the communication processing to the second calculation node.
[0050] Accordingly, the second computing node can receive the first communication result of each sub-data transmitted by the first communication node one by one. Moreover, after receiving the first communication result of a sub-data each time, the second computing node can first perform calculation based on the received first communication result of the sub-data.
[0051] That is to say, while the first communication node is performing communication processing on the first calculation result and transmitting the first communication result obtained by the communication processing, the second computing node can synchronously perform calculations on the first communication result that has been received, thereby realizing time multiplexing of communication and calculation, and avoiding the situation where subsequent computing nodes are idle during the communication process, resulting in a reduction in the overall reasoning efficiency.
[0052] It should be noted that in order to achieve synchronous execution of communication based on the first communication node and calculation based on the second computing node, it is necessary to limit the first communication node and the second computing node to be deployed on different hardware resources, thereby ensuring the parallelism of communication and calculation, and then ensuring the improvement of model reasoning efficiency.
[0053] For example, Figure 4 This is the second flow chart of the model reasoning method provided by the present invention. Figure 4As shown, the input data can be divided into 4 sub-data, 1-4 in total. After the first computing node completes the calculation of the 4 sub-data to obtain the first calculation result 1, the first calculation result 2, the first calculation result 3, and the first calculation result 4, the first communication node can sequentially perform communication processing on the first calculation result 1, the first calculation result 2, the first calculation result 3, and the first calculation result 4, thereby sequentially obtaining the first communication result 1, the first communication result 2, the first communication result 3, and the first communication result 4 and transmitting them to the second computing node. Moreover, during the transmission process of the first communication node, the second computing node can perform calculations on the first communication result 1 after receiving the first communication result 1, that is, while the first communication node transmits the first communication result 2 to the second computing node, the second computing node can perform calculations on the first communication result 1 that has been received. Similarly, while the first communication node transmits the first communication result 3 to the second computing node, the second computing node can perform calculations on the first communication result 2 that has been received, and while the first communication node transmits the first communication result 4 to the second computing node, the second computing node can perform calculations on the first communication result 3 that has been received. It can be understood that Figure 4 There is a dependency relationship between the data marked with the same number, that is, the first communication result 1 used for calculation at the second computing node needs to wait for the first communication result 1 at the first communication node to be transmitted before the calculation can start.
[0054] Figure 5 This is one of the distribution diagrams of the model reasoning time provided by the present invention. Figure 5 As shown, during the communication process, the next computing node can start computing after receiving the communication result of the first sub-data, thereby realizing the parallelism of communication and computing, hiding the time consumption of most communication nodes, and effectively improving the efficiency of model reasoning.
[0055] In the method provided in an embodiment of the present invention, the input data is divided into multiple sub-data, so that while the second computing node is used to calculate the received first communication result, the first communication node is used to communicate and process the first calculation result of the next sub-data, thereby achieving parallel communication and calculation. As a result, under distributed model reasoning, most of the time spent on communication operations can be hidden by the calculation time, and the situation where the computing nodes are idle is minimized, thereby effectively improving the efficiency of model reasoning.
[0056] Furthermore, in the method provided in the embodiment of the present invention, different hardware resources are divided for computing nodes and communication nodes, and combined with the feature that most nodes in model reasoning support independent processing in units of tokens, the input data is segmented, thereby making it possible to perform computing and communication in parallel, thereby reducing the model reasoning delay and improving the model reasoning efficiency.
[0057] Based on the above embodiment, in step 320, the first computing node applied in the model-based reasoning performs computing on the multiple sub-data, including: Based on the first computing node used in the model reasoning, sequentially computing the plurality of sub-data; Accordingly, in step 330, the first communication node based on the application in the model reasoning performs communication processing on the first calculation result of a sub-data, including: While calculating one sub-data based on the first computing node, performing communication processing on a first calculation result of the previous sub-data based on the first communication node; The first computing node and the first communication node are deployed on different hardware resources.
[0058] Specifically, in order to improve the efficiency of model reasoning, in the process of calculating multiple sub-data based on the first computing node, the first computing node can be applied to calculate the multiple sub-data one by one. Thus, each time the first computing node completes the calculation of one of the sub-data, that is, each time the first calculation result of a sub-data is obtained, the first communication node can be controlled to perform communication processing on the obtained first calculation result and then transmit it to the second computing node.
[0059] That is to say, during the calculation process of the first computing node, the first communication node can synchronously perform communication processing and transmission on the first calculation result of the sub-data that has been calculated, thereby realizing time multiplexing of calculation and communication, thereby further improving the efficiency of model reasoning.
[0060] It should be noted that in order to achieve synchronous execution of calculations based on the first computing node and communication based on the first communication node, it is necessary to limit the first computing node and the first communication node to be deployed on different hardware resources, thereby ensuring the parallelism of calculations and communications, and further ensuring the improvement of model reasoning efficiency.
[0061] Based on any of the above embodiments, in step 310, dividing the input data into a plurality of sub-data includes: The input data is divided into multiple sub-data in batches, and the model reasoning corresponding to each sub-data is independent of each other.
[0062] Specifically, when there are multiple batches of input data, the input data can be segmented in batches to obtain multiple sub-data. It can be understood that each sub-data obtained here is a batch of data, and since the processing of each batch of data is completely independent during the model reasoning process, different batches do not affect each other, so the model reasoning process corresponding to each sub-data is also independent of each other. In the subsequent model reasoning process, the calculation results of different sub-data can be transmitted to the subsequent calculation units for calculation, thereby greatly improving the efficiency of model reasoning.
[0063] For example, Figure 6 This is one of the schematic diagrams for segmenting the input data provided by the present invention. Figure 6 As shown, the input data includes 8 batches of data. Considering that each batch of data is completely independent in processing, it can be divided according to batches, thereby obtaining sub-data of each of the 8 batches.
[0064] Figure 7 This is the third flow chart of the model reasoning method provided by the present invention. Figure 7 As shown in the figure, for the multiple sub-data obtained by batch segmentation, FFN (Feed-Forward Neural Network) calculations can be performed on the computing nodes respectively, or Attention calculations can be performed on the computing nodes respectively. In this process, the calculation results obtained for each sub-data can be transmitted through the communication node through Allreduce.
[0065] Based on any of the above embodiments, step 310, dividing the input data into a plurality of sub-data, includes: The input data is divided into a plurality of sub-data according to the data volume, and the word units contained in each sub-data are complete word units.
[0066] Specifically, when the input data cannot be divided based on batches, the input data can also be divided based on the data volume of the input data. Here, for different types of input data, the corresponding data volume measurement is also different. For example, for text-type input data, the data can be divided based on the text length as the data volume. For example, for video-type input data, the data can be divided based on the number of image frames as the data volume. For example, for voice-type input data, the data can be divided based on the number of voice frames as the data volume.
[0067] Each sub-data obtained in this way may require full data on some computing nodes in model reasoning, that is, all sub-data to realize calculation. However, since most nodes can realize independent token calculation functions in model reasoning, the realization of this part of the calculation function can also realize part of the time reuse of communication and calculation, thereby achieving the effect of improving the efficiency of model reasoning.
[0068] Furthermore, in order to ensure that the independent computing function of the word unit can be realized, when the input data is segmented based on the data volume, it is necessary to ensure that the word units contained in each sub-data obtained by segmentation are complete word units, that is, the segmentation will not cause the complete word units to be cut off. The unit referred to here is also the same as the type of input data. For example, for text-type input data, a token is a word, a word, or a character; for image-type input data, each small patch of the image can be a token.
[0069] For example, Figure 8 This is the second schematic diagram of the segmentation of input data provided by the present invention. Figure 8 As shown, in the case where a single batch of text is used as input data, the text can be segmented based on the length, thereby segmenting the text into four text segments, namely, four sub-data.
[0070] The four sub-data obtained by this segmentation can be calculated independently in units of Tokens in some computing nodes, while in some computing nodes, the full amount of sub-data is required for calculation.
[0071] For example, Fig. 9 This is the fourth flow chart of the model reasoning method provided by the present invention. Fig. 9 As shown, for multiple sub-data obtained by segmentation according to the data volume, FFN calculations can be performed on the computing nodes respectively. In addition, in this process, the calculation results of each sub-data can be transmitted through the communication node through Allreduce.
[0072] Fig.10 This is the fifth flow chart of the model reasoning method provided by the present invention. Fig.10 As shown in the figure, for multiple sub-data obtained by segmentation according to the data volume, Attention calculation can be performed on the computing node in the form of full data. As a result, the time required to obtain the calculation result of each sub-data may be extended, but Allreduce transmission can still be performed through the communication node after the calculation result is obtained. As a result, part of the communication process can still be effectively hidden in the calculation process, thereby improving the efficiency of model reasoning.
[0073] It should be noted that in the scheme for splitting input data in the above embodiment, no matter it is split by batch or by data volume, when processing large-scale input data, each computing node used in model inference needs to consume all computing power to perform calculations for each sub-data, and each communication node needs to consume all bandwidth to transmit the calculation results of the sub-data. That is, splitting the input data does not cause the nodes to be idle, and the maximum utilization of the nodes can still be guaranteed.
[0074] Based on this, from the perspective of the overall process of model reasoning, splitting the input data will hardly affect the end-to-end latency during model reasoning, and can ensure that communication and computing can be carried out in parallel.
[0075] Based on any of the above embodiments, after step 330, the method further includes: While calculating the first communication result of the next sub-data based on the second computing node, the second computing result of one sub-data is communicated and processed based on the second communication node used in the model reasoning, and transmitted to the third computing node used in the model reasoning for calculation, wherein the second computing result of the one sub-data is obtained by calculating the first communication result of the one sub-data by the second computing node; and, while calculating the second communication result of the received one sub-data based on the third computing node, performing communication processing on the second calculation result of the next sub-data based on the second communication node, the second communication result of the one sub-data being obtained by the second communication node performing communication processing on the second calculation result of the one sub-data; The second computing node and the second communication node are deployed on different hardware resources, and the third computing node and the second communication node are deployed on different hardware resources.
[0076] Specifically, in the model reasoning process, the communication node that performs communication processing on the calculation result of the second calculation node and transmits it to the next calculation node can be recorded as the second communication node. In addition, the calculation node that receives the communication result transmitted by the second communication node and performs calculation based on the communication result can be recorded as the third calculation node.
[0077] During the process of the first communication node performing communication processing and transmission of the first calculation result, the second calculation node can synchronously perform calculations on the first communication result that has been received, that is, the second calculation node can perform calculations on the received first communication results one by one. Thus, the second calculation node can output the calculation results for each first communication result one by one, that is, the second calculation results.
[0078] Therefore, each time the second computing node completes the calculation of one of the sub-data, that is, each time the second calculation result of a sub-data is obtained, the second communication node can be controlled to perform communication processing on the obtained second calculation result of the sub-data and transmit it to the third computing node.
[0079] That is to say, while the second computing node is calculating the next sub-data of the sub-data, the second communication node can synchronously perform communication processing on the second calculation result of the sub-data that has been calculated to obtain and transmit the second communication result, thereby realizing time multiplexing of calculation and communication, thereby further improving the efficiency of model reasoning.
[0080] It should be noted that in order to achieve the synchronous execution of calculation based on the second computing node and communication based on the second communication node, it is necessary to limit the second computing node and the second communication node to be deployed on different hardware resources, thereby ensuring the parallelism of calculation and communication, and then ensuring the improvement of model reasoning efficiency.
[0081] Furthermore, while the second communication node is transmitting the second communication result of the next sub-data of a sub-data, the third computing node can synchronously perform calculations on the second communication result of the sub-data that has been received, thereby achieving time multiplexing of communication and calculation, and avoiding the situation where subsequent computing nodes are idle during the communication process, resulting in a reduction in overall reasoning efficiency.
[0082] It should be noted that in order to achieve the synchronous execution of communication based on the second communication node and calculation based on the third computing node, it is necessary to limit the second communication node and the third computing node to be deployed on different hardware resources, thereby ensuring the parallelism of communication and calculation, and then ensuring the improvement of model reasoning efficiency.
[0083] Based on any of the above embodiments, the first computing node and the second computing node are deployed in a stream processor cluster, and the first communication node is deployed in a direct memory read module.
[0084] Specifically, computing nodes and communication nodes are deployed on different hardware resources, thereby ensuring that computing and communication can be implemented synchronously. Among them, computing nodes can be deployed in a stream processing cluster (Streaming Processing Cluster, SPC), and communication nodes can be deployed in a direct memory access (Direct Memory Access, DMA) module. Furthermore, different computing nodes, such as the first computing node and the second computing node, and for example, the second computing node and the third computing node can be deployed in the same stream processor cluster, or in different stream processor clusters; different communication nodes, such as the first communication node and the second communication node can also be deployed in the same direct memory access module, or in different direct memory access modules, and the embodiments of the present invention do not specifically limit this.
[0085] It should be noted that there are some hardware resources in the related art that can be used for both computing and communication. Although the communication function implemented by such hardware resources has higher performance, because the communication nodes and computing nodes occupy the same hardware resources under such hardware resources, communication and computing cannot be carried out in parallel, which affects the optimization of model reasoning performance. Therefore, in the embodiment of the present invention, the hardware resources deployed by the communication nodes and computing nodes are differentiated from the software level, thereby ensuring that communication and computing can be carried out in parallel during the model reasoning process.
[0086] The model reasoning device provided by the present invention is described below. The model reasoning device described below and the model reasoning method described above can be referenced to each other.
[0087] Fig.11 Schematic diagram of the structure of the model inference device provided by the present invention. Fig.11 As shown, the device comprises: A segmentation unit 1110, used for segmenting the input data into a plurality of sub-data; A first reasoning unit 1120, configured to calculate the plurality of sub-data based on a first computing node used in model reasoning to obtain a first calculation result for each sub-data; The second reasoning unit 1130 is used to perform communication processing on a first calculation result of a sub-data based on the first communication node used in the model reasoning, and transmit it to the second calculation node used in the model reasoning for calculation, and perform communication processing on the first calculation result of the next sub-data based on the first communication node while calculating the first communication result of the received sub-data based on the second calculation node; the first communication result of the sub-data is obtained by the first communication node performing communication processing on the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
[0088] In the device provided in the embodiment of the present invention, the input data is divided into multiple sub-data, so that while the second computing node is used to calculate the received first communication result, the first communication node is used to communicate and process the first calculation result of the next sub-data, thereby achieving parallel communication and calculation. As a result, under distributed model reasoning, most of the time spent on communication operations can be hidden by the calculation time, and the situation where the computing nodes are idle is minimized, thereby effectively improving the efficiency of model reasoning.
[0089] Furthermore, in the device provided in the embodiment of the present invention, different hardware resources are divided for computing nodes and communication nodes, and combined with the feature that most nodes in model reasoning support independent processing in units of tokens, the input data is segmented, thereby making it possible to perform computing and communication in parallel, thereby reducing the model reasoning delay and improving the model reasoning efficiency.
[0090] Based on any of the foregoing embodiments, the first reasoning unit is specifically configured to: Based on the first computing node used in the model reasoning, sequentially computing the plurality of sub-data; The second reasoning unit is specifically used for: While calculating one sub-data based on the first computing node, performing communication processing on a first calculation result of the previous sub-data based on the first communication node; The first computing node and the first communication node are deployed on different hardware resources.
[0091] Based on any of the above embodiments, the segmentation unit is specifically used for: The input data is divided into multiple sub-data in batches, and the model reasoning corresponding to each sub-data is independent of each other.
[0092] Based on any of the above embodiments, the segmentation unit is specifically used for: The input data is divided into a plurality of sub-data according to the data volume, and the word units contained in each sub-data are complete word units.
[0093] Based on any of the above embodiments, the device further includes a third reasoning unit, configured to: While calculating the first communication result of the next sub-data based on the second computing node, the second computing result of one sub-data is communicated and processed based on the second communication node used in the model reasoning, and transmitted to the third computing node used in the model reasoning for calculation, wherein the second computing result of the one sub-data is obtained by calculating the first communication result of the one sub-data by the second computing node; and, while calculating the second communication result of the received one sub-data based on the third computing node, performing communication processing on the second calculation result of the next sub-data based on the second communication node, the second communication result of the one sub-data being obtained by the second communication node performing communication processing on the second calculation result of the one sub-data; The second computing node and the second communication node are deployed on different hardware resources, and the third computing node and the second communication node are deployed on different hardware resources.
[0094] Based on any of the above embodiments, the first computing node and the second computing node are deployed in a stream processor cluster, and the first communication node is deployed in a direct memory read module.
[0095] Fig.12 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230 and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call the logic instructions in the memory 1230 to execute the model reasoning method, which includes: Split the input data into multiple sub-data; Based on the first computing node used in the model reasoning, the plurality of sub-data are calculated to obtain a first calculation result of each sub-data; Based on the first communication node used in the model reasoning, a first calculation result of a sub-data is communicated and processed, and transmitted to the second calculation node used in the model reasoning for calculation, and while the first communication result of the received sub-data is calculated based on the second calculation node, a first calculation result of the next sub-data is communicated and processed based on the first communication node; the first communication result of the sub-data is obtained by the first communication node communicating and processing the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
[0096] In addition, the logic instructions in the above-mentioned memory 1230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0097] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the model reasoning method provided by the above methods, the method comprising: Split the input data into multiple sub-data; Based on the first computing node used in the model reasoning, the plurality of sub-data are calculated to obtain a first calculation result of each sub-data; Based on the first communication node used in the model reasoning, a first calculation result of a sub-data is communicated and processed, and transmitted to the second calculation node used in the model reasoning for calculation, and while the first communication result of the received sub-data is calculated based on the second calculation node, a first calculation result of the next sub-data is communicated and processed based on the first communication node; the first communication result of the sub-data is obtained by the first communication node communicating and processing the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
[0098] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the model reasoning method provided by the above methods, the method comprising: Split the input data into multiple sub-data; Based on the first computing node used in the model reasoning, the plurality of sub-data are calculated to obtain a first calculation result of each sub-data; Based on the first communication node used in the model reasoning, a first calculation result of a sub-data is communicated and processed, and transmitted to the second calculation node used in the model reasoning for calculation, and while the first communication result of the received sub-data is calculated based on the second calculation node, a first calculation result of the next sub-data is communicated and processed based on the first communication node; the first communication result of the sub-data is obtained by the first communication node communicating and processing the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A model reasoning method, characterized in that: include: Split the input data into multiple sub-data; Based on the first computing node used in the model reasoning, the plurality of sub-data are calculated to obtain a first calculation result of each sub-data; Based on the first communication node used in the model reasoning, a first calculation result of a sub-data is communicated and processed, and transmitted to the second calculation node used in the model reasoning for calculation, and while the first communication result of the received sub-data is calculated based on the second calculation node, a first calculation result of the next sub-data is communicated and processed based on the first communication node; the first communication result of the sub-data is obtained by the first communication node communicating and processing the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
2. The model reasoning method according to claim 1, characterized in that: The first computing node applied in the model-based reasoning, computing the plurality of sub-data, comprises: Based on the first computing node used in the model reasoning, sequentially computing the plurality of sub-data; The first communication node applied in the model reasoning performs communication processing on a first calculation result of a sub-data, including: While calculating one sub-data based on the first computing node, performing communication processing on a first calculation result of the previous sub-data based on the first communication node; The first computing node and the first communication node are deployed on different hardware resources.
3. The model reasoning method according to claim 1, characterized in that: The step of dividing the input data into a plurality of sub-data includes: The input data is divided into multiple sub-data in batches, and the model reasoning corresponding to each sub-data is independent of each other.
4. The model reasoning method according to claim 1, characterized in that: The step of dividing the input data into a plurality of sub-data includes: The input data is divided into a plurality of sub-data according to the data volume, and the word units contained in each sub-data are complete word units.
5. The model reasoning method according to any one of claims 1 to 4, characterized in that: The model reasoning method also includes: While calculating the first communication result of the next sub-data based on the second computing node, the second computing result of one sub-data is communicated and processed based on the second communication node used in the model reasoning, and transmitted to the third computing node used in the model reasoning for calculation, wherein the second computing result of the one sub-data is obtained by calculating the first communication result of the one sub-data by the second computing node; and, while calculating the second communication result of the received one sub-data based on the third computing node, performing communication processing on the second calculation result of the next sub-data based on the second communication node, the second communication result of the one sub-data being obtained by the second communication node performing communication processing on the second calculation result of the one sub-data; The second computing node and the second communication node are deployed on different hardware resources, and the third computing node and the second communication node are deployed on different hardware resources.
6. The model reasoning method according to any one of claims 1 to 4, characterized in that: The first computing node and the second computing node are deployed in a stream processor cluster, and the first communication node is deployed in a direct memory read module.
7. A model reasoning device, characterized in that: include: A segmentation unit, used for segmenting the input data into a plurality of sub-data; A first reasoning unit, configured to calculate the plurality of sub-data based on a first computing node used in model reasoning to obtain a first calculation result for each sub-data; A second reasoning unit is used to perform communication processing on a first calculation result of a sub-data based on the first communication node used in the model reasoning, and transmit it to the second calculation node used in the model reasoning for calculation, and perform communication processing on the first calculation result of the next sub-data based on the first communication node while calculating the first communication result of the received sub-data based on the second calculation node; the first communication result of the sub-data is obtained by the first communication node performing communication processing on the first calculation result of the sub-data; The second computing node and the first communication node are deployed on different hardware resources.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the model reasoning method as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model reasoning method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model reasoning method according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Processing method and device of reasoning model, electronic equipment and storage medium
CN121212368A
Model reasoning method and device, computer equipment, computer readable storage medium and computer program product
CN121960793A
