Model reasoning method applied to computing device, computing device, medium and program product

Abstract

The invention provides a model reasoning method applied to computing equipment, the computing equipment, a medium and a program product, and can be applied to the technical field of artificial intelligence. The method comprises the steps that a control unit sends a to-be-processed task to a plurality of calculation units, each calculation unit is configured with at least two data subsets, and the data subsets are obtained by splitting a weight set of a target reasoning model needed for executing the to-be-processed task; the plurality of calculation units respectively execute the to-be-processed task to obtain a plurality of sub-processing results; the control unit obtains a target execution result according to the multiple sub-processing results; any one of the plurality of computing units executes the to-be-processed task according to the following mode: processing the to-be-processed task in sequence by utilizing the configured at least two data subsets to obtain at least two sub-processing results, and executing the to-be-processed task according to any one of the plurality of computing units under the condition that any one sub-processing result is obtained. And synchronously sending any sub-processing result to other computing units in the plurality of computing units for processing the tasks to be processed in parallel.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model reasoning method applied to a computing device, a computing device, a medium and a program product. Background Art

[0002] With the successful application of deep learning models in various fields, people have begun to pay attention to how to expand deep learning models to a larger scale to improve the data processing capabilities, accuracy and performance of the models. Based on this, ultra-large-scale deep learning models came into being. Ultra-large-scale deep learning models face the pressure of memory and reasoning speed. However, the memory of a single computing unit is very limited. Therefore, a computing strategy in which each computing unit jointly performs reasoning is proposed. For example, in the computing strategy in which each computing unit jointly performs reasoning, how to reasonably utilize the computing resources of the computing unit to achieve rapid response to users is crucial. Summary of the invention

[0003] In view of the above problems, the present invention provides a model reasoning method applied to a computing device, a computing device, a medium and a program product.

[0004] According to a first aspect of the present invention, there is provided a model reasoning method applied to a computing device, the computing device comprising: a control unit and a plurality of computing units for executing tasks to be processed in parallel, the model reasoning method comprising: the control unit sends the tasks to be processed to the plurality of computing units respectively, any of the plurality of computing units is configured with at least two data subsets, the data subsets are obtained by splitting a weight set of a target reasoning model required to execute the tasks to be processed; the plurality of computing units respectively execute the tasks to be processed to obtain a plurality of sub-processing results; the control unit obtains a target execution result of the tasks to be processed based on the plurality of sub-processing results; wherein any of the plurality of computing units executes the tasks to be processed in the following manner: using the at least two configured data subsets, processing the tasks to be processed in sequence to obtain at least two sub-processing results, wherein, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units in the plurality of computing units that process the tasks to be processed in parallel.

[0005] The second aspect of the present invention provides a computing device, comprising: a control unit, used to send the tasks to be processed to multiple computing units respectively; multiple computing units, any computing unit among the multiple computing units is configured with at least two data subsets, and the data subsets are obtained by splitting the weight set of the target inference model required to execute the tasks to be processed; the multiple computing units are used to execute the tasks to be processed respectively to obtain multiple sub-processing results; the control unit is also used to obtain the target execution result of the task to be processed based on the multiple sub-processing results; wherein any computing unit among the multiple computing units is used to execute the tasks to be processed in the following manner: using the configured at least two data subsets, processing the tasks to be processed in sequence to obtain at least two sub-processing results, wherein, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units among the multiple computing units that process the tasks to be processed in parallel.

[0006] The third aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0007] The fourth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0008] According to an embodiment of the present invention, since at least two data subsets are configured in the computing unit, and the tasks to be processed are processed in sequence using the configured at least two data subsets, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units in the multiple computing units that process the tasks to be processed in parallel, therefore, the computing unit calculation and communication are parallel, and after traversing the configured at least two data subsets, only the sub-processing result obtained by the last processing of the tasks to be processed needs to be sent. Compared with the computing unit calculation and communication serial, since the sub-processing result obtained by the last processing of the tasks to be processed is only a small part of the sub-processing result of the computing unit executing the tasks to be processed, when the tasks to be processed are completed, the communication time and the amount of communication data after the computing unit calculates can be reduced, thereby reducing the idle time of the computing unit computing resources, improving the computing efficiency and resource utilization of the computing unit, and thus improving the task processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0010] Figure 1 An application scenario diagram of a model reasoning method applied to a computing device, a computing device, a medium, and a program product according to an embodiment of the present invention is shown.

[0011] Figure 2 A flow chart of a model reasoning method applied to a computing device according to an embodiment of the present invention is shown.

[0012] Figure 3A A schematic diagram of executing tasks to be processed in parallel according to an embodiment of the present invention is shown.

[0013] Figure 3B A schematic diagram showing any computing unit executing a task to be processed according to an embodiment of the present invention is shown.

[0014] Figure 4 A schematic diagram of determining a target execution result of a task to be processed according to an embodiment of the present invention is shown.

[0015] Figure 5 A structural block diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0016] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0017] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0018] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0019] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0020] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, invention and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0021] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0022] In the process of implementing the embodiments of the present invention, it is found that in the computing strategy of each computing unit jointly executing reasoning, how to reasonably utilize the computing power of the computing unit to achieve rapid response to users is crucial. Although tensor parallelism realizes simultaneous reasoning of multiple cards through multi-card or multi-machine parallel computing, which meets the deployment and reasoning requirements of ultra-large-scale models, the calculation and data communication of a single card in tensor parallelism are executed serially. After the calculation is completed, the calculation result is transmitted through inter-card communication to complete the reduction operation. Since the computing resources of the card are idle during inter-card communication transmission, the computing resources of the card will be wasted. In addition, after the calculation is completed, the calculation result is transmitted through inter-card communication. If the calculation result needs to occupy more transmission resources, it cannot respond to the user in time, which will affect the user experience.

[0023] Based on this, an embodiment of the present invention provides a model reasoning method applied to a computing device, the computing device comprising: a control unit and a computing unit for executing tasks to be processed in parallel, the model reasoning method comprising: the control unit sends the tasks to be processed to multiple computing units respectively, any of the multiple computing units is configured with at least two data subsets, the data subsets are obtained by splitting the weight set of the target reasoning model required to execute the tasks to be processed; the multiple computing units respectively execute the tasks to be processed to obtain multiple sub-processing results; the control unit obtains the target execution result of the task to be processed based on the multiple sub-processing results; wherein any of the multiple computing units executes the tasks to be processed in the following manner: using at least two configured data subsets, processing the tasks to be processed in sequence to obtain at least two sub-processing results, wherein, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units in the multiple computing units that process the tasks to be processed in parallel.

[0024] Figure 1 An application scenario diagram of a model reasoning method applied to a computing device, a computing device, a medium, and a program product according to an embodiment of the present invention is shown.

[0025] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a control unit 101, a first computing unit 102_1, a second computing unit 102_2, ..., and an Nth computing unit 102_N. N is an integer greater than or equal to 2. The control unit 101 and any computing unit among the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N, and any two computing units among the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N interact through a network, and the network may be a medium that provides a communication link. The network may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0026] The user can use the terminal device to interact with the control unit 101 to receive the target execution result or send the task to be processed, etc. The first computing unit 102_1, the second computing unit 102_2, ..., the Nth computing unit 102_N can be installed on the terminal device. The terminal device may include various communication client applications, such as web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples). The terminal device can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc.

[0027] The control unit 101 may be a central processing unit or other devices in a server. The control unit 101 may also include at least one of the following: a baseboard management controller (BMC), a complex programmable logic device (CPLD) microcontroller, a field programmable gate array (FPGA), etc.

[0028] For example, the control unit 101 may send the to-be-processed task input by the user to the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N, respectively. After the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N respectively execute the to-be-processed task to obtain a plurality of sub-processing results, the plurality of sub-processing results are processed to obtain a target execution result of the to-be-processed task. The target execution result is fed back to the terminal device.

[0029] The first computing unit 102_1, the second computing unit 102_2, ..., the Nth computing unit 102_N may each include a graphics processor or a tensor processing unit, etc.

[0030] It should be understood that Figure 1 The number of control units and computing units in the embodiment is only for illustration. Any number of control units and computing units may be provided according to implementation requirements.

[0031] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The model reasoning method applied to a computing device according to an embodiment of the present invention is described in detail.

[0032] Figure 2 A flow chart of a model reasoning method applied to a computing device according to an embodiment of the present invention is shown.

[0033] like Figure 2 As shown, the model reasoning method applied to the computing device of this embodiment includes operations S210 to S230, and the model reasoning method can be executed by the computing device. The computing device may include: a control unit and a computing unit for executing tasks to be processed in parallel.

[0034] In operation S210 , the control unit sends the tasks to be processed to the plurality of computing units respectively.

[0035] In operation S220 , the plurality of computing units respectively execute the tasks to be processed to obtain a plurality of sub-processing results.

[0036] In operation S230, the control unit obtains a target execution result of the task to be processed according to the plurality of sub-processing results.

[0037] In an embodiment of the present invention, the tasks to be processed may be determined according to user needs. For example, the tasks to be processed may include but are not limited to image classification tasks, text generation tasks, question answering tasks, query tasks, object detection, semantic segmentation, sentiment analysis, etc.

[0038] In an embodiment of the present invention, any computing unit among the multiple computing units may be configured with at least two data subsets, where the data subsets are obtained by splitting a weight set of a target reasoning model required to execute a task to be processed.

[0039] For example, for text generation tasks, the target inference model can be a large language model. The weight set can be determined based on the weight file of the large language model, and then the weight set can be split to obtain multiple data subsets. The weight set of the large language model can refer to the set of all parameters in the model, which can include weight matrices, bias terms, normalization parameters, etc. These parameters can be split to obtain multiple unrelated data subsets, and the multiple data subsets can be configured to computing units for parallel execution of tasks to be processed, with at least two data subsets configured on each computing unit.

[0040] In an embodiment of the present invention, any computing unit among multiple computing units executes the task to be processed in the following manner: using at least two configured data subsets, processing the task to be processed in sequence to obtain at least two sub-processing results, wherein, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units among the multiple computing units that process the task to be processed in parallel.

[0041] For example, processing the tasks to be processed in sequence may be performing calculations on any data subset of at least two data subsets and the tasks to be processed, such as matrix multiplication, activation function, convolution operation, pooling operation, normalization operation, and other operations.

[0042] For example, the control unit may combine multiple sub-processing results to obtain the target execution result of the task to be processed. In special cases, the control unit may also determine a main computing unit from multiple computing units, and then the main computing unit returns the target execution result of the task to be processed obtained based on the multiple sub-processing results to the control unit.

[0043] According to an embodiment of the present invention, since at least two data subsets are configured in the computing unit, and the tasks to be processed are processed in sequence using the configured at least two data subsets, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units in the multiple computing units that process the tasks to be processed in parallel, therefore, the computing unit calculation and communication are parallel, and after traversing the configured at least two data subsets, only the sub-processing result obtained by the last processing of the tasks to be processed needs to be sent. Compared with the computing unit calculation and communication serial, since the sub-processing result obtained by the last processing of the tasks to be processed is only a small part of the sub-processing result of the computing unit executing the tasks to be processed, when the tasks to be processed are completed, the communication time and the amount of communication data after the computing unit calculates can be reduced, thereby reducing the idle time of the computing unit computing resources, improving the computing efficiency and resource utilization of the computing unit, and thus improving the task processing efficiency.

[0044] According to an embodiment of the present invention, the at least two data subsets may include N data subsets, where N is an integer greater than or equal to 2. Using the configured at least two data subsets, sequentially processing the tasks to be processed, and obtaining at least two sub-processing results may include an operation: using the first data subset of the N data subsets to process the tasks to be processed, and obtaining the first sub-processing result. When n is greater than or equal to 2 and less than or equal to N, using the nth data subset of the N data subsets to process the tasks to be processed, obtaining the nth sub-processing result, and sending the n-1th sub-processing result to other computing units.

[0045] Figure 3A A schematic diagram showing parallel execution of tasks to be processed according to an embodiment of the present invention is shown; Figure 3B A schematic diagram showing any computing unit executing a task to be processed according to an embodiment of the present invention is shown.

[0046] In the embodiment of the present invention, three computing units, each of which is configured with three data subsets, are taken as an example. It should be noted that the number of computing units and data subsets in this example is only exemplary, and any number of computing units and data subsets may be used according to implementation requirements.

[0047] like Figure 3AAs shown, the control unit can send the pending task 301 to the first computing unit 102_1, the second computing unit 102_2 and the third computing unit 102_3 respectively. The first computing unit 102_1, the second computing unit 102_2 and the third computing unit 102_3 execute the pending task 301 in parallel to obtain multiple sub-processing results 303, such as the first computing unit 102_1 uses the configured three data subsets to sequentially process the pending task 301 to obtain the first sub-processing result, the second sub-processing result and the third sub-processing result, the second computing unit 102_2 uses the configured three data subsets to sequentially process the pending task 301 to obtain the fourth sub-processing result, the fifth sub-processing result and the sixth sub-processing result, and the third computing unit 102_3 uses the configured three data subsets to sequentially process the pending task 301 to obtain the seventh sub-processing result, the eighth sub-processing result and the ninth sub-processing result. It should be noted that the three data subsets configured by the first computing unit 102_1, the second computing unit 102_2 and the third computing unit 102_3 are all different.

[0048] For any computing unit among the first computing unit 102_1, the second computing unit 102_2 and the third computing unit 102_3, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units that are processing the tasks to be processed in parallel.

[0049] For example, Figure 3B As shown, taking the first computing unit 102_1 as an example, when receiving the task to be processed 301, the first data subset 302_1 is used to process the task to be processed 301, and the first sub-processing result 303_1 is obtained. The first sub-processing result 303_1 is sent to the second computing unit 102_2 and the third computing unit 102_3 respectively, and the second data subset 302_2 is used to process the task to be processed, and the second sub-processing result 303_2 is obtained. The second sub-processing result 303_2 is sent to the second computing unit 102_2 and the third computing unit 102_3 respectively, and the third data subset 302_3 is used to process the task to be processed, and the third sub-processing result 303_3 is obtained, and the third sub-processing result 303_3 is sent to the second computing unit 102_2 and the third computing unit 102_3 respectively.

[0050] According to an embodiment of the present invention, since the nth data subset is used to process the task to be processed and the nth sub-processing result is obtained, and the n-1th sub-processing result is sent to other computing units at the same time, when the Nth data subset is used to process the task to be processed and the Nth sub-processing result is obtained, it is only necessary to send the Nth sub-processing result to other computing units, and the Nth sub-processing result is only 1 / N of the target execution result. Therefore, after the computing unit completes processing the task to be processed using the Nth data subset, the time for transmitting the processing results is shortened, thereby shortening the time for executing the task to be processed and improving the reasoning efficiency.

[0051] According to an embodiment of the present invention, any computing unit among the plurality of computing units is configured with a number of execution rounds of the task to be processed.

[0052] Figure 4 A schematic diagram of determining a target execution result of a task to be processed according to an embodiment of the present invention is shown.

[0053] like Figure 4 As shown, determining the target execution result of the task to be processed may include operations S401 to S406.

[0054] In operation S401 , any computing unit among a plurality of computing units processes a task to be processed.

[0055] Exemplarily, when processing a task to be processed, any one of the plurality of computing units obtains an nth sub-processing result and sends it to other computing units.

[0056] In operation S402, it is determined whether the Nth sub-processing result transmitted by other computing units has been received.

[0057] Exemplarily, if it is determined that the Nth sub-processing result sent by other computing units has been received, multiple sub-processing results can be spliced ​​to obtain an initial execution result, and operation S403 is performed. If it is determined that the Nth sub-processing result sent by other computing units has not been received, S401 is continued.

[0058] In operation S403 , it is determined whether the execution round number M of the task to be processed is a single round.

[0059] Exemplarily, when it is determined that the execution round number of the task to be processed is a single time, operation S406 is performed, and the initial execution result can be used as the target execution result. When it is determined that the execution round number of the task to be processed is not a single time, operation S404 is performed.

[0060] In operation S404, based on multiple sub-processing results from other computing units in the m-1th execution of the task to be processed, the mth task to be processed is processed in sequence using at least two configured data subsets to obtain the mth execution result.

[0061] Exemplarily, m is greater than 1, and M is an integer greater than 1.

[0062] Exemplarily, any computing unit may concatenate at least two sub-processing results obtained in the m-1th execution of the task to be processed and multiple sub-processing results from other computing units in the m-1th execution of the task to be processed, to obtain the m-1th execution result obtained in the m-1th execution of the task, and use the m-1th execution result as the mth task to be processed, and use the configured at least two data subsets to sequentially process the mth task to be processed. Since any computing unit can use the sub-processing results sent by other computing units to determine the m-1th execution result when executing the task to be processed for the mth time, when repeatedly executing the task to be processed, there is no need to obtain the task to be processed from the control unit again, saving transmission resources and transmission time for the control unit to send data to the computing unit.

[0063] In operation S405, it is determined whether m=M.

[0064] Exemplarily, when m=M, operation S406 is performed, and the mth execution result can be used as the target execution result. When m≠M, m'=m+1 is set, and then operation S404 is repeatedly performed.

[0065] In operation S406, a target execution result is determined.

[0066] According to the embodiment of the present invention, by determining the number of execution rounds of the task to be processed, different reasoning processes are corresponding to different processing tasks, which has universal applicability. In addition, for tasks to be processed with a large number of execution rounds, since the idle time of the calculation unit in each round is reduced, it is possible to respond to user needs quickly and accurately, thereby improving user experience.

[0067] According to another embodiment of the present invention, the model reasoning method applied to the computing device includes the following steps: Figure 2 In addition to the operations S210 to S230 shown, the control unit may also include an operation of determining the model structure of the target reasoning model according to the configuration file of the target reasoning model, and determining the number of rounds of executing the tasks to be processed in parallel based on the model structure and the tasks to be processed.

[0068] Exemplarily, the target reasoning model is different, and the model structure and weight set are different. When the target reasoning model is trained, the model structure can be stored in a configuration file, and the weight set can be stored in a weight file.

[0069] Exemplarily, different model structures and different tasks to be processed have different numbers of rounds for executing the tasks to be processed. For example, for tasks to be processed involving complex logical reasoning, multi-step decision-making, or generating long texts, and models with complex model structures, the tasks to be processed can be executed in multiple rounds. For simple logical tasks or models with simple structures, the tasks to be processed can be executed once. Models with complex model structures may include, but are not limited to, recurrent neural networks, Transformer models, autoregressive models, etc. Models with simple model structures may include, but are not limited to, convolutional neural networks, fully connected networks, etc.

[0070] For example, semantic analysis may be performed on the tasks to be processed to determine the types of the tasks to be processed, and the number of rounds for parallel execution of the tasks to be processed may be determined based on the types of the tasks to be processed and the model structure.

[0071] For example, if the task to be processed is a text processing task, the type of the text sequence can be obtained as a long text generation type based on the text sequence of the text processing task. The model structure of the target reasoning model is a Transformer model. Then the number of rounds of parallel execution of the task to be processed can be determined based on the length of the text sequence and the model padding. For example, if the model structure of the target reasoning model is a convolutional neural network, a fully connected network, etc., and the task to be processed is a simple logical task, such as a simple question and answer, the number of rounds of parallel execution of the task to be processed is determined to be a single time.

[0072] According to an embodiment of the present invention, by rationally configuring the number of rounds for executing tasks to be processed in parallel through the model structure and the tasks to be processed, the idle time of the computing unit can be reduced, the resource utilization efficiency can be maximized, and the throughput and response speed of the processing tasks can be improved.

[0073] According to another embodiment of the present invention, the model reasoning method applied to a computing device may further include: for high concurrency scenarios, if multiple users send multiple tasks to be processed at the same time, by performing semantic analysis on the multiple tasks to be processed, the types and target reasoning models of the multiple tasks to be processed can be determined, and the multiple tasks to be processed can be sorted according to the types of the multiple tasks to be processed and the model structure of the target reasoning model, and the tasks to be processed can be executed in parallel in sequence according to the sorting.

[0074] For example, the tasks to be processed may be first arranged in descending order based on the complexity of logical reasoning based on their respective types, and then arranged again in descending order based on the complexity of the model structure based on the model structure of the target reasoning model, to obtain the multiple tasks to be processed that are finally sorted.

[0075] According to an embodiment of the present invention, since multiple tasks to be processed are sorted based on the model structure of the target reasoning model and the types of each of the multiple tasks to be processed, simple tasks to be processed can be given priority, thereby reducing the response waiting time of simple tasks. In addition, combined with the parallel strategy of computing and communication when executing the tasks to be processed in parallel, the response waiting time of the task can be further reduced. The combination of the two can quickly respond to user needs and improve user experience.

[0076] According to an embodiment of the present invention, the time required for multiple computing units to process tasks to be processed in the same round is the same.

[0077] In an embodiment of the present invention, the time required to process the task to be processed can be the total time it takes for any computing unit to use at least two configured data subsets to process the task to be processed in sequence, obtain at least two sub-processing results, and send both sub-processing results to other computing units.

[0078] According to an embodiment of the present invention, the time required for multiple computing units to process tasks to be processed in the same round is the same, which can balance the load of multiple computing units and significantly reduce system delays, thereby improving user experience.

[0079] According to another embodiment of the present invention, the model reasoning method applied to the computing device includes the following steps: Figure 2 In addition to the operations S210~S230 shown, the following operations may also be included: the control unit splits the weight set of the target inference model based on the number of multiple computing units to execute the tasks to be processed, the performance parameters of each of the multiple computing units, and the attributes of the weight set to obtain multiple data subsets.

[0080] Exemplarily, the performance parameters may include computing power and memory capacity, etc. The properties of the weight set may include the dimension of the weight matrix, the dimension of the bias term, the dimension of the normalization parameter, etc.

[0081] Exemplarily, the weight set of the target inference model may be split according to a splitting strategy for load balancing among multiple computing units when multiple data subsets are distributed to multiple computing units.

[0082] According to an embodiment of the present invention, the weight set of the target inference model is split based on the number of multiple computing units to execute the task to be processed, the performance parameters of each of the multiple computing units, and the properties of the weight set. This is beneficial to load balancing among multiple computing units when the granularity of the weight set splitting is small.

[0083] According to an embodiment of the present invention, the weight set may include a weight matrix, and the attribute of the weight set may include the dimension of the weight matrix. The control unit splits the weight set based on the number of units of the multiple computing units and the performance parameters of each of the multiple computing units and the attribute of the weight set to obtain multiple data subsets, which may include the following operations: based on the number of units of the multiple computing units and the performance parameters of each of the multiple computing units and the dimension of the weight matrix, the weight matrix is ​​initially split to obtain the number of units of initial sub-matrices; based on the type of task performed by the computing unit, the number of units of the initial sub-matrices are split again to obtain multiple data subsets.

[0084] In an embodiment of the present invention, when the initial sub-matrix is ​​split again, the computing units that perform the same type of tasks may be split into a group.

[0085] For example, if the task type performed by the computing unit is a long text generation task, the matrix rows related to the same attention head in the initial submatrix can be grouped together when split. In this way, each attention head will focus on an independent subspace of the task to be processed without increasing communication.

[0086] According to an embodiment of the present invention, by initially splitting the weight matrix into initial sub-matrices and then splitting them again based on the task type performed by the computing unit, it is possible to ensure that each data subset is used to independently process the task to be processed, and to achieve a finer-grained division of the weight matrix, which is conducive to realizing a strategy of parallel computing and communication.

[0087] According to an embodiment of the present invention, the performance parameter may include computing power. Based on the number of units of multiple computing units, multiple respective performance parameters and the dimension of the weight matrix, the initial splitting of the weight matrix may include the following operations: when the dimension of the weight matrix is ​​divisible by the number of units and the computing power of each of the multiple computing units is the same, the weight matrix is ​​proportionally split into the number of units, the number of initial sub-matrices. In the case where the dimension of the weight matrix is ​​not divisible by the number of units, the weight matrix is ​​split based on the computational complexity of the computing unit to perform the task, so that the computational complexity of the tasks performed by each of the multiple computing units is consistent.

[0088] In an embodiment of the present invention, the computational complexity of tasks executed by multiple computing units may be consistent, which means that the computational complexity of tasks executed by multiple computing units is exactly the same or the difference in computational complexity of tasks executed by any two computing units among the multiple computing units satisfies a predetermined error.

[0089] Exemplarily, when the dimension of the weight matrix is ​​divisible by the number of units and the computing capabilities of the multiple computing units are the same, the weight matrix can be split into initial sub-matrices of the number of units in equal proportion by rows.

[0090] For example, a weight matrix A has a dimension of (p, k). The number of computing units is 2. The weight matrix A can be initially split into two initial sub-matrices, such as A1 and A2. The dimension of each initial sub-matrix is ​​(p / 2, k). The initial sub-matrices, such as A1 and A2, are split again. A1 and A2 are split into four data subsets, such as A11, A12, A13, A14, B11, B12, B13 and B14. The dimensions of A11, A12, A13, A14, B11, B12, B13 and B14 are all (p / 8, k). A11, A12, A13 and A14 can be configured on computing unit 1. B11, B12, B13 and B14 can be configured on computing unit 2.

[0091] According to an embodiment of the present invention, since the weight matrix is ​​split based on computing power, number of units, dimension of the weight matrix and computational complexity of the tasks performed by the computing units, the waiting time between devices can be reduced.

[0092] According to another embodiment of the present invention, each of the multiple computing units may be configured with a buffer pool, which is configured by the control unit based on the tasks to be processed, the target reasoning model and the memory resources of the computing unit, and is used to cache sub-processing results from other computing units.

[0093] Exemplarily, the estimated memory usage of the target execution result generated after executing the task to be processed can be predicted based on the task to be processed and the target reasoning model. Then, the size of the buffer pool is configured based on the estimated memory usage and the memory resources of the computing unit. For example, the historical processing tasks that match the task type of the task to be processed and the model structure of the target reasoning model can be determined, and then the estimated memory usage of the target execution result generated after executing the task to be processed can be determined based on the memory usage of the historical target execution results obtained from the historical processing tasks.

[0094] Exemplarily, sub-processing results from other computing units are cached in a buffer pool. When the number of execution rounds of the task to be processed is M times, the control unit can control the computing device to perform reduction operations, such as matrix addition or matrix multiplication, when it determines that multiple computing units have sent the Nth sub-processing result.

[0095] According to the embodiment of the present invention, since the buffer pool is built in the computing unit, when executing the next task to be processed, it is not necessary to obtain the task to be executed from the control unit again, which can reduce the communication time and save communication resources.

[0096] According to another embodiment of the present invention, the model reasoning method applied to the computing device includes the following steps: Figure 2In addition to the operations S210 to S230 shown, the operation may also include: using the mth execution result to update the m-1th execution result cached in the buffer pool.

[0097] Exemplarily, the mth execution result may be used to overwrite the m-1th execution result cached in the buffer pool.

[0098] According to an embodiment of the present invention, by updating the execution results in the cache, the situation where the buffer pool memory overflows due to the execution results occupying a large amount of memory is avoided. In addition, outdated execution results may lead to invalid calculations. Real-time updating of the execution results in the cache can save unnecessary computing resources, improve computing efficiency, and save storage resources.

[0099] According to another embodiment of the present invention, the model reasoning method applied to the computing device includes the following steps: Figure 2 In addition to the operations S210 to S230 shown, the following operation may be further included: when it is determined that the target execution result is obtained and the next task to be processed is to be executed, deleting the target execution result cached in the buffer pool.

[0100] According to an embodiment of the present invention, since the memory in the buffer pool is limited, when the next task to be processed is to be executed, the target execution result cached in the buffer pool is deleted in time, so as to prevent the buffer pool from overflowing.

[0101] According to another embodiment of the present invention, the model reasoning method applied to the computing device includes the following steps: Figure 2 In addition to the operations S210~S230 shown, the following operations may also be included: when executing the next task to be processed, it may be determined whether the next task to be processed and the current task to be processed are the same task. If the next task to be processed and the current task to be processed are the same task, the target execution result stored in the buffer pool is used as the target execution result of the next task to be processed.

[0102] Exemplarily, the buffer pool may store the results in a key-value format, where the key is the pending task and the value is the target execution result. When the next pending task is received, the pending task matching the next pending task may be determined from the buffer pool, and the target execution result of the pending task matching the next pending task may be used as the target execution result for executing the next pending task.

[0103] According to the embodiment of the present invention, since the execution results of the tasks to be processed are stored in the buffer pool, the reuse rate of the execution results can be improved.

[0104] Based on the above model reasoning method applied to a computing device, the present invention also provides a computing device. Figure 5 The computing device is described in detail.

[0105] Figure 5 A structural block diagram of a computing device according to an embodiment of the present invention is shown.

[0106] like Figure 5 As shown, the computing device 500 of this embodiment includes a control unit 101, a first computing unit 102_1, a second computing unit 102_2, ..., and an Nth computing unit 102_N.

[0107] The control unit 101 is used to send tasks to be processed to multiple computing units respectively.

[0108] Each of the first computing unit 520102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N is configured with at least two data subsets, and the data subsets are obtained by splitting the weight set of the target inference model required to execute the task to be processed.

[0109] The first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N are used to respectively execute tasks to be processed and obtain multiple sub-processing results.

[0110] The control unit 101 is further configured to obtain a target execution result of the task to be processed according to the multiple sub-processing results.

[0111] Any computing unit among the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N is used to execute the task to be processed in the following manner: using at least two configured data subsets, processing the task to be processed in sequence to obtain at least two sub-processing results, wherein, when any sub-processing result is obtained, any sub-processing result is synchronously sent to other computing units in the multiple computing units that process the task to be processed in parallel.

[0112] According to an embodiment of the present invention, at least two data subsets may include N data subsets, where N is an integer greater than or equal to 2; the first computing unit 102_1, the second computing unit 102_2, ..., and any computing unit among the Nth computing unit 102_N use the configured at least two data subsets to process the tasks to be processed in sequence, and obtaining at least two sub-processing results may include: using the first data subset of the N data subsets to process the tasks to be processed to obtain the first sub-processing result; when n is greater than or equal to 2 and less than or equal to N, using the nth data subset of the N data subsets to process the tasks to be processed to obtain the nth sub-processing result, and sending the n-1th sub-processing result to other computing units.

[0113] According to an embodiment of the present invention, any of the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N is configured with the number of execution rounds of the task to be processed.

[0114] According to an embodiment of the present invention, the control unit 101 is also used to obtain an initial execution result when it is determined that multiple computing units have sent the Nth sub-processing result; when the number of execution rounds of the task to be processed is single, the initial execution result is used as the target execution result; when the number of execution rounds of the task to be processed is M times, when m is greater than 1 and less than M, based on multiple sub-processing results from other computing units in the m-1th execution task, using at least two configured data subsets, the mth task to be processed is processed in sequence to obtain the mth execution result, where M is an integer greater than 1; when m=M, the mth execution result is used as the target execution result.

[0115] According to an embodiment of the present invention, the control unit 101 is further used to determine the model structure of the target reasoning model according to the configuration file of the target reasoning model; and determine the number of rounds for executing the tasks to be processed in parallel based on the model structure and the tasks to be processed.

[0116] According to an embodiment of the present invention, the time required for multiple computing units to process tasks to be processed in the same round is the same.

[0117] According to an embodiment of the present invention, the control unit 101 is also used to split the weight set of the target inference model based on the number of multiple computing units to execute the task to be processed, the performance parameters of each of the multiple computing units, and the attributes of the weight set to obtain multiple data subsets.

[0118] According to an embodiment of the present invention, the weight set includes a weight matrix, and the attributes of the weight set include the dimension of the weight matrix; the control unit 101 splits the weight set based on the number of units of the multiple computing units, the performance parameters of each of the multiple computing units, and the attributes of the weight set to obtain multiple data subsets, including: based on the number of units of the multiple computing units, the performance parameters of each of the multiple computing units, and the dimension of the weight matrix, the weight matrix is ​​initially split to obtain the number of units initial sub-matrices; based on the task type performed by the computing unit, the number of units initial sub-matrices are split again to obtain multiple data subsets.

[0119] According to an embodiment of the present invention, the performance parameter includes computing power. The control unit 101 initially splits the weight matrix based on the number of units of the plurality of computing units, the plurality of respective performance parameters and the dimension of the weight matrix, including: when the dimension of the weight matrix is ​​divisible by the number of units and the computing power of each of the plurality of computing units is the same, the weight matrix is ​​proportionally split into the number of initial sub-matrices of the number of units; when the dimension of the weight matrix is ​​not divisible by the number of units, the weight matrix is ​​split based on the computational complexity of the computing unit to perform the task, so that the computational complexity of the tasks performed by each of the plurality of computing units is consistent.

[0120] According to an embodiment of the present invention, any one of the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N is configured with a buffer pool, which is configured by the control unit based on the tasks to be processed, the target reasoning model and the memory resources of the computing unit, and is used to cache sub-processing results from other computing units.

[0121] According to an embodiment of the present invention, the control unit 101 is further configured to update the (m-1)th execution result cached in the buffer pool using the (m)th execution result.

[0122] According to an embodiment of the present invention, the control unit 101 is further configured to delete the target execution result cached in the buffer pool when it is determined that the target execution result is obtained and the next task to be processed is to be executed.

[0123] According to an embodiment of the present invention, any multiple units of the control unit 101, the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N can be combined into one unit for implementation, or any one of them can be split into multiple units. Alternatively, at least part of the functions of one or more of these units can be combined with at least part of the functions of other units and implemented in one unit. According to an embodiment of the present invention, at least one of the control unit 101, the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the control unit 101, the first computing unit 102_1, the second computing unit 102_2, ..., and the Nth computing unit 102_N may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0124] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0125] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include one or more memories.

[0126] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0127] When the computer program is executed by the processor, the above functions defined in the system / device of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0128] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through a communication part, and / or installed from a removable medium. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0129] In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a processor, the above-mentioned functions defined in the system of an embodiment of the present invention are performed. According to an embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0130] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0133] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A model reasoning method applied to a computing device, characterized in that: The computing device comprises: a control unit and a plurality of computing units for executing tasks to be processed in parallel, and the method comprises: The control unit sends the tasks to be processed to the multiple computing units respectively, and any computing unit among the multiple computing units is configured with at least two data subsets, and the data subsets are obtained by splitting the weight set of the target inference model required to execute the tasks to be processed; The multiple computing units respectively execute the tasks to be processed to obtain multiple sub-processing results; The control unit obtains the target execution result of the task to be processed according to the multiple sub-processing results; wherein any one of the multiple computing units executes the task to be processed in the following manner: Using the at least two configured data subsets, the tasks to be processed are processed in sequence to obtain at least two sub-processing results, wherein when any sub-processing result is obtained, the any sub-processing result is synchronously sent to other computing units in the multiple computing units that process the tasks to be processed in parallel.

2. The method according to claim 1, characterized in that The at least two data subsets include N data subsets, where N is an integer greater than or equal to 2; the at least two data subsets configured are used to sequentially process the tasks to be processed, and at least two sub-processing results are obtained, including: Using a first data subset of the N data subsets to process the task to be processed, to obtain a first sub-processing result; When n is greater than or equal to 2 and less than or equal to N, the task to be processed is processed using the nth data subset of the N data subsets to obtain the nth sub-processing result, and the n-1th sub-processing result is sent to the other computing units.

3. The method according to claim 2, characterized in that Obtaining the number of execution rounds of the task to be processed configured by any computing unit among the multiple computing units; The method further comprises: In the case of determining that the Nth sub-processing result sent by the other computing unit has been received, obtaining an initial execution result; In the case where the number of execution rounds of the task to be processed is single, taking the initial execution result as the target execution result; When the number of execution rounds of the task to be processed is M times, In the case where m is greater than 1 and less than M, based on the multiple sub-processing results from the other computing units in the m-1th execution of the task to be processed, the at least two configured data subsets are used to sequentially process the mth task to be processed to obtain the mth execution result, the mth task to be processed is the m-1th execution result, and M is an integer greater than 1; When m=M, the mth execution result is used as the target execution result.

4. The method according to claim 3, characterized in that The method further comprises: The control unit determines the model structure of the target reasoning model according to the configuration file of the target reasoning model; Based on the model structure and the tasks to be processed, the number of rounds for executing the tasks to be processed in parallel is determined.

5. The method according to claim 3, characterized in that: The time required for the multiple computing units to process the tasks to be processed in the same round is the same.

6. The method according to claim 1, characterized in that The method further comprises: The control unit splits the weight set of the target inference model based on the number of the multiple computing units to execute the tasks to be processed, the performance parameters of the multiple computing units respectively, and the attributes of the weight set to obtain the multiple data subsets.

7. The method according to claim 6, characterized in that The weight set includes a weight matrix, and the attributes of the weight set include the dimension of the weight matrix; The control unit splits the weight set of the target reasoning model based on the number of the plurality of computing units to execute the task to be processed, the performance parameters of the plurality of computing units, and the attributes of the weight set to obtain the plurality of data subsets, including: Based on the number of the plurality of computing units, the performance parameters of the plurality of computing units, and the dimension of the weight matrix, initially splitting the weight matrix to obtain initial sub-matrices of the number of units; Based on the task type executed by the computing unit, the initial sub-matrices of the number of units are split again to obtain the multiple data subsets.

8. The method according to claim 7, characterized in that The performance parameters include computing power; The initial splitting of the weight matrix based on the number of the plurality of computing units, the performance parameters of the plurality of computing units, and the dimension of the weight matrix includes: When the dimension of the weight matrix is ​​divisible by the number of units and the computing capabilities of the plurality of computing units are the same, the weight matrix is ​​divided into initial sub-matrices of the number of units in equal proportion; When the dimension of the weight matrix cannot be divided by the number of units, the weight matrix is ​​split based on the computational complexity of the computing unit executing the task to be processed, so that the computational complexity of each of the multiple computing units executing the task to be processed is consistent.

9. The method according to claim 3, characterized in that: Each of the multiple computing units is configured with a buffer pool, which is configured by the control unit based on the tasks to be processed, the target reasoning model and the memory resources of the computing unit, and is used to cache the sub-processing results from the other computing units.

10. The method according to claim 9, characterized in that The method further includes: using the mth execution result to update the (m-1)th execution result cached in the buffer pool.

11. The method according to claim 9, characterized in that The method further comprises: When it is determined that the target execution result is obtained and the next task to be processed is to be executed, the target execution result cached in the buffer pool is deleted.

12. A computing device comprising: A control unit, used for sending the tasks to be processed to the multiple computing units respectively; The multiple computing units, any computing unit among the multiple computing units is configured with at least two data subsets, and the data subsets are obtained by splitting the weight set of the target reasoning model required to perform the task to be processed; The multiple computing units are used to respectively execute the tasks to be processed to obtain multiple sub-processing results; The control unit is further used to obtain the target execution result of the task to be processed according to the multiple sub-processing results; Wherein, any computing unit among the multiple computing units is used to execute the task to be processed in the following manner: Using the at least two configured data subsets, the tasks to be processed are processed in sequence to obtain at least two sub-processing results, wherein when any sub-processing result is obtained, the any sub-processing result is synchronously sent to other computing units in the multiple computing units that process the tasks to be processed in parallel.

13. The computing device according to claim 12, characterized in that The at least two data subsets include N data subsets, where N is an integer greater than or equal to 2; the computing unit uses the configured at least two data subsets to sequentially process the tasks to be processed, and obtains at least two sub-processing results including: Using a first data subset of the N data subsets to process the task to be processed, to obtain a first sub-processing result; When n is greater than or equal to 2 and less than or equal to N, the task to be processed is processed using the nth data subset of the N data subsets to obtain the nth sub-processing result, and the n-1th sub-processing result is sent to the other computing units.

14. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 11.

Images