Text generation system and method

Abstract

The present disclosure relates to the field of data processing, and provides a text generation system and method. In this system, in response to receiving a first task and the address of a first language understanding module, a first result generation module transmits the first task to the first language understanding module, and generates the second token to the Nth token based on the first token and M tensors returned by the first language understanding module. The first token to the Nth token are used to generate a text corresponding to the first task. The text generation system according to the embodiments of the present disclosure uses the first language understanding module to complete the functions of the large language model in the language understanding stage, and uses the first result generation module to complete the functions of the large language model in the result generation stage. The two modules are independent, and the computing power of the first result generation module is less than that of the first language understanding module. On the premise that the time consumption in the language understanding stage is less, the waste of computing power in the result generation stage is also reduced, improving the text generation performance.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular, to a text generation system and method. Background Art

[0002] The Transformer based Autoregressive Language Model, also known as the large language model, is one of the models commonly used in the field of text generation. The large language model is usually deployed on a dedicated computing device (such as a Graphics Processing Unit (GPU)). By receiving a piece of text input by the user, and then inside the large language model, the next possible word is predicted based on the text content and returned to the user until several words are predicted and returned and then it stops.

[0003] The prediction of words inside the large language model can be divided into two different stages: the language understanding stage and the result generation stage. In the language understanding stage, the large language model "understands" the meaning of the text input by the user through calculation, manifested as calculating the state tensor (KV-Cache) inside the large language model and the first predicted word. The characteristic of this stage is that the calculation is very intensive. In the result generation stage, the large language model uses the state tensor calculated in the language understanding stage as a parameter, and the previously predicted word as an input to predict the next word, thus predicting several words. The characteristic of this stage is that the calculation is relatively sparse.

[0004] In the related art, the language understanding stage and the result generation stage of the large language model are implemented based on the same computing device, that is, the computing power available for the two stages is the same. If a relatively large computing power matching the language understanding stage is provided to the two stages, it will result in waste of computing power in the result generation stage; if a relatively small computing power matching the result generation stage is provided to the two stages, it will result in an overly long time consumption in the language understanding stage. Therefore, how to reduce the waste of computing power in the result generation stage while consuming less time in the language understanding stage and improve the text generation performance has become a research focus in this field. Summary of the Invention

[0005] In view of this, the present disclosure proposes a text generation system and method. The text generation system in the embodiments of the present disclosure uses a first language understanding module to complete the functions of the large language model in the language understanding stage, and uses a first result generation module to complete the functions of the large language model in the result generation stage. The two modules are independent, and the computing power of the first result generation module is less than that of the first language understanding module. While consuming less time in the language understanding stage, the waste of computing power in the result generation stage is also reduced, improving the text generation performance.

[0006] According to one aspect of the present disclosure, a text generation system is provided. The system includes a first result generation module and a first language understanding module. The computing power of the first result generation module is less than that of the first language understanding module. The first result generation module is configured to, in response to receiving a first task and the address of the first language understanding module, transmit the first task to the first language understanding module according to the address of the first language understanding module. The first language understanding module is configured to parse the first task to generate a first token and M tensors, and return them to the first result generation module. The first result generation module is further configured to, in response to receiving the first token and M tensors returned by the first language understanding module, generate a second token to an Nth token according to the first token and M tensors, where M and N are positive integers, N is greater than 2, and the first token to the Nth token are used to generate text corresponding to the first task.

[0007] In a possible implementation, the system further includes a first interface scheduling module. The first interface scheduling module is configured to: receive a first request from an interface visible to the user, generate the first task according to the first request and store it in a task queue to be assigned; assign tasks in the task queue to be assigned, where, in response to the first task being assigned to the first result generation module and the first language understanding module, transmit the first task and the address of the first language understanding module to the first result generation module.

[0008] In a possible implementation, the first result generation module is further configured to output the first token to the Nth token to the first interface scheduling module. The first interface scheduling module is further configured to generate text corresponding to the first task according to the received tokens and output it to an interface visible to the user.

[0009] In a possible implementation, the first result generation module is one of multiple result generation modules included in the text generation system, and the first language understanding module is one of multiple language understanding modules included in the text generation system. The computing power of the result generation module is less than that of the language understanding module. The first interface scheduling module is specifically configured to: in response to A result generation modules and B language understanding modules being idle, select A result generation modules and B language understanding modules to process Y tasks in the task queue to be assigned, where Y is less than or equal to the sum of the numbers of tasks that A result generation modules can process simultaneously and less than or equal to the sum of the numbers of tasks that B language understanding modules can process simultaneously, and A, B, and Y are positive integers.

[0010] In a possible implementation, each result generation module and each language understanding module include multiple computing devices, and the models of different computing devices may be the same or different; the number of tasks that any two result generation modules can process simultaneously may be the same or different, and the difference in the time required for any two language understanding modules to process the same number of tasks is less than a first threshold.

[0011] In a possible implementation, the first result generation module is further configured to, in response to receiving the address of the first language understanding module, allocate a corresponding first storage space for the first language understanding module from its own memory or video memory, and output the address of the first result generation module and the address of the first storage space to the first language understanding module; specifically, the first language understanding module is configured to transmit the M tensors to the first storage space according to the address of the first result generation module and the address of the first storage space.

[0012] In a possible implementation, after generating a token, the first result generation module transmits it to the first interface scheduling module in real time. Specifically, the first interface scheduling module is configured to, in response to each received token, generate a corresponding text once and output it to an interface visible to the user.

[0013] In a possible implementation, the first result generation module is specifically configured to generate the (i + 1)-th token according to the i-th token and M tensors, where 1 ≤ i < N and i is an integer.

[0014] In a possible implementation, the first interface scheduling module is specifically configured to perform tokenization processing on the first request to generate the first task, where the first task includes W tokens, and the first to N-th tokens are associated with the W tokens, and W is an integer.

[0015] According to another aspect of the present disclosure, a text generation method is provided. The method is applied to a text generation system, which includes a first result generation module and a first language understanding module, and the computing power of the first result generation module is less than that of the first language understanding module. The method includes: in response to receiving a first task and the address of the first language understanding module, the first result generation module transmits the first task to the first language understanding module according to the address of the first language understanding module; the first language understanding module parses the first task to generate the first token and M tensors, and returns them to the first result generation module; in response to receiving the first token and M tensors returned by the first language understanding module, the first result generation module generates the second to N-th tokens according to the first token and M tensors, where M and N are positive integers, N is greater than 2, and the first to N-th tokens are used to generate a text corresponding to the first task.

[0016] In a possible implementation, the system further includes a first interface scheduling module, and the method further includes: The first interface scheduling module receives a first request from an interface visible to the user, generates the first task according to the first request, and stores it in the task queue to be assigned; The first interface scheduling module assigns tasks in the task queue to be assigned. Among them, in response to the first task being assigned to the first result generation module and the first language understanding module, the first task and the address of the first language understanding module are transmitted to the first result generation module.

[0017] In a possible implementation, the method further includes: The first result generation module outputs the first token to the Nth token to the first interface scheduling module; The first interface scheduling module generates text corresponding to the first task according to the received tokens and outputs it to the interface visible to the user.

[0018] In a possible implementation, the first result generation module is one of multiple result generation modules included in the text generation system, and the first language understanding module is one of multiple language understanding modules included in the text generation system. The computing power of the result generation module is less than that of the language understanding module. The first interface scheduling module assigns tasks in the task queue to be assigned, including: When A result generation modules and B language understanding modules are idle, the first interface scheduling module selects A result generation modules and B language understanding modules to process Y tasks in the task queue to be assigned, where Y is less than or equal to the sum of the number of tasks that A result generation modules can process simultaneously and less than or equal to the sum of the number of tasks that B language understanding modules can process simultaneously. A, B, and Y are positive integers.

[0019] In a possible implementation, each result generation module and each language understanding module each include multiple computing devices, and the models of different computing devices are the same or different; The number of tasks that any two result generation modules can process simultaneously may be the same or different, and the difference in the time required for any two language understanding modules to process the same number of tasks is less than a first threshold.

[0020] In a possible implementation, the method further includes: in response to receiving the address of the first language understanding module, the first result generation module allocates a corresponding first storage space for the first language understanding module from its own memory or video memory, and outputs the address of the first result generation module and the address of the first storage space to the first language understanding module; the first language understanding module parses the first task to generate the first token and M tensors, and returns them to the first result generation module, including: the first language understanding module transfers the M tensors to the first storage space according to the address of the first result generation module and the address of the first storage space.

[0021] In a possible implementation, after generating a token, the first result generation module transmits it to the first interface scheduling module in real time. The first interface scheduling module generates text corresponding to the first task based on the received token and outputs it to an interface visible to the user, including: in response to each received token, the first interface scheduling module generates corresponding text once and outputs it to an interface visible to the user.

[0022] In a possible implementation, generating the second token to the Nth token according to the first token and M tensors includes: generating the (i + 1)th token according to the ith token and M tensors, where 1 ≤ i < N and i is an integer.

[0023] In a possible implementation, generating a first task according to the first request and storing it in the task queue to be allocated includes: performing tagging processing on the first request to generate the first task, where the first task includes W tokens, and the first token to the Nth token are associated with the W tokens, and W is an integer.

[0024] According to another aspect of the present disclosure, an electronic device is provided, including the above-mentioned text generation system.

[0025] A text generation system according to an embodiment of the present disclosure is provided with a first result generation module and a first language understanding module. The computing power of the first result generation module is less than that of the first language understanding module. Therefore, the first result generation module and the first language understanding module are independent modules. In response to receiving a first task and the address of the first language understanding module, the first result generation module transmits the first task to the first language understanding module according to the address of the first language understanding module, enabling the first language understanding module to determine that the text generation work corresponding to the first task needs to be completed by itself and the first result generation module, and parsing the first task to generate a first token and M tensors and returning them to the first result generation module to complete the work in the language understanding stage of the first task; when the first result generation module receives the first token and M tensors returned by the first language understanding module, it can generate the second token to the Nth token according to the first token and M tensors to complete the work in the result generation stage corresponding to the first task. The first token to the Nth token can be used to generate the text corresponding to the first task. The text generation system according to the embodiment of the present disclosure uses the first language understanding module to complete the functions of the large language model in the language understanding stage, and uses the first result generation module to complete the functions of the large language model in the result generation stage. The two modules are independent, and the computing power of the first result generation module is less than that of the first language understanding module. Therefore, the computing power of the first language understanding module can be set to be higher to adapt to the greater computing power requirements in the language understanding stage, so that the time-consuming in the language understanding stage is less; the computing power of the first result generation module can be set to be lower to adapt to the smaller computing power requirements in the result generation stage, so that the waste of computing power in the result generation stage is reduced or even can achieve the effect of no waste; in summary, the text generation system according to the embodiment of the present disclosure reduces the waste of computing power in the result generation stage on the premise of less time-consuming in the language understanding stage, and improves the text generation performance.

[0026] Other features and aspects of the present disclosure will become apparent from the following detailed description of the exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings included in the specification and constituting a part of the specification, together with the specification, illustrate the exemplary embodiments, features, and aspects of the present disclosure and are used to explain the principles of the present disclosure.

[0028] Figure 1 A schematic diagram showing the framework of a large language model of related art one.

[0029] Figure 2 A schematic diagram showing the framework of a large language model of related art two.

[0030] Figure 3 An exemplary application scenario of a text generation system according to an embodiment of the present disclosure is shown.

[0031] Figure 4a A schematic diagram showing the structure of a text generation system according to an embodiment of the present disclosure.

[0032] Figure 4b A schematic diagram showing the structure of a text generation system according to an embodiment of the present disclosure.

[0033] Figure 5 A schematic diagram showing the structure of a text generation system according to an embodiment of the present disclosure.

[0034] Figure 6 A schematic diagram showing the structure of a text generation system according to an embodiment of the present disclosure.

[0035] Figure 7 A schematic diagram showing the flow of a text generation method according to an embodiment of the present disclosure. Detailed implementation manners

[0036] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. Identical reference numerals in the drawings denote functionally identical or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0037] The term "exemplary" used herein means "serving as an example, embodiment, or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

[0038] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0039] The frameworks of two existing large language models in the related art are introduced below.

[0040] Figure 1 A schematic diagram showing the framework of the large language model of Related Art One.

[0041] Such as Figure 1As shown, Related Technology 1 is an implementation solution for large language models that is currently the most widely used. This solution directly conducts the language understanding stage and the result generation stage within the same framework. The computation in the language understanding stage includes M layers (M is a positive integer). The input for the computation in the first layer is an array of tokens obtained by processing the request issued by the user through tokenizing and other operations. The output of the previous layer serves as the input for the next layer, and the output of the last layer can be the predicted first token. Tokenizing generally refers to splitting the text into multiple tokens. The splitting can use single characters and / or words as the smallest granularity units. Each layer also calculates an internal state tensor (KV-Cache), and the M internal state tensors calculated in the M layers are used as parameters in the result generation stage. After converting the first token into text, it is returned to the user. From the user's perspective, the automatically generated first text can be received. This stage has a relatively high demand for computing power and takes a relatively long time.

[0042] The language understanding stage supports simultaneously responding to multiple requests issued by multiple users. For example, when supporting simultaneously responding to P requests, the input for the first layer in the language understanding stage can be a P×Q array of tokens (P is the number of requests responded to simultaneously, and Q is the number of tokens obtained after tokenizing a single request).

[0043] The computation in the result generation stage also includes multiple layers, and the number of layers is the same as that in the language understanding stage. The input for the first layer is the previously predicted token (such as the jth token, where j is a positive integer). The output of the previous layer serves as the input for the next layer. Each layer uses the internal state tensor calculated in the same layer of the language understanding stage as a parameter. The output of the last layer can be the token predicted this time (such as the (j + 1)th token). Therefore, for each request issued by a user, the result generation stage can continuously predict multiple tokens until a stop condition is reached (such as the total number of tokens predicted for each request issued by a user reaches W). Each predicted token is converted into text and returned to the user. From the user's perspective, multiple generated texts can be continuously received. This stage has a relatively low demand for computing power.

[0044] The result generation stage supports simultaneously responding to multiple tokens. That is, the output of the language understanding stage (a column of P×1 tokens) can directly serve as the input for the first layer of the result generation stage. The stop condition for the result generation stage is that when the total number of tokens predicted for each request issued by a user reaches W, the result generation stage can include W - 1 rounds of computation. Each round of computation includes multiple layers, where the input for the first layer is a column of P×1 tokens obtained from the previous prediction, and the output of the last layer is a column of P×1 tokens predicted this time.

[0045] One disadvantage of Related Technology 1 is that since the same framework is used in both stages, the available computing power in the two stages is substantially the same. As the two stages of the large language model have different requirements for computing power, a large amount of computing power is needed in the language understanding stage, while the computing power required in the result generation stage is very low. More computing devices need to be deployed to provide a large amount of computing power matching that of the language understanding stage. The computing power of the large language model is provided by computing devices. In the result generation stage, most of the computing devices are actually not used, and most of the provided computing power is wasted. The result generation stage usually accounts for more than 80% of the total time in the entire text generation process. Therefore, the waste of computing power is very serious, and the utilization efficiency of computing devices is low.

[0046] Another disadvantage of Related Technology 1 is that the computing devices deployed for the large language model must be homogeneous computing devices of the same model. This results in an inflexible deployment method and poor compatibility for the large language model.

[0047] Figure 2 A schematic diagram showing the framework of the large language model of Related Technology 2.

[0048] As Figure 2 shown, Related Technology 2 proposes to complete the calculations in the language understanding stage in the same way as in the result generation stage. The language understanding stage supports responding to multiple requests sent by multiple users simultaneously. For example, when supporting simultaneous response to P requests, after tokenizing and other processing of the P requests, a token array of P×Q is obtained (Q is the number of tokens obtained after tokenizing a single request). At this time, the language understanding stage includes Q rounds of calculations. The input for each round of calculation can be a column of tokens in the token array of P×Q. Each round of calculation includes M layers. The input of the first layer is the input of this round of calculation. Each layer of calculation obtains a corresponding internal state tensor. All the internal state tensors obtained from this round of calculation are used as the output of this round of calculation and used as parameters in the next round of calculation. The first round of calculation does not use the internal state tensor as a parameter. All the internal state tensors obtained from the Qth round of calculation are used as the parameters for the result generation stage. The Mth layer of the Qth round of calculation also outputs the predicted first token.

[0049] That is to say, a large calculation in the language understanding stage of Related Technology 1 is split into Q small calculations (Q usually ranges from several hundred to thousands) in Related Technology 2. The result generation stage of Related Technology 2 can be the same as that of Related Technology 1 and will not be elaborated here.

[0050] The performance of the large language model is closely related to two metrics: first-word latency and generation speed. Among them, the first-word latency refers to the time from when the user sends a request to when the first text is received. The generation speed refers to the speed at which the user subsequently receives multiple texts continuously.

[0051] The disadvantage of Related Art Two is that the time consumption of the Q-th smallest calculation is much greater than that of one large calculation. Therefore, the time consumption in the language understanding stage is too long, resulting in too large a first-character delay. This solution is only applicable to application scenarios that do not care about the first-character delay and need to respond to a large number of requests simultaneously, which is usually unacceptable in most application scenarios.

[0052] In summary, the solution of the related art cannot reduce the waste of computing power in the result generation stage on the premise of less time consumption in the language understanding stage. Therefore, there is still room for improvement in the text generation performance of the large language model.

[0053] In view of this, the present disclosure proposes a text generation system and method. The text generation system according to the embodiments of the present disclosure uses a first language understanding module to complete the functions of the large language model in the language understanding stage, and uses a first result generation module to complete the functions of the large language model in the result generation stage. The two modules are independent, and the computing power of the first result generation module is less than that of the first language understanding module. On the premise of less time consumption in the language understanding stage, the waste of computing power in the result generation stage is also reduced, improving the text generation performance.

[0054] Figure 3 An exemplary application scenario of the text generation system according to the embodiments of the present disclosure is shown.

[0055] As Figure 3 shown, the application scenario may include a user, a first device, and a second device. The first device may be a server in the cloud, and the second device may be a terminal device facing the user. The text generation system according to the embodiments of the present disclosure may be provided on the first device.

[0056] When the user uses the text generation system to automatically generate text, a request including text data may be initiated through the second device. After the request is transmitted to the first device, it is received by the text generation system. The text generation system may generate a first task according to the request. The first task includes a plurality of tokens associated with the text data in the request. The text generation system may successively generate several tokens according to the first task. At this time, the generated tokens are associated with the tokens included in the first task. The text generation system may further generate corresponding text according to the generated tokens and return it to the user. From the user's perspective, after initiating a request including text data, new text generated by the text generation system and related to the text data included in the request is continuously received.

[0057] Those skilled in the art should understand that generating the first task according to the request may also be completed by the second device, and the second device may directly send the first task to the first device. The present disclosure does not limit whether the first device receives a first request or a first task.

[0058] Those skilled in the art should understand that the text generation system can also be set on the second device, and the embodiments of the present disclosure do not limit whether the text generation system is specifically set on the server or the terminal device.

[0059] Figure 4a A schematic diagram showing the structure of a text generation system according to an embodiment of the present disclosure.

[0060] As Figure 4a shown, in a possible implementation manner, the present disclosure proposes a text generation system, the system includes a first result generation module and a first language understanding module, and the computing power of the first result generation module is less than that of the first language understanding module.

[0061] The first result generation module is configured to, in response to receiving a first task and the address of the first language understanding module, transmit the first task to the first language understanding module according to the address of the first language understanding module.

[0062] The first language understanding module is configured to parse the first task to generate a first token and M tensors, and return them to the first result generation module.

[0063] The first result generation module is configured to, in response to receiving the first token and the M tensors returned by the first language understanding module, generate a second token to an Nth token according to the first token and the M tensors, where M and N are positive integers, N is greater than 2, and the first token to the Nth token are used to generate text corresponding to the first task.

[0064] For example, the text generation system may include a first result generation module and a first language understanding module. Among them, the first language understanding module can be used to complete the work in the language understanding stage of the large language model, and the first result generation module can be used to complete the work in the result generation stage of the large language model. The first result generation module and the first language understanding module can be independent of each other, and each module includes a certain number of computing devices to provide computing power for the module. Exemplarily, the computing device can be a graphics processor, a central processing unit or other types of computing devices, and the embodiments of the present disclosure do not limit the types of computing devices used by each module.

[0065] The first task can be generated according to a first request from a user and may include at least one token, so that the text generation system can generate more tokens according to the first task. Generating the first task can be implemented based on the prior art, and its exemplary implementation manner is given later.

[0066] The first task may be first received by the first result generation module used to complete the result generation phase. When the first result generation module receives the first task and the address of the first language understanding module, it may determine that it needs to complete the result generation phase of the first task, and the language understanding phase of the first task is completed by the first language understanding module. Since the result generation phase needs to wait until the language understanding phase ends before it can begin, the first result generation module may first notify the first language understanding module to complete the language understanding phase of the first task. Exemplarily, the first result generation module may transmit the first task to the first language understanding module according to the address of the first language understanding module, thereby completing the above notification process.

[0067] The first language understanding module can parse the first task to generate the first word and M tensors corresponding to the first task. The first word corresponding to the first task is also the first word predicted by the text generation system for the first request. The first language understanding module can generate the first word and M tensors according to the calculation method of the language understanding stage in the related technology 1. For examples, see the above and Figure 1 At this time, the computing power required by the first language understanding module is relatively large, and the first word delay is relatively low.

[0068] Those skilled in the art should understand that the first language understanding module can also generate the first word unit and M vectors according to the calculation method of the language understanding stage of the related technology 2, and its examples can be seen above and Figure 2 Although the first word delay is relatively high at this time, compared with the solution of the related technology 2, the flexibility of the setting method of the computing power of the first language understanding module and the first result generating module can be increased.

[0069] After obtaining the first word unit and the M tensors, the language understanding phase of the first task ends. The first language understanding module can return the first word unit and the M tensors to the first result generation module.

[0070] When the first result generation module receives the first word and M tensors returned by the first language understanding module, it starts to complete the result generation phase of the first task. The result generation phase may include multiple rounds of calculations, and the end condition of the multiple rounds of calculations may be that the text generation system predicts N words for the first request (N is a positive integer, N is greater than 2). Since the language understanding phase has calculated the first word, the result generation phase can perform N-1 rounds of calculations, and each round of calculation obtains 1 word. When the second word to the Nth word are calculated, the result generation phase of the first task ends. For an exemplary method in which the first result generation module generates the second word to the Nth word based on the first word and M tensors, please refer to the relevant description below. The values of M and N can be set according to the application scenario requirements, and the present disclosure does not limit this.

[0071] The subsequent text generation system can generate the text corresponding to the first task based on the first to the Nth tokens of the first task and return it to the user. In this case, the predicted N tokens can be displayed to the user in text form.

[0072] According to the text generation system of the embodiments of the present disclosure, a first result generation module and a first language understanding module are provided. The computing power of the first result generation module is less than that of the first language understanding module. Therefore, the first result generation module and the first language understanding module are independent modules. The first result generation module, in response to receiving the first task and the address of the first language understanding module, transmits the first task to the first language understanding module according to the address of the first language understanding module, so that the first language understanding module can determine that the text generation work corresponding to the first task needs to be completed by itself and the first result generation module, and parses the first task to generate the first token and M tensors and returns them to the first result generation module to complete the work in the language understanding stage of the first task; when the first result generation module receives the first token and M tensors returned by the first language understanding module, it can generate the second to the Nth tokens according to the first token and M tensors to complete the work in the result generation stage corresponding to the first task. The first to the Nth tokens can be used to generate the text corresponding to the first task. The text generation system of the embodiments of the present disclosure uses the first language understanding module to complete the functions of the large language model in the language understanding stage, and uses the first result generation module to complete the functions of the large language model in the result generation stage. The two modules are independent, and the computing power of the first result generation module is less than that of the first language understanding module. Therefore, the computing power of the first language understanding module can be set to be relatively high to adapt to the greater computing power requirements in the language understanding stage, so that the time-consuming in the language understanding stage is less; the computing power of the first result generation module can be set to be relatively low to adapt to the smaller computing power requirements in the result generation stage, so that the waste of computing power in the result generation stage is reduced or even can achieve the effect of no waste; in summary, the text generation system of the embodiments of the present disclosure reduces the waste of computing power in the result generation stage on the premise of less time-consuming in the language understanding stage, and improves the text generation performance.

[0073] Figure 4b A schematic diagram showing the structure of the text generation system according to the embodiments of the present disclosure.

[0074] In a possible implementation, the system further includes a first interface scheduling module, and the first interface scheduling module is used for:

[0075] Receive a first request from an interface visible to the user, generate a first task according to the first request and store it in the task queue to be assigned;

[0076] Tasks in the task queue to be assigned are assigned. Among them, in response to a first task being assigned to a first result generation module and a first language understanding module, the first task and the address of the first language understanding module are transmitted to the first result generation module.

[0077] For example, as Figure 4b shown, the text generation system may further include a first interface scheduling module. The first interface scheduling module can communicate with an interface visible to the user and receive a first request from this interface. The interface visible to the user can be Figure 3 the interface on the second device mentioned above. The first request may include a piece of text. The first interface scheduling module can generate a first task according to the first request and store it in the task queue to be assigned. The task queue to be assigned can be located on the first interface scheduling module. The first interface scheduling module generating the first task according to the first request can be implemented based on the prior art, and its exemplary manner can be referred to the relevant descriptions later.

[0078] There may be multiple tasks stored in the task queue to be assigned. These tasks can be unassigned tasks generated according to requests from users. The first interface scheduling module can assign the tasks in the task queue to be assigned once at regular intervals (such as 0.2 s). The embodiments of the present disclosure do not limit the interval time between any two assignments. Among them, when the first task is assigned to the first result generation module and the first language understanding module, it means that the language understanding stage of the first task is completed by the first language understanding module, and the result generation stage of the first task is completed by the first result generation module.

[0079] In this case, the first interface scheduling module can transmit the first task and the address of the first language understanding module to the first result generation module, so that the first result generation module can determine that it needs to complete the result generation stage of the first task according to the received information, and the language understanding stage of the first task is completed by the first language understanding module.

[0080] The following introduces an exemplary method for the embodiments of the present disclosure to generate the first task according to the first request.

[0081] In a possible implementation manner, the first interface scheduling module is specifically used for:

[0082] Perform tagging processing on the first request to generate a first task. The first task includes W word tokens, and the first to Nth word tokens are associated with the W word tokens. W is an integer.

[0083] For example, the first interface scheduling module may first perform tokenization on the received first request, process the text data included in the first request into W tokens, and then generate a first task including W tokens. Optionally, the W tokens may also be processed, such as by clipping and searching, before generating the first task. Tokenization, clipping, and searching can all be implemented based on existing technologies and will not be elaborated here.

[0084] Those skilled in the art should understand that, based on the tokenization process, there can be more ways to process the first request to generate the first task, or other processing methods with the same function as tokenization can be selected to replace tokenization, as long as the first task can include W tokens and the W tokens correspond to the text data included in the first request. The embodiments of the present disclosure do not limit how the first interface scheduling module generates the first task.

[0085] In a possible implementation, the first result generation module is one of the multiple result generation modules included in the text generation system, and the first language understanding module is one of the multiple language understanding modules included in the text generation system. The computing power of the result generation module is less than that of the language understanding module.

[0086] Specifically, the first interface scheduling module is configured to

[0087] In response to A result generation modules and B language understanding modules being idle, the first interface scheduling module selects A result generation modules and B language understanding modules to process Y tasks in the task queue to be assigned, where Y is less than or equal to the sum of the number of tasks that A result generation modules can process simultaneously and less than or equal to the sum of the number of tasks that B language understanding modules can process simultaneously. A, B, and Y are positive integers.

[0088] For example, the text generation system may include multiple result generation modules and multiple language understanding modules. Each module includes a certain number of computing devices that provide computing power for the module. The computing power of the result generation module can be less than that of the language understanding module. The first result generation module can be one of the multiple result generation modules, and the first language understanding module can be one of the multiple language understanding modules.

[0089] The first interface scheduling module can monitor the load status of each result generation module and language understanding module. When the first interface scheduling module distributes the tasks in the task queue to be assigned, it can preferentially assign the tasks to the result generation module and language understanding module with the lightest load.

[0090] The assigned task can be the task with the highest priority in the task queue to be assigned. When assigning tasks, it is necessary to further consider the number of tasks that the result generation module and the language understanding module can process simultaneously, so that after the assignment, the number of tasks that each result generation module and each language understanding module need to process simultaneously does not exceed the number of tasks that they can process simultaneously.

[0091] In one example, assume that the first interface scheduling module determines that A result generation modules and B language understanding modules are idle, and there are Y tasks in the task queue to be assigned. If Y is less than or equal to the sum of the number of tasks that A result generation modules can process simultaneously, and less than or equal to the sum of the number of tasks that B language understanding modules can process simultaneously, then the first interface scheduling module can select A result generation modules and B language understanding modules to process Y tasks in the task queue to be assigned, that is, assign Y tasks to A result generation modules and B language understanding modules. Among them, A result generation modules are responsible for the result generation stage of Y tasks, and B language understanding modules are responsible for the language understanding stage of Y tasks. A, B, and Y are positive integers and can be determined according to the requirements of the application scenario. The specific values of A, B, and Y in the embodiments of the present disclosure are not limited.

[0092] Figure 5 A schematic diagram showing the structure of a text generation system according to an embodiment of the present disclosure.

[0093] As Figure 5 shown, the first interface scheduling module determines that 1 (A = 1) result generation module A1 and 2 (B = 2) language understanding modules B1 and B2 are idle, and there are 4 (Y = 4) tasks (task 1 to task 4) in the task queue to be assigned. If the number of tasks that result generation module A1 can process simultaneously is greater than or equal to 4, and the number of tasks that 2 language understanding modules B1 and B2 can process simultaneously is also greater than or equal to 4, then the first interface scheduling module can select result generation module A1 and language understanding modules B1 and B2 to process 4 tasks in the task queue to be assigned.

[0094] Furthermore, it is possible to assign which task or tasks' result generation stage each result generation module is specifically responsible for, and which task or tasks' language understanding stage each language understanding module is responsible for. As long as the number of tasks that a module needs to process simultaneously after the task is assigned to the module does not exceed the maximum number of tasks that the module can process simultaneously, the embodiments of the present disclosure do not limit the specific assignment relationship between tasks and result generation modules and language understanding modules. In Figure 5 the example, language understanding module B1 is used to process task 1 and task 2, and language understanding module B2 is used to process task 3 and task 4.

[0095] In this case, the 4 tasks simultaneously input to the result generation module A1 can be a 4×W token array. The 2 tasks simultaneously input to the language understanding module B1 and the 2 tasks simultaneously input to the language understanding module B2 can be 2×W token arrays respectively.

[0096] Furthermore, the text generation system may also include multiple interface scheduling modules. Each interface scheduling module corresponds to a certain number (such as a second threshold) of result generation modules and a certain number (such as a third threshold) of language understanding modules, and detects the load status of its corresponding result generation modules and language understanding modules. The second threshold and the third threshold can be set according to the application scenario requirements, and the specific values of the second threshold and the third threshold are not limited in the embodiments of the present disclosure.

[0097] The result generation modules corresponding to different interface scheduling modules are different, and the language understanding modules corresponding thereto are also different. In the embodiments of the present disclosure, the first interface scheduling module may be one of the multiple interface scheduling modules, and corresponds to the first result generation module and the first language understanding module.

[0098] Those skilled in the art should understand that there can be more ways to allocate the tasks in the task queue to be allocated. For example, tasks can be allocated one by one. When allocating the current task, judge the load conditions of multiple result generation modules and multiple language understanding modules after the previous task allocation is completed, and select the result generation module and language understanding module with the lightest load as the result generation module and language understanding module for the current task allocation, etc. As long as the task allocation satisfies the condition of making the load of each module in the text generation system more uniform, the specific allocation method of the tasks is not limited in the embodiments of the present disclosure.

[0099] In this way, the load of each module in the text generation system becomes more uniform.

[0100] For the sake of clear description, in the following, the first interface scheduling module, the first language understanding module, and the first result generation module are taken as examples. However, those skilled in the art should understand that the functions of each language understanding module in the embodiments of the present disclosure can be the same, the functions of each result generation module can be the same, and the functions of each interface scheduling module can be the same.

[0101] Figure 6 A schematic diagram showing the structure of a text generation system according to an embodiment of the present disclosure.

[0102] Such as Figure 6As shown, the text generation system may further include multiple isolation components for separating different modules. The first request may first be transmitted from the user-visible interface to the isolation component 1, and then from the isolation component 1 to the first interface scheduling module; the first interface scheduling module may first transmit the first task and the address of the first language understanding module to the isolation component 2, and then from the isolation component 2 to the first result generation module; the first result generation module may first transmit the first task to the isolation component 3, and then from the isolation component 3 to the first language understanding module. Conversely, the first language understanding module may first transmit the first token and M tensors to the isolation component 3, and then from the isolation component 3 to the first result generation module; the first result generation module may first transmit the first token to the Nth token to the isolation component 2, and then from the isolation component 2 to the first interface scheduling module; the first interface scheduling module may first transmit the text converted from the first token to the Nth token to the isolation component 1, and then from the isolation component 1 to the user-visible interface.

[0103] Exemplarily, the isolation component and the first interface scheduling module may communicate through the generate interface on the first interface scheduling module. The isolation component and the first result generation module may communicate through the forward interface on the first result generation module. The isolation component and the first language understanding module may communicate through the forward interface on the first language understanding module.

[0104] Those skilled in the art should understand that the isolation component and each module may also communicate using other interfaces, and the embodiments of the present disclosure do not limit this.

[0105] The following describes the usage of the result generation module and the language understanding module of the embodiments of the present disclosure for a computing device.

[0106] In a possible implementation, each result generation module and each language understanding module include multiple computing devices, and the models of different computing devices may be the same or different;

[0107] The number of tasks that any two result generation modules can process simultaneously may be the same or different, and the difference in the time required for any two language understanding modules to process the same number of tasks is less than the first threshold.

[0108] For example, the result generation module of the present disclosure embodiment may include a plurality of homogeneous computing devices or a plurality of heterogeneous computing devices. When including a plurality of homogeneous computing devices, the number and model of the computing devices included in each result generation module are the same. At this time, the computing capabilities of each result generation module may be the same, and the number of tasks that each result generation module can process simultaneously is the same. When including a plurality of heterogeneous computing devices, the models of the computing devices included in any two result generation modules may be different, the computing capabilities of any two result generation modules may be the same or different, and the number of tasks that any two result generation modules can process simultaneously may be the same or different. The present disclosure embodiment does not limit the specific models of the computing devices included in the result generation module.

[0109] Those skilled in the art should understand that if the computing capability of a single computing device can already meet the computing capability requirements of the result generation stage, the result generation module may also include only one computing device. For example, when the computing capability of 1 computing device of a certain model can simultaneously complete the result generation stage of 4 tasks, the result generation module A1 may include 1 computing device of this model. The present disclosure embodiment does not limit the specific number of computing devices included in the result generation module.

[0110] The computing capability of the result generation module only needs to ensure that the generation speed meets the user's requirements. The present disclosure embodiment does not limit the specific computing capabilities of each result generation module.

[0111] The language understanding module of the present disclosure embodiment may include a plurality of homogeneous computing devices or a plurality of heterogeneous computing devices. When including a plurality of homogeneous computing devices, the number and model of the computing devices included in each language understanding module are the same. At this time, the computing capabilities of each language understanding module may be the same, and the number of tasks that each language understanding module can process simultaneously is the same.

[0112] When a single task is large and a single computing device cannot complete it alone, existing technical means such as tensor model parallelism and pipeline model parallelism can be used to slice the task into multiple computing devices for completion. The present disclosure embodiment does not limit how many computing devices jointly complete a single task.

[0113] For example, when the computing capabilities of 4 computing devices of a certain model can simultaneously complete the language understanding stage of 2 tasks, the language understanding modules B1 and B2 may respectively include 4 computing devices of this model.

[0114] When the language understanding module includes multiple heterogeneous computing devices, the models of the computing devices included in any two language understanding modules can be different, and the computing capabilities of any two language understanding modules can be similar, that is, the difference in the time required for any two language understanding modules to process the same number of tasks is less than the first threshold, so as to ensure that the first-word delays corresponding to different users are similar. For example, when two computing devices of the first model and two computing devices of the second model can complete the language understanding stage of two tasks at the same time, the language understanding module B1 can include two computing devices of the first model and two computing devices of the second model, and the language understanding module B2 can also include two computing devices of the first model and two computing devices of the second model. The specific models of the computing devices included in the language understanding module in the embodiments of the present disclosure are not limited.

[0115] Those skilled in the art should understand that if the computing power of a single computing device can already meet the computing power requirements of the language understanding stage, then the language understanding module can also include only one computing device. The specific number of computing devices included in the language understanding module in the embodiments of the present disclosure is not limited.

[0116] The interface scheduling module does not undertake the processing work of the first task. The embodiments of the present disclosure do not limit the number and model of the computing devices included in the interface scheduling module.

[0117] In this way, the text generation system of the embodiments of the present disclosure supports multiple heterogeneous computing devices with different models, which is convenient to select more suitable computing devices according to the computing power requirements of the module, and can improve the system compatibility and the usage efficiency of the computing devices.

[0118] In a possible implementation manner, the first result generation module is further configured to, in response to receiving the address of the first language understanding module, allocate a corresponding first storage space for the first language understanding module from its own memory or video memory, and output the address of the first result generation module and the address of the first storage space to the first language understanding module;

[0119] The first language understanding module is specifically configured to transmit M tensors to the first storage space.

[0120] For example, since the embodiments of the present disclosure use two independent modules to complete the result generation stage and the language understanding stage respectively, the M tensors and the first token obtained in the language understanding stage of a certain task need to be transmitted to the module that completes the result generation stage in order to continue to complete the result generation stage of the task.

[0121] Taking the language understanding stage of the first task completed by the first language understanding module and the result generation stage of the first task completed by the first result generation module as an example, that is, the first language understanding module needs to transmit the first token and M tensors to the first result generation module. The data volume of the first token is relatively small and can be transmitted through application layer protocols such as the HyperText Transfer Protocol (HTTP) in the prior art. However, the data volume of the M tensors is relatively large, and when using the application layer protocol for transmission, the number of transmission handshakes is also large, resulting in a relatively large transmission delay.

[0122] In this regard, it can be set that the first result generation module is further configured to, in response to receiving the address of the first language understanding module, allocate a corresponding first storage space for the first language understanding module from its own memory or video memory. The first storage space can be specifically used to store the tensors generated in the language understanding stage of the first task from the first language understanding module, and the size of the first storage space can be greater than or equal to the size of the M tensors. The first result generation module outputs the address of the first result generation module and the address of the first storage space to the first language understanding module. The first language understanding module can transmit the tensors to the first storage space through the Remote Direct Memory Access (RDMA) method in the prior art according to the address of the first result generation module and the address of the first storage space. When the subsequent first interface scheduling module wants to use a certain tensor, it can read the corresponding tensor from the first storage space and use it.

[0123] In this way, the transmission delay of the tensors is reduced.

[0124] In a possible implementation manner, the first language understanding module transmits the generated tensors to the first storage space in real time.

[0125] For example, as described above, when the first language understanding module completes the language understanding stage, it includes M layers of calculations. Each layer of calculation obtains 1 tensor, and the output of the previous layer of calculation is used as the input of the next layer of calculation. Therefore, the M tensors are not obtained simultaneously but sequentially. In this regard, the first language understanding module can transmit the generated tensors to the first storage space in real time. In this case, the M tensors can be transmitted asynchronously, and the transmission processes of different tensors can be parallel to the calculation processes, further reducing the delay and reducing the time consumption of the language understanding stage.

[0126] Next, an exemplary method for the first result generation module of the present disclosure embodiment to generate the second to Nth tokens according to the first token and M tensors is introduced.

[0127] In a possible implementation manner, the first interface scheduling module is specifically configured to,

[0128] Generate the (i + 1)-th token based on the i-th token and M tensors, where 1 ≤ i < N and i is an integer.

[0129] For example, as described above, the first language understanding module has obtained the 1st token corresponding to the first task. The first result generation module undergoes N - 1 rounds of calculations in the result generation stage of the first task, and 1 token is obtained in each round of calculation. Then, in the i-th round of calculation, the (i + 1)-th token can be generated based on the i-th token and M tensors. In each round of calculation, M layers of calculations are experienced, and one of the M tensors is used as a parameter in each layer.

[0130] In one example, if the task to be processed by a result generation module is assigned to a language understanding module, then the result generation module only receives M tensors from the language understanding module. In the M layers of calculations in each round of calculation, the specific calculation method used in each layer is the same as that used by the large language model in the prior art in the result generation stage. Refer to the above and Figure 1 、 Figure 2 the relevant descriptions of, and details will not be repeated here.

[0131] In another example, the task to be processed by a result generation module may be assigned to multiple language understanding modules. Therefore, the result generation module will receive multiple sets of tensors from multiple language understanding modules, and each set of tensors includes M tensors. As shown in Figure 5 , tasks 1 - 4 processed by result generation module A1 are assigned to 2 language understanding modules B1 and B2. At this time, result generation module A1 will receive two sets of tensors from 2 language understanding modules B1 and B2. In this regard, in any layer of the first round of calculation of result generation module A1, the 2 tensors obtained by 2 language understanding modules B1 and B2 in the same layer of calculation can be first combined into one tensor (such as by weighted summation, etc.), and then the processed tensor is used for the calculation of the current layer. In any layer of each subsequent round of calculation, the tensor combined in the same layer of the first round is used for the calculation of the current layer.

[0132] Those skilled in the art should understand that other methods in the prior art can also be used for tensor combination, and the present disclosure does not limit this.

[0133] Result generation module A1 can start processing tasks when the number of tasks corresponding to the received tensors reaches the number of tasks that it can process simultaneously and it also receives the 1st token of the tasks corresponding to the tensors. Or a certain threshold can also be set, and tasks are started to be processed when the number of tasks corresponding to the received tensors reaches the preset threshold and it also receives the 1st token of the tasks corresponding to the tensors. The embodiments of the present disclosure do not limit the timing for result generation module A1 to start processing tasks.

[0134] In a possible implementation, the first result generation module is further configured to output the first token to the Nth token to the first interface scheduling module;

[0135] The first interface scheduling module is further configured to generate text corresponding to the first task based on the received tokens and output it to the user-visible interface.

[0136] For example, since the first interface scheduling module is responsible for interacting with the user-visible interface, the first result generation module can output the first token to the Nth token to the first interface scheduling module, and the first interface scheduling module generates text corresponding to the first task based on the received tokens and outputs it to the user-visible interface. The following describes an exemplary manner in which the first interface scheduling module generates text corresponding to the first task based on the received tokens.

[0137] In a possible implementation, after the first result generation module generates a token, it is transmitted to the first interface scheduling module in real time.

[0138] Specifically, the first interface scheduling module is configured to generate corresponding text once in response to each received token and output it to the user-visible interface.

[0139] For example, as described above, when the first result generation module completes the result generation stage, it includes N-1 rounds of calculations, and the output of the previous round of calculations is used as the input of the next round of calculations. Therefore, the second token to the Nth token are not obtained simultaneously, but sequentially. In this regard, the first result generation module can transmit the generated tokens to the first interface scheduling module in real time. In this case, the second token to the Nth token can be transmitted asynchronously.

[0140] The first interface scheduling module can be set to generate corresponding text once in response to each received token and output it to the user-visible interface. In this case, the conversion from token to text and the transmission of tokens can be parallel, further reducing the latency, reducing the time consumption of the result generation stage, and bringing a better user experience.

[0141] The present disclosure also proposes a text generation method. Figure 7 A schematic diagram showing the flow of the text generation method according to an embodiment of the present disclosure.

[0142] As Figure 7 shown, in a possible implementation, the method is applied to a text generation system, the system includes a first result generation module, a first language understanding module, and the computing power of the first result generation module is less than that of the first language understanding module.

[0143] The method includes:

[0144] Step S71: In response to receiving the first task and the address of the first language understanding module, the first result generation module transmits the first task to the first language understanding module according to the address of the first language understanding module;

[0145] Step S72: The first language understanding module parses the first task to generate the first token and M tensors, and returns them to the first result generation module;

[0146] Step S73: In response to receiving the first token and M tensors returned by the first language understanding module, the first result generation module generates the second token to the Nth token according to the first token and M tensors, where M and N are positive integers, N is greater than 2, and the first token to the Nth token are used to generate the text corresponding to the first task.

[0147] In a possible implementation, the system further includes a first interface scheduling module, and the method further includes: The first interface scheduling module receives a first request from an interface visible to the user, generates the first task according to the first request and stores it in the task queue to be assigned; The first interface scheduling module assigns the tasks in the task queue to be assigned, where, in response to the first task being assigned to the first result generation module and the first language understanding module, the first task and the address of the first language understanding module are transmitted to the first result generation module.

[0148] In a possible implementation, the method further includes: The first result generation module outputs the first token to the Nth token to the first interface scheduling module; The first interface scheduling module generates the text corresponding to the first task according to the received tokens and outputs it to the interface visible to the user.

[0149] In a possible implementation, the first result generation module is one of multiple result generation modules included in the text generation system, the first language understanding module is one of multiple language understanding modules included in the text generation system, the computing power of the result generation module is less than that of the language understanding module, and the first interface scheduling module assigns the tasks in the task queue to be assigned, including: When A result generation modules and B language understanding modules are idle, the first interface scheduling module selects A result generation modules and B language understanding modules to process Y tasks in the task queue to be assigned, where Y is less than or equal to the sum of the task quantities that A result generation modules can process simultaneously and less than or equal to the sum of the task quantities that B language understanding modules can process simultaneously, and A, B, and Y are positive integers.

[0150] In a possible implementation manner, each result generation module and each language understanding module include multiple computing devices, and the models of different computing devices may be the same or different; the number of tasks that any two result generation modules can process simultaneously may be the same or different, and the difference in the time required for any two language understanding modules to process the same number of tasks is less than a first threshold.

[0151] In a possible implementation manner, the method further includes: in response to receiving the address of the first language understanding module, the first result generation module allocates a corresponding first storage space for the first language understanding module from its own memory or video memory, and outputs the address of the first result generation module and the address of the first storage space to the first language understanding module; the first language understanding module parses the first task to generate the first token and M tensors, and returns them to the first result generation module, including: the first language understanding module transfers the M tensors to the first storage space according to the address of the first result generation module and the address of the first storage space.

[0152] In a possible implementation manner, after generating a token, the first result generation module transmits it to the first interface scheduling module in real time, and the first interface scheduling module generates a text corresponding to the first task according to the received token and outputs it to an interface visible to the user, including: in response to each received token, the first interface scheduling module generates a corresponding text once and outputs it to an interface visible to the user.

[0153] In a possible implementation manner, generating the second token to the Nth token according to the first token and M tensors includes: generating the (i + 1)th token according to the ith token and M tensors, where 1 ≤ i < N and i is an integer.

[0154] In a possible implementation manner, generating a first task according to the first request and storing it in the task queue to be allocated includes: performing tagging processing on the first request to generate the first task, where the first task includes W tokens, and the first token to the Nth token are associated with the W tokens, and W is an integer.

[0155] The embodiments of the present disclosure also propose an electronic device including the above-mentioned text generation system. The electronic device may be a terminal device or a server, and the present disclosure does not limit this.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0157] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A text generation system, characterized in that, The system includes a first result generation module and a first language understanding module, and the computing power of the first result generation module is less than that of the first language understanding module. The first result generation module is configured to, in response to receiving a first task and the address of the first language understanding module, determine that the result generation stage of the first task is completed by the first result generation module and the language understanding stage of the first task is completed by the first language understanding module, and transmit the first task to the first language understanding module according to the address of the first language understanding module, where the computing power requirement of the result generation stage is less than that of the language understanding stage. The first language understanding module is configured to parse the first task to generate a first token and M tensors, and return them to the first result generation module. The first result generation module is further configured to, in response to receiving the first token and the M tensors returned by the first language understanding module, generate a second token to an Nth token based on the first token and the M tensors, where M and N are positive integers, N is greater than 2, and the first token to the Nth token are used to generate text corresponding to the first task. Wherein, when the first language understanding module completes the language understanding stage, it includes M layers of calculations, and one tensor is obtained for each layer of calculation, and the output of the previous layer of calculation is used as the input of the next layer of calculation; specifically, the first result generation module is configured to generate an (i + 1)th token based on the ith token and the M tensors, where 1 ≤ i < N and i is an integer.

2. The system according to claim 1, characterized in that The system further includes a first interface scheduling module, and the first interface scheduling module is configured to: Receive a first request from an interface visible to the user, generate the first task according to the first request, and store it in the task queue to be assigned. Allocate tasks in the task queue to be assigned. Wherein, in response to the first task being assigned to the first result generation module and the first language understanding module, transmit the first task and the address of the first language understanding module to the first result generation module.

3. The system according to claim 2, wherein The first result generation module is further configured to output the first token to the Nth token to the first interface scheduling module. The first interface scheduling module is further configured to generate text corresponding to the first task based on the received tokens and output it to an interface visible to the user.

4. The system according to claim 2, wherein The first result generation module is one of multiple result generation modules included in the text generation system, and the first language understanding module is one of multiple language understanding modules included in the text generation system. The computing power of the result generation module is less than that of the language understanding module. Specifically, the first interface scheduling module is configured to In response to A result generation modules and B language understanding modules being idle, select A result generation modules and B language understanding modules to process Y tasks in the task queue to be assigned, where Y is less than or equal to the sum of the number of tasks that A result generation modules can process simultaneously and less than or equal to the sum of the number of tasks that B language understanding modules can process simultaneously, and A, B, and Y are positive integers.

5. The system according to claim 4, wherein Each result generation module and each language understanding module include multiple computing devices, and the models of different computing devices are the same or different; The number of tasks that any two result generation modules can process simultaneously is the same or different, and the difference in the time required for any two language understanding modules to process the same number of tasks is less than the first threshold.

6. The system according to claim 1, wherein The first result generation module is further configured to, in response to receiving the address of the first language understanding module, allocate a corresponding first storage space for the first language understanding module from its own memory or video memory, and output the address of the first result generation module and the address of the first storage space to the first language understanding module; The first language understanding module is specifically configured to transmit the M tensors to the first storage space according to the address of the first result generation module and the address of the first storage space.

7. The system according to claim 3, wherein The first result generation module transmits the generated tokens to the first interface scheduling module in real time, The first interface scheduling module is specifically configured to, in response to each received token, generate a corresponding text and output it to an interface visible to the user.

8. The system according to claim 2, characterized in that, The first interface scheduling module is specifically configured to perform tagging processing on the first request to generate the first task, where the first task includes W tokens, and the first token to the Nth token are associated with the W tokens, and W is an integer.

9. A text generation method, characterized in that, The method is applied to a text generation system, the system includes a first result generation module and a first language understanding module, the computing power of the first result generation module is less than that of the first language understanding module, and the method includes: The first result generation module, in response to receiving the first task and the address of the first language understanding module, determines that the result generation stage of the first task is completed by the first result generation module, and the language understanding stage of the first task is completed by the first language understanding module, and transmits the first task to the first language understanding module according to the address of the first language understanding module, and the computing power requirement of the result generation stage is less than that of the language understanding stage; The first language understanding module parses the first task to generate the first token and M tensors, and returns them to the first result generation module; The first result generation module, in response to receiving the first token and M tensors returned by the first language understanding module, generates the second token to the Nth token according to the first token and M tensors, where M and N are positive integers, N is greater than 2, and the first token to the Nth token are used to generate the text corresponding to the first task; Wherein, when the first language understanding module completes the language understanding stage, it includes M layers of calculations, and each layer of calculation obtains 1 tensor, and the output of the previous layer of calculation is used as the input of the next layer of calculation; generating the second token to the Nth token according to the first token and M tensors includes: generating the (i + 1)th token according to the i-th token and M tensors, where 1 ≤ i < N and i is an integer.

Images