Post-processing method and device for large model

Abstract

The embodiment of the invention provides a post-processing method and device for a large model, relates to the technical field of artificial intelligence, and is used for improving the reasoning speed of the large model. The method comprises the steps that a candidate word matrix and a plurality of sampling parameter matrixes are obtained from a host, sampling parameter values in each sampling parameter matrix are sampling parameter values of all to-be-reasoned problems under the same sampling strategy, and the sampling parameter matrixes are obtained by preprocessing the sampling parameter values of all the to-be-reasoned problems and then combining the sampling parameter values of all the to-be-reasoned problems; based on the plurality of sampling parameter matrixes, sampling in sequence in the candidate word matrixes by using corresponding sampling strategies, and determining output candidate words of the to-be-reasoned problems; and writing the output candidate words of the to-be-reasoned problems into the host.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a post-processing method and device for a large model. Background Art

[0002] Large model technology, especially large pre-trained language models, has become an important direction in the field of artificial intelligence. The reasoning performance of large models is an area of ​​great concern in current research.

[0003] Post-processing of large models is an important part of large model reasoning. The post-processing process usually uses different sampling strategies to balance the diversity and accuracy of the output. Different sampling strategies are processed as independent operators during sampling. Each sampling strategy requires data reading and writing operations during sampling, which causes the entire post-processing process to be time-consuming and affects the reasoning speed of large models. Summary of the invention

[0004] The embodiments of the present application provide a post-processing method, device and storage medium for a large model to improve the reasoning speed of the large model.

[0005] On the one hand, an embodiment of the present application provides a post-processing method for a large model, comprising:

[0006] Acquire a candidate word matrix and multiple sampling parameter matrices from a host, wherein the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy, and the multiple sampling parameter matrices are obtained by combining the sampling parameter values ​​of each problem to be inferred after preprocessing;

[0007] Based on the multiple sampling parameter matrices, respectively use corresponding sampling strategies to perform sampling in the candidate word matrix in sequence to determine the output candidate words of each problem to be inferred;

[0008] The output candidate words of each of the questions to be inferred are written into the host.

[0009] On the one hand, the embodiment of the present application provides another large model post-processing method, including:

[0010] Obtain the candidate word matrix and the sampling parameter values ​​of each question to be inferred;

[0011] Preprocessing the sampling parameter values, and combining the preprocessed sampling parameter values ​​to obtain multiple sampling parameter matrices, where the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy;

[0012] Writing the candidate word matrix and the plurality of sampling parameter matrices into a device;

[0013] The output candidate words of each reasoning problem determined in the device are read, wherein the output candidate words of each reasoning problem are determined by the device based on the multiple sampling parameter matrices, respectively, by sampling in the candidate word matrix in sequence using corresponding sampling strategies.

[0014] On the one hand, an embodiment of the present application provides a post-processing device for a large model, comprising:

[0015] A receiving unit is used to obtain a candidate word matrix and multiple sampling parameter matrices from a host, wherein the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy, and the multiple sampling parameter matrices are obtained by combining the sampling parameter values ​​of each problem to be inferred after preprocessing;

[0016] A processing unit, configured to sequentially sample in the candidate word matrix using corresponding sampling strategies based on the multiple sampling parameter matrices, and determine output candidate words for each of the questions to be inferred;

[0017] The sending unit is used to write the output candidate words of each problem to be inferred into the host.

[0018] Optionally, the multiple sampling parameter matrices are obtained in the following manner:

[0019] The sampling parameter values ​​of the problems to be inferred are obtained, the sampling parameter values ​​are preprocessed, and the sampling parameter values ​​belonging to the same sampling strategy among the sampling parameter values ​​of the problems to be inferred are combined into a sampling parameter matrix to obtain multiple sampling parameter matrices.

[0020] Optionally, the preprocessing includes one or more of the following:

[0021] Set missing sampling parameter values ​​to pre-configured default values;

[0022] Adding a random sampling parameter for adjusting the range of output candidate words, and setting the sampling parameter value of the random sampling parameter to a preset value or a random value;

[0023] The sampling parameter values ​​that are not within the pre-configured value interval are adjusted to be within the pre-configured value interval.

[0024] Optionally, the processing unit is specifically used to:

[0025] By at least one computing unit, the output candidate words of each problem to be inferred are sorted according to the output probability of each output candidate word in the candidate word matrix to obtain a sorting result, and the output candidate words with the top M output probabilities are selected from the sorting results of the output candidate words corresponding to each problem to be inferred, where M is a positive integer;

[0026] Based on the multiple sampling parameter matrices, corresponding sampling strategies are used to sequentially sample the top M output candidate words with output probabilities corresponding to the questions to be inferred, so as to determine the output candidate words for the questions to be inferred.

[0027] Optionally, the processing unit is specifically used to:

[0028] For the output candidate words of each of the problems to be inferred, the following operations are performed respectively:

[0029] Divide the output candidate words into N groups, where N is a positive integer less than or equal to the number of the computing units;

[0030] By using a plurality of computing units, the output probabilities of N groups of output candidate words are sorted respectively to obtain N output results, each of which includes the output candidate words with the top L output probabilities in a group of output candidate words, where L is a positive integer and the product of L and N is greater than or equal to M;

[0031] Each computing unit is controlled to obtain the output results of other computing units, and its own output results and the output results of other computing units are fully sorted to obtain a fully sorted result, and the output candidate words with the top M output probabilities are selected from the fully sorted result.

[0032] Optionally, the processing unit is specifically used to:

[0033] In the case where the multiple computing units do not have a shared memory, controlling each computing unit to store its own output result into a pre-configured temporary cache space;

[0034] Each computing unit is controlled to obtain output results of other computing units from the temporary cache space.

[0035] Optionally, the processing unit is specifically used to:

[0036] For a group of output candidate words, the following operations are performed respectively until all the output candidate words in the group of output candidate words are sorted, and the output candidate words with the top L output probabilities are selected from the candidate word list as the output results of the group of output candidate words:

[0037] Selecting a preset number of output candidate words from the group of output candidate words, performing full sorting on the output probabilities of the selected output candidate words to obtain an intermediate sorting result, and selecting the output candidate words with the top L output probabilities from the intermediate sorting result;

[0038] Writing the first L output candidate words into the first L positions or the last L positions in a pre-configured candidate word list, the candidate word list comprising 2L positions, wherein when not writing for the first time, the output probability arrangement order of the output candidate words written into the candidate word list is opposite to the output probability arrangement order of the output candidate words in the candidate word list;

[0039] The output candidate words in the candidate word list are arranged in a two-tone ascending order or a two-tone descending order, and output candidate words are continuously selected from the group of output candidate words for sorting.

[0040] On the one hand, the embodiment of the present application provides another large model post-processing device, including:

[0041] An acquisition unit, used to acquire a candidate word matrix and sampling parameter values ​​of each question to be inferred;

[0042] A processing unit, used to preprocess the sampling parameter values ​​and combine the preprocessed sampling parameter values ​​to obtain multiple sampling parameter matrices, wherein the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy;

[0043] A sending unit, used for writing the candidate word matrix and the plurality of sampling parameter matrices into a device;

[0044] A receiving unit is used to read the output candidate words of each reasoning problem determined in the device, where the output candidate words of each reasoning problem are determined by the device based on the multiple sampling parameter matrices, using corresponding sampling strategies to sample in the candidate word matrix in sequence.

[0045] Optionally, the processing unit is specifically used to:

[0046] The sampling parameter values ​​are preprocessed, and the sampling parameter values ​​belonging to the same sampling strategy among the sampling parameter values ​​of the problems to be inferred are combined into a sampling parameter matrix to obtain a plurality of sampling parameter matrices.

[0047] Optionally, the preprocessing includes one or more of the following:

[0048] Set missing sampling parameter values ​​to pre-configured default values;

[0049] Adding a random sampling parameter for adjusting the range of output candidate words, and setting the sampling parameter value of the random sampling parameter to a preset value or a random value;

[0050] The sampling parameter values ​​that are not within the pre-configured value interval are adjusted to be within the pre-configured value interval.

[0051] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, wherein the processor chip implements the steps of the above-mentioned post-processing method of the large model when executing the program.

[0052] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned large model post-processing method.

[0053] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned large model post-processing method.

[0054] In an embodiment of the present application, a candidate word matrix and multiple sampling parameter matrices are obtained from a host through a data read and write operation, wherein the sampling parameter values ​​in each sampling parameter matrix are the sampling parameter values ​​of each problem to be inferred under the same sampling strategy, and the multiple sampling parameter matrices are obtained by combining the sampling parameter values ​​of each problem to be inferred after preprocessing, and then, based on the multiple sampling parameter matrices, corresponding sampling strategies are used to perform sampling in the candidate word matrix in turn to determine the output candidate words of each problem to be inferred, and finally, the output candidate words of each problem to be inferred are written to the host through a write operation.

[0055] The embodiment of the present application can complete the post-processing process of the large model through a single data read and write operation. Compared with the related art in which each sampling strategy needs to perform data read and write operations when sampling, the number of data read and write operations is reduced, and the time consumption of the post-processing process of the large model is shortened, thereby improving the reasoning speed of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0058] Figure 2 A schematic diagram of a process flow of a post-processing method for a large model provided in an embodiment of the present application;

[0059] Figure 3 A schematic flow chart of another large model post-processing method provided in an embodiment of the present application;

[0060] Figure 4 A flowchart of a specific implementation process of a post-processing method for a large model provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of the structure of a large-scale post-processing device provided in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the structure of another large model post-processing device provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0065] It is understandable that in the following specific implementation methods of this application, problems and sampling parameters are involved. When the various embodiments of this application are applied to specific products or technologies, relevant licenses or consents need to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0066] The following is an introduction to some concepts involved in the embodiments of the present application.

[0067] 1. Token, or word unit, usually refers to a discrete text unit in a large language model. It can be a word, punctuation mark, number or other language elements, which are used as the basic unit for training and generating text.

[0068] 2. Top-K sampling is an optimization of greedy sampling (selecting the token with the highest probability from all tokens). Its parameter K represents the number of tokens to be selected. K can be set by the user and is usually a positive integer. When K=1, Top-K sampling is greedy sampling.

[0069] When sampling, the first K tokens are selected from the token probability descending order, and a random selection is made from the K tokens as the output. Although this sampling method is simple and efficient, it may lead to insufficient diversity in the generated text.

[0070] 3. Top-P sampling, which selects tokens by percentage from the token probability descending sort results. Its parameter P is equivalent to a cumulative probability threshold. P is greater than or equal to 0 and less than or equal to 1. P is usually set to a relatively high value.

[0071] When sampling, we start from the token with the highest probability in descending order, calculate the cumulative probability of the first L tokens, and then randomly select a token as output from the minimum set whose cumulative probability is greater than or equal to probability P (that is, the set including the first L tokens when the first L tokens are greater than or equal to P for the first time), where L is a natural number. This sampling strategy can increase the diversity of the text while maintaining a certain degree of accuracy.

[0072] For example, assuming that token1 is "girl" with a probability of 66%, token2 is "shoes" with a probability of 20%, token3 is "elephant" with a probability of 10%, and token4 is "watermelon" with a probability of 4%. If the parameter P of Top-P sampling is set to 0.85 (85%), then the cumulative probability corresponding to token1 is 66%, which is less than P. The cumulative probability of the set composed of token1 and token2 is 86%, which is greater than P for the first time. Then the sampling results of Top-P are token1 and token2.

[0073] In practical applications, during the Top-P sampling process, a random sampling parameter r can also be set to use the token added when the cumulative probability is greater than or equal to P*r for the first time as the final selected token, thereby controlling the final selected token and increasing the diversity of the sampling results. The random sampling parameter r can be any value in the interval [0, 1].

[0074] For example, assuming that token1 is "apple" with a probability of 40%, token2 is "pear" with a probability of 30%, token3 is "banana" with a probability of 20%, and token4 is "orange" with a probability of 10%. If the parameter P of Top-P sampling is set to 0.8 (80%) and no random sampling parameter is set, the sampling results of Top-P are token1, token2, and token3.

[0075] If the random sampling parameter is set to 0.6, the cumulative probability corresponding to token1 is 0.4, which is less than 0.48 (P*r, that is, 0.8*0.6), and the cumulative probability of token1 and token2 is 0.7, which is greater than 0.48 for the first time, then the sampling result is token2; if the random sampling parameter is set to 0.9, in this case, the cumulative probability corresponding to token1 is 0.4, which is less than 0.72 (P*r, that is, 0.8*0.9), and the cumulative probability of token1 and token2 is 0.7, which is less than 0.72, and the cumulative probability of token1, token2 and token3 is 0.9, which is greater than 0.72 for the first time, then the sampling result is token3.

[0076] 4. Temperature sampling, by adjusting the temperature parameter of the probability distribution, to control the randomness of the generated text. A higher temperature value will increase the sampling chance of low-probability tokens, thereby improving the diversity of the text.

[0077] 5. Bitonic sorting is a parallel computing sorting method. Taking ascending order as an example, the sorting involves the following steps:

[0078] (1) Sort a given random sequence into a bitonic sequence. A bitonic sequence is a sequence in which the elements are first in increasing (decreasing) order and then start decreasing (increasing) after a certain element.

[0079] (2) Compare the element value of the first element in the first half with the element value of the first element in the second half. If the element value of the first element in the second half is less than the element value of the first element in the first half, swap the positions of the first element in the first half and the first element in the second half. If the element value of the first element in the second half is greater than or equal to the element value of the first element in the first half, there is no need to swap the positions of the two elements. Compare the element value of the second element in the first half with the element value of the second element in the second half. Compare. If the element value of the second element in the second half is less than the element value of the second element in the first half, swap the positions of the second element in the first half and the second element in the second half. If the element value of the second element in the second half is greater than or equal to the element value of the second element in the first half, there is no need to swap the positions of the two elements... and so on, until each element in the first half is compared with the element in the corresponding position in the second half, and the element values ​​of all elements in the first half are less than the element values ​​of all elements in the second half.

[0080] (3) For the first half mentioned in (2), it is divided into two parts with the same number of elements, the first part is used as the first half, and the second part is used as the second half, and the sorting and swapping operations in step (2) are continued, and this cycle is repeated many times until it cannot be further divided; for the second half mentioned in (2), it is also divided into two parts with the same number of elements, the first part is used as the first half, and the second part is used as the second half, and the sorting and swapping operations in step (2) are continued, and this cycle is repeated many times until it cannot be further divided.

[0081] After multiple cycles of sorting and swapping operations, the first half and the second half mentioned in step (2) cannot be further divided, and the final ascending sorting result can be obtained.

[0082] For example, suppose there are 8 elements in the random sequence.

[0083] (1) Sort the random sequence into a bitonic sequence, in which the 1st to 4th elements are sorted in ascending order, and the 5th to 8th elements are sorted in descending order.

[0084] (2) Compare the element value of the first element with the element value of the fifth element. If the element value of the fifth element is less than the element value of the first element, swap the positions of the first element and the fifth element. If the element value of the fifth element is greater than or equal to the element value of the first element, there is no need to swap the positions of the two elements. Compare the element value of the second element with the element value of the sixth element. If the element value of the sixth element is less than the element value of the second element, swap the positions of the second element and the sixth element. If the element value of the sixth element is greater than or equal to the element value of the second element, there is no need to swap the positions of the two elements. ; compare the element value of the 3rd element with the element value of the 7th element, if the element value of the 7th element is less than the element value of the 3rd element, swap the positions of the 3rd and 7th elements, if the element value of the 7th element is greater than or equal to the element value of the 3rd element, there is no need to swap the positions of the two elements; compare the element value of the 4th element with the element value of the 8th element, if the element value of the 8th element is less than the element value of the 4th element, swap the positions of the 4th and 8th elements, if the element value of the 8th element is greater than or equal to the element value of the 4th element, there is no need to swap the positions of the two elements.

[0085] (3) Compare the element value of the first element with the element value of the third element. If the element value of the third element is less than the element value of the first element, swap the positions of the first element and the third element. If the element value of the third element is greater than or equal to the element value of the first element, there is no need to swap the positions of the two elements. Compare the element value of the second element with the element value of the fourth element. If the element value of the fourth element is less than the element value of the second element, swap the positions of the second element and the fourth element. If the element value of the fourth element is greater than or equal to the element value of the second element, there is no need to swap the positions of the two elements. ; compare the element value of the 5th element with the element value of the 7th element, if the element value of the 7th element is less than the element value of the 5th element, swap the positions of the 5th and 7th elements, if the element value of the 7th element is greater than or equal to the element value of the 5th element, there is no need to swap the positions of the two elements; compare the element value of the 6th element with the element value of the 8th element, if the element value of the 8th element is less than the element value of the 6th element, swap the positions of the 6th and 8th elements, if the element value of the 8th element is greater than or equal to the element value of the 6th element, there is no need to swap the positions of the two elements.

[0086] (4) Compare the element value of the first element with the element value of the second element. If the element value of the second element is less than the element value of the first element, swap the positions of the first element and the second element. If the element value of the second element is greater than or equal to the element value of the first element, there is no need to swap the positions of the two elements. Compare the element value of the third element with the element value of the fourth element. If the element value of the fourth element is less than the element value of the third element, swap the positions of the third element and the fourth element. If the element value of the fourth element is greater than or equal to the element value of the third element, there is no need to swap the positions of the two elements. ; compare the element value of the 5th element with the element value of the 6th element, if the element value of the 6th element is less than the element value of the 5th element, swap the positions of the 5th and 6th elements, if the element value of the 6th element is greater than or equal to the element value of the 5th element, there is no need to swap the positions of the two elements; compare the element value of the 7th element with the element value of the 8th element, if the element value of the 8th element is less than the element value of the 7th element, swap the positions of the 7th and 8th elements, if the element value of the 8th element is greater than or equal to the element value of the 7th element, there is no need to swap the positions of the two elements.

[0087] After the above four steps, the ascending order of the 8 elements in the random sequence can be obtained.

[0088] The following is a brief introduction to the design concept of the embodiment of the present application:

[0089] Large model technology, especially large pre-trained language models, has become an important direction in the field of artificial intelligence. The reasoning performance of large models is an area of ​​great concern in current research.

[0090] Post-processing of large models is an important part of large model reasoning. The post-processing process usually uses different sampling strategies to balance the diversity and accuracy of the output. Different sampling strategies are processed as independent operators during sampling. Each sampling strategy requires data reading and writing operations during sampling, which causes the entire post-processing process to be time-consuming and affects the reasoning speed of large models.

[0091] In view of this, an embodiment of the present application provides a post-processing method and device for a large model, which obtains a candidate word matrix and multiple sampling parameter matrices from a host through a data read and write operation, wherein the sampling parameter values ​​in each sampling parameter matrix are the sampling parameter values ​​of each problem to be inferred under the same sampling strategy, and the multiple sampling parameter matrices are obtained by combining the sampling parameter values ​​of each problem to be inferred after preprocessing, and then based on the multiple sampling parameter matrices, the corresponding sampling strategies are used to sample in the candidate word matrix in turn to determine the output candidate words of each problem to be inferred, and finally the output candidate words of each problem to be inferred are written to the host through a write operation.

[0092] The embodiment of the present application can complete the post-processing process of the large model through a single data read and write operation. Compared with the related art in which each sampling strategy needs to perform data read and write operations when sampling, the number of data read and write operations is reduced, and the time consumption of the post-processing process of the large model is shortened, thereby improving the reasoning speed of the large model.

[0093] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0094] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes any terminal device 110 among multiple terminal devices and any server 120 among multiple servers. The server 120 includes a device and a host, wherein the device may be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), etc., and the host may be a central processing unit (CPU), etc.

[0095] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers and other devices; a large model reasoning client can be installed on the terminal device 110, which can receive questions raised by users, perform reasoning, and give corresponding replies. The client can be software, or a web page, a small program, etc. The server 120 is the background service end of the large model reasoning client. The devices and hosts in the server 120 cooperate with each other to complete the reasoning of the problem. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0096] It should be noted that the post-processing method of the large model in the embodiment of the present application can be executed by the server 120, specifically by the devices and the host in the server 120. First, the host obtains the questions to be inferred and the sampling parameter values ​​corresponding to each question to be inferred input by the user through the terminal device 110. The host determines the candidate word matrix of each question to be inferred, and pre-processes the sampling parameter values ​​of each question to be inferred in the host. The pre-processed sampling parameter values ​​are combined to obtain multiple sampling parameter matrices. The sampling parameter values ​​in each sampling parameter matrix are the sampling parameter values ​​of each question to be inferred under the same sampling strategy.

[0097] During post-sampling processing, the device obtains a candidate word matrix and multiple sampling parameter matrices from the host, and based on the multiple sampling parameter matrices, uses corresponding sampling strategies to sample in the candidate word matrix in turn, determines the output candidate words of each problem to be inferred, and writes the output candidate words of each problem to be inferred to the host. The host displays the output results of each problem to be inferred to the user through the client in the terminal device 110.

[0098] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network, which may be a wired network or a wireless network.

[0099] It should be noted that Figure 1 What is shown is just an example. In fact, the number and communication method of terminal devices and servers are not limited. When there are multiple servers, multiple servers can form a blockchain, and the server is a node on the blockchain, which is not specifically limited in the embodiments of the present application.

[0100] The following describes the post-processing method of the large model provided by the exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate the understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.

[0101] See also Figure 2 As shown, it is an implementation flow chart of a post-processing method of a large model in an embodiment of the present application, with the host in the server as the execution subject, and the specific implementation process of the method is as follows S201-S204:

[0102] S201: Obtain a candidate word matrix and sampling parameter values ​​of each question to be inferred.

[0103] It should be noted that the candidate word matrix is ​​usually generated by a pre-trained large model (or a large language model) based on the reasoning of the problem to be reasoned, which includes the output candidate words and the corresponding output probabilities of each problem to be reasoned. The training process of the specific large model and the process of determining the output candidate words by reasoning of the large model are not specifically limited in the embodiments of the present application. The output candidate words mentioned in the embodiments of the present application can be words, punctuation marks, numbers or other language elements, which are not limited in the embodiments of the present application.

[0104] In specific implementation, the dimension of the candidate words is determined by the number of problems to be reasoned and the number of output candidate words for each problem to be reasoned. For example, for a reasoning process, if there are bs problems to be reasoned and each problem to be reasoned has vs output candidate words, then the dimension of the candidate word matrix is ​​bs×vs.

[0105] Among them, the problem to be inferred and the sampling parameter values ​​set for each problem to be inferred can be input by the user or generated in batches by other artificial intelligence tools, and the embodiments of the present application do not limit this. The sampling parameter values ​​mentioned in the embodiments of the present application refer to the parameter values ​​required for specific sampling of various sampling strategies, for example, the sampling parameter value K in the Top-K sampling strategy, the sampling parameter P in the Top-P sampling strategy, the sampling parameter value T in the temperature sampling strategy, and the random sampling parameter r in the random sampling strategy.

[0106] In an example, taking the example of a user inputting a question to be inferred and a sampling parameter value of the question to be inferred respectively, the user inputs question to be inferred 1 "What do rabbits like to eat?" through a terminal device, and sets the sampling parameter value K for Top-K sampling to 40, the sampling parameter value P for Top-P sampling to 0.8, and the sampling parameter value T for temperature sampling to 1.5; the user inputs question to be inferred 2 "What are the fish in the ocean?" through the terminal device, and sets the sampling parameter value K for Top-K sampling to 100, and the sampling parameter value T for temperature sampling to -1; the user inputs question to be inferred 3 "What is the smallest bird in the world?" through the terminal device, and sets the sampling parameter value P for Top-P sampling to 0.5.

[0107] S202: Preprocess the sampling parameter values, and combine the preprocessed sampling parameter values ​​to obtain multiple sampling parameter matrices, where the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy.

[0108] In specific implementation, the sampling parameter value is preprocessed, including one or more of the following processes:

[0109] Processing 1: Set the missing sampling parameter values ​​to pre-configured default values. Specifically, taking the post-processing of the large model including Top-K sampling, Top-P sampling and temperature sampling as an example, if one or more of the Top-K sampling parameter values, Top-P sampling parameter values ​​and temperature sampling parameter values ​​are not set for a problem to be inferred, the unset sampling parameter values ​​are set to the pre-configured default values.

[0110] Among them, the pre-configured default value can be set according to the empirical value. For Top-K sampling, in actual applications, the sum of the output probabilities of the first 10 output candidate words of an excellent large model can usually reach more than 95%. In some cases, there are even only one or two output candidate words to choose from. Therefore, the default value of the Top-K sampling parameter can be flexibly set. For example, the default value of the pre-configured Top-K sampling parameter is 64. For Top-P sampling, if the Top-P sampling parameter value is missing, it can be regarded as not performing Top-P sampling or Top-P sampling does not filter the output candidate words, that is, the value of P is 1, therefore, the default value of the Top-P sampling parameter can be set to 1. For temperature sampling, if the temperature sampling value is missing, it can be regarded as not performing temperature adjustment, that is, the temperature sampling parameter value is 1, therefore, the default value of the temperature sampling parameter can be set to 1. Of course, in other embodiments of the present application, the pre-configured default value may also be other values.

[0111] Taking the above example, problem 2 to be inferred is set with Top-K sampling parameter value and temperature sampling parameter value, but lacks Top-P sampling parameter value. In this case, during preprocessing, the Top-P sampling parameter value of problem 2 to be inferred can be set to 1. Problem 3 to be inferred only sets Top-P sampling parameter value, but lacks Top-K sampling parameter value and temperature sampling parameter value. In this case, during preprocessing, the Top-K sampling parameter value of problem 3 to be inferred can be set to 64 and the temperature sampling parameter value can be set to 1.

[0112] Processing 2: Add a random sampling parameter for adjusting the range of output candidate words, and set the sampling parameter value of the random sampling parameter to a preset value or a random value. The value range of the random sampling parameter is [0, 1], which can be combined with Top-P sampling to adjust the size of the sampling parameter value P, thereby adjusting the range of output candidate words.

[0113] After the random sampling parameter is specifically added, the sampling parameter value of the random sampling parameter can be set to a random value, for example, a random value is generated by a random function, or can be set to a preset value, for example, the preset value can be any value in the interval [0, 1].

[0114] Still taking the above example, random sampling parameters are added for problem 1 to be reasoned, problem 2 to be reasoned, and problem 3 to be reasoned respectively, and the sampling parameter value of the random sampling parameter of problem 1 to be reasoned is set to 0.4, the sampling parameter value of the random sampling parameter of problem 2 to be reasoned is set to 0.5, and the sampling parameter value of the random sampling parameter of problem 3 to be reasoned is set to 0.5.

[0115] Processing 3: Adjust the sampling parameter values ​​that are not in the pre-configured value range to the pre-configured value range. The value ranges of each sampling parameter value can be pre-configured according to actual needs. For example, the value range of the Top-K sampling parameter value is [1, 64], the value range of the Top-P sampling parameter value is [0, 1], and the value range of the temperature sampling parameter value is (0, +∞].

[0116] In practical applications, after pre-configuring the value intervals of each sampling parameter value, after obtaining the sampling parameter value of the problem to be inferred, if it is determined that the sampling parameter value is not within the corresponding value interval, the sampling parameter value is adjusted to the pre-configured value interval. When adjusting the sampling parameter value to the pre-configured value interval, the sampling parameter value can be adjusted to a pre-configured default value, or the sampling parameter value can be adjusted to the maximum value of the interval or the minimum value of the interval, which is not limited in the embodiments of the present application.

[0117] Still taking the above example, the Top-K sampling parameter value K set for problem 2 to be inferred is 100, and the temperature sampling parameter value T is -1. The sampling parameter value K is not in the value range [1, 64]. The sampling parameter value K can be adjusted to 64. The temperature sampling parameter value T is also not in the value range (0, +∞]. The temperature sampling parameter value T can be adjusted to 1. After processing 3, the Top-K sampling parameter value K of problem 2 to be inferred is 64, and the temperature sampling parameter value T is 1.

[0118] In specific implementation, the embodiment of the present application pre-processes the sampling parameter values ​​of each problem to be inferred, wherein the missing sampling parameter values ​​are filled (set to pre-configured default values) and random sampling parameters are added, so that each problem to be inferred has a sampling parameter value under each sampling strategy, and the sampling parameter values ​​that are not in the pre-configured value range are adjusted to the pre-configured value range, so that each sampling parameter value is in the corresponding value range. Therefore, when sampling is performed in the post-processing process of the large model, it can be processed uniformly without executing different logic for the special case of a certain sampling parameter value, thereby improving the efficiency of post-processing of the large model.

[0119] Assuming that the preprocessing includes the above-mentioned processing 1, processing 2 and processing 3, in the above example, the Top-K sampling parameter value K after preprocessing of the reasoning problem 1 is 40, the Top-P sampling parameter value P is 0.8, the temperature sampling parameter value T is 1.5, and the sampling parameter value of the random sampling parameter is 0.4; the Top-K sampling parameter value K after preprocessing of the reasoning problem 2 is 64, the Top-P sampling parameter value P is 1, the temperature sampling parameter value T is 1, and the sampling parameter value of the random sampling parameter is 0.5; the Top-K sampling parameter value K after preprocessing of the reasoning problem 3 is 64, the Top-P sampling parameter value P is 0.5, the temperature sampling parameter value T is 1, and the sampling parameter value of the random sampling parameter is 0.5.

[0120] In specific implementation, after preprocessing the sampling parameter values, the preprocessed sampling parameter values ​​can be combined, and the sampling parameter values ​​belonging to the same sampling strategy among the sampling parameter values ​​of each problem to be inferred can be combined into a sampling parameter matrix to obtain multiple sampling parameter matrices.

[0121] In the above example, for problems 1, 2, and 3 to be inferred, the Top-K sampling parameter values ​​are combined into a sampling parameter matrix (40, 64, 64), the Top-P sampling parameter values ​​are combined into a sampling parameter matrix (0.8, 1, 0.5), the temperature sampling parameter values ​​are combined into a sampling parameter matrix (1.5, 1, 1), and the sampling parameter values ​​of the random sampling parameters are combined into a sampling parameter matrix (0.4, 0.5, 0.5), resulting in a total of 4 sampling parameter matrices.

[0122] S203: Writing the candidate word matrix and multiple sampling parameter matrices into the device.

[0123] In specific implementation, the specific sampling is usually carried out in the device during the post-processing of the large model. Therefore, after the sampling parameters of the problem to be inferred are combined into multiple sampling parameter matrices, the candidate word matrix and the multiple sampling parameter matrices are written into the device, so that the device uses the corresponding sampling strategies based on the multiple sampling parameter matrices to sample in the candidate word matrix in turn to determine the output candidate words for each problem to be inferred.

[0124] S204: Reading the output candidate words of each reasoning problem determined in the device. The output candidate words of each reasoning problem are determined by the device based on multiple sampling parameter matrices and using corresponding sampling strategies to sample in the candidate word matrix in sequence.

[0125] In a specific implementation, after the device determines the output candidate words for each question to be inferred, the output candidate words for each question can be read from the device and written into the host of the server.

[0126] In the post-processing method of the large model provided in the embodiment of the present application, the candidate word matrix and multiple sampling parameter matrices are written into the device, and the output candidate words of each reasoning problem are read from the device. The post-processing process of the large model can be completed through one data reading and writing operation, which reduces the number of data reading and writing times, shortens the time consumption of the post-processing process of the large model, and thus improves the reasoning speed of the large model.

[0127] See also Figure 3 As shown, it is an implementation flow chart of another large model post-processing method in the embodiment of the present application, with the device in the server as the execution subject, and the specific implementation process of the method is as follows S301-S303:

[0128] S301: Obtain a candidate word matrix and multiple sampling parameter matrices from a host, wherein the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy, and the multiple sampling parameter matrices are obtained by combining the sampling parameter values ​​of each problem to be inferred after preprocessing.

[0129] During specific implementation, the device obtains a candidate word matrix and multiple sampling parameter matrices from the host. The generation method of the candidate word matrix and multiple sampling parameter matrices has been described in detail in other embodiments of the present application and will not be repeated here.

[0130] S302: Based on multiple sampling parameter matrices, use corresponding sampling strategies to sample in the candidate word matrix in turn to determine the output candidate words of each problem to be inferred.

[0131] In a specific implementation, multiple sampling parameter matrices include, but are not limited to: a Top-K sampling parameter matrix, a Top-P sampling parameter matrix, a temperature sampling parameter matrix, and a random sampling parameter matrix.

[0132] In practical applications, based on multiple sampling parameter matrices, corresponding sampling strategies are used to sample in the candidate word matrix in sequence to determine the output candidate words of each problem to be reasoned. At least one computing unit (Compute Unit, CU) can be used to sort the output candidate words of each problem to be reasoned according to the output probability of each output candidate word in the candidate word matrix to obtain a sorting result, and select the output candidate words with the top M output probabilities from the sorted results of the output candidate words corresponding to each problem to be reasoned, where M is a positive integer; then based on multiple sampling parameter matrices, corresponding sampling strategies are used to sample in sequence from the output candidate words with the top M output probabilities corresponding to each problem to be reasoned to determine the output candidate words of each problem to be reasoned.

[0133] It should be noted that the value of M can be set according to empirical values. In practical applications, the value of M needs to be greater than or equal to the maximum value of the sampling parameter K in the Top-K sampling strategy. For example, in an embodiment of the present application, the value interval of the sampling parameter K is [1, 64], then the value of M needs to be greater than or equal to 64 to ensure that the output candidate words before the selected output probability meet the Top-K sampling requirements.

[0134] In a specific implementation, when the output candidate words for the reasoning problem are sorted, the output candidate words for the reasoning problem can be sorted in one computing unit, and the output candidate words with the top M output probabilities are selected from the final sorting results.

[0135] In practical applications, bitonic sorting is used to sort the output candidate words of the inference problem, and threads need to access data out of order during sorting. The data must be placed in shared memory, and the shared memory is limited by the hardware CU. Each CU has 32 kilobytes (KB) of shared memory. Each CU includes 4 execution units (EUs), each EU has 32 threads, and each CU can use 128 (32*4) threads.

[0136] In order to break the limitation of 128 parallel threads, when sorting the output candidate words of the problem to be inferred, multiple computing units can be used to sort the output candidate words of the problem to be inferred in parallel. Specifically, taking a streaming processing cluster (SPC) as an example, each SPC includes 4 CUs, and then 4 CUs can be used to sort the output candidate words of a problem to be inferred, increasing the number of concurrent threads to 512 (128*4), thereby greatly improving the sorting speed.

[0137] In the specific implementation, the output candidate words of the problem to be inferred are divided into 4 groups, and then the 4 groups of output candidate words are sorted in parallel in 4 CUs respectively to obtain 4 output results, each of which includes the output candidate words with the top L output probabilities in a group of output candidate words.

[0138] After the four CUs calculate the output results respectively, if the memory can be shared among the four CUs, each computing unit obtains the output results of other computing units, and fully sorts its own output results and the output results of other computing units (4*L output candidate words) to obtain the fully sorted results, and selects the output candidate words with the top M output probabilities from the fully sorted results.

[0139] If the four CUs cannot share memory, they exchange data through a pre-configured temporary cache space. Specifically, each computing unit stores its own output results in the temporary cache space, obtains the output results of other computing units from the temporary cache space, and fully sorts its own output results and the output results of other computing units (4*L output candidate words) to obtain a fully sorted result, and selects the top M output candidate words with the highest output probability from the fully sorted result.

[0140] The temporary cache space may be designed in the hardware architecture. For example, the main data block (GEMM Main Buffer, GMB) space used for matrix multiplication in the SPC may be used as the temporary cache space.

[0141] It should be noted that when grouping output candidate words for reasoning problems, the output candidate words can be divided into N groups, where N is a positive integer less than or equal to the number of computing units, and the output result of each group of output candidate words includes L output candidate words, where L is a positive integer, and the product of L and N is greater than or equal to M, to ensure that the output candidate words with the top M output probabilities can be selected from the full sorting results.

[0142] Specifically, in a computing unit, the output candidate words for the reasoning problem are sorted, and when the output result is obtained, a bitonic sorting method can be used.

[0143] Specifically, for a set of output candidate words, the following operations are performed:

[0144] Operation 1, selecting a preset number of output candidate words from the group of output candidate words, fully sorting the output probabilities of the selected output candidate words to obtain an intermediate sorting result, and selecting the output candidate words with the top L output probabilities from the intermediate sorting result.

[0145] The preset number can be set according to experience. Based on the characteristics of bitonic sorting, its value is preferably 2. n When the group of output candidate words is fully sorted, the change of the preset number does not affect the number of calculations, but the smaller the preset number is, the more times the output candidate words with the top L output probabilities are selected, so the larger the preset number, the better. Under the 32KB shared memory limit, the preset number can be 512, 1024 or 2048.

[0146] Operation 2, writes the first L output candidate words into the first L positions or the last L positions in a pre-configured candidate word list, where the candidate word list contains 2L positions, wherein, when not writing for the first time, the output probability arrangement order of the output candidate words written into the candidate word list is opposite to the output probability arrangement order of the output candidate words in the candidate word list.

[0147] In the specific implementation, since it is ultimately necessary to select the top L output candidate words with the highest output probability from the candidate word list as the output result of the group of output candidate words, and operation 2 needs to write the top L output candidate words selected in operation 1 into the top L positions or the bottom L positions in the candidate word list. This write operation will overwrite the original output candidate words in the candidate word list. In order to avoid overwriting the larger L output candidate words in the candidate word list, when writing the top L output candidate words selected in operation 1 into the candidate word list, it needs to be determined according to the sorting method of the output candidate words in the candidate word list.

[0148] In one embodiment, if the output candidate words in the candidate word list are arranged in ascending order, the larger L output candidate words in the candidate word list are located in the last L positions. When the first L output candidate words obtained in operation 1 are written into the candidate word list, the first L output candidate words obtained in operation 1 are written into the first L positions of the candidate word list, and to ensure that the sequence in the candidate word list after writing is a bitonal sequence, when the first L output candidate words obtained in operation 1 are written into the first L positions of the candidate word list, they are written in descending order of output probability.

[0149] In another embodiment, if the output candidate words in the candidate word list are arranged in descending order, the larger L output candidate words in the candidate word list are located in the first L positions, and when the first L output candidate words obtained in operation 1 are written into the candidate word list, the first L output candidate words obtained in operation 1 are written into the last L positions of the candidate word list, and in order to ensure that the sequence in the candidate word list after writing is a bitonal sequence, when the first L output candidate words obtained in operation 1 are written into the last L positions of the candidate word list, they are written in ascending order of output probability.

[0150] It should be noted that the initial values ​​of each position in the pre-configured candidate word list can be all 0 or all -inf (negative infinity), and the present embodiment does not limit this. When the first L output candidate words obtained in operation 1 are first written into the first L positions or the last L positions of the pre-configured candidate word list, they can be written directly.

[0151] By setting a candidate word list and adjusting the way the output candidate words obtained in operation 1 are written into the candidate word list, the sequence in the candidate word list is made into a bitonal sequence. Therefore, when sorting the output candidate words in the candidate word list, the bitonal sequence sorting operation can be directly performed without first sorting the output candidate words in the candidate word list into a bitonal sequence through a sorting operation, thereby improving the sorting efficiency.

[0152] Operation 3, arrange the output candidate words in the candidate word list in a two-tone ascending order or a two-tone descending order, and continue to perform operation 1 until all the output candidate words in the group of output candidate words are involved in the sorting, and select the output candidate words with the top L output probabilities from the candidate word list as the output results of the group of output candidate words.

[0153] In specific implementation, when the output candidate words in the candidate word list are sorted by bitonal sorting, bitonal ascending sorting can be adopted, or bitonal descending sorting can be adopted. As mentioned in operation 2, when the output candidate words in the candidate word list are sorted by bitonal sorting, if bitonal ascending sorting is adopted, then in operation 2, the first L output candidate words obtained in operation 1 are written into the first L positions of the candidate word list. If bitonal descending sorting is adopted, then in operation 2, the first L output candidate words obtained in operation 1 are written into the last L positions of the candidate word list.

[0154] In specific implementation, after selecting the top M output candidate words with output probabilities from the sorted results of the output candidate words corresponding to each problem to be inferred, corresponding sampling strategies can be used based on multiple sampling parameter matrices to sample in sequence among the top M output candidate words with output probabilities corresponding to each problem to be inferred, so as to determine the output candidate words for each problem to be inferred.

[0155] Specifically, when sampling is performed in sequence among the top M output candidate words with the output probability corresponding to each problem to be inferred using the corresponding sampling strategies, batch processing can be performed based on each sampling parameter matrix. Specifically, Top-K sampling can be performed in the candidate word matrix based on the Top-K sampling parameter matrix, and then temperature sampling can be performed in the Top-K sampling result based on the temperature sampling parameter matrix. Finally, Top-P sampling can be performed in the temperature sampling result based on the Top-P sampling parameter matrix and the random sampling parameter matrix.

[0156] Specifically, since the top M output candidate words with the highest output probability have been selected from the sorted results of the output candidate words corresponding to each problem to be inferred, and the M output candidate words are arranged in descending order, when Top-K sampling is performed among the M output candidate words corresponding to each problem to be inferred based on the Top-K sampling parameter matrix, the top K output candidate words can be directly selected from the M output candidate words corresponding to each problem to be inferred.

[0157] In specific implementation, after Top-K sampling, the output probability of the output candidate words that exceed the sampling parameter K (K+1 to M) in the M output candidate words corresponding to each problem to be inferred can be set to negative infinity (-inf), and then based on the temperature sampling parameter matrix, the output probabilities of the K output candidate words in the Top-K sampling results of each problem to be inferred are respectively adjusted by temperature, and the Softmax algorithm in the related technology is used to normalize the output probabilities of the K output candidate words in the Top-K sampling results of each problem to be inferred. Among them, the maximum output probability required to be used in the calculation process of the Softmax algorithm can directly use the output probability of the first output candidate word in the M output candidate words, without repeating the sorting operation.

[0158] Through the Softmax algorithm, the output probabilities of the K output candidate words in the Top-K sampling results of each problem to be inferred are recalculated respectively, and the K output candidate words are still arranged in descending order, and the cumulative sum of the output probabilities of the K output candidate words in the Top-K sampling results of each problem to be inferred is 1. In this case, when Top-P sampling is performed based on the Top-P sampling parameter matrix and the random sampling parameter matrix, there is no need to reorder the K output candidate words, which further improves the post-processing speed of the large model.

[0159] Specifically, when Top-P sampling is performed based on the Top-P sampling parameter matrix and the random sampling parameter matrix, the cumulative probability is set for each problem to be inferred, and among the K output candidate words corresponding to each problem to be inferred, each output candidate word is traversed from the first output candidate word, and the cumulative probability corresponding to each output candidate word is calculated in turn, and the output candidate word whose cumulative probability is greater than or equal to the target probability for the first time is used as the output candidate word for each problem to be inferred, where the target probability is the product of the Top-P sampling parameter value P of each problem to be inferred and the random sampling parameter value.

[0160] Of course, in other embodiments of the present application, other sampling orders may also be used, such as first performing temperature sampling in the candidate word matrix based on the temperature sampling parameter matrix, then performing Top-K sampling in the temperature sampling results based on the Top-K sampling parameter matrix, and finally performing Top-P sampling in the temperature sampling results based on the Top-P sampling parameter matrix and the random sampling parameter matrix.

[0161] In practical applications, based on multiple sampling parameter matrices, when corresponding sampling strategies are used to perform sampling in the candidate word matrix in sequence, the sampling strategy implemented later can reuse the calculation results of the sampling strategy implemented earlier. For example, Top-P sampling can reuse the output candidate word ranking results of Top-K sampling without re-sorting, thereby improving the utilization rate of intermediate calculation results, avoiding repeated sorting or repeated calculation of the maximum value, and improving the efficiency and speed of post-processing of large models.

[0162] S303: Writing the output candidate words of each question to be inferred into the host.

[0163] In specific implementation, after the device determines the output candidate words for each problem to be inferred, the output candidate words for each problem to be inferred can be written into the host to complete the output candidate word sampling in the post-processing process of the large model.

[0164] It should be noted that the embodiment of the present application is explained by taking the candidate word matrix including output candidate words as an example. In actual applications, the candidate word matrix may also include the identifier of the output candidate words without including the real output candidate words. Accordingly, the video memory device finally determines the output candidate word identifier of each reasoning problem, and writes the output candidate word identifier of each reasoning problem into the host.

[0165] Combine the following Figure 4 , taking the data interaction process between the host and the device in the server as an example, the specific implementation process of the post-processing method of the large model provided in the embodiment of the present application is described in detail. Figure 4 As shown, the specific implementation process of the post-processing method of the large model provided in the embodiment of the present application includes:

[0166] Step 401: The host obtains the problem to be inferred and the sampling parameter values ​​of each problem to be inferred through the client.

[0167] In step 402, the host preprocesses the sampling parameters of each problem to be inferred, and combines the preprocessed sampling parameter values ​​to obtain multiple sampling parameter matrices.

[0168] Step 403: The host determines the output candidate words for each question to be inferred and generates a candidate word matrix.

[0169] Step 404: The device reads a candidate word matrix and a plurality of sampling parameter matrices from the host.

[0170] In step 405, the device uses corresponding sampling strategies to sample in the candidate word matrix in sequence based on multiple sampling parameter matrices to determine the output candidate words for each problem to be inferred.

[0171] Step 406: The device writes the output candidate words of each question to be inferred into the host.

[0172] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of a large-scale post-processing device, such as Figure 5 As shown, the post-processing device 500 of the large model includes:

[0173] A receiving unit 501 is used to obtain a candidate word matrix and multiple sampling parameter matrices from a host, wherein the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy, and the multiple sampling parameter matrices are obtained by combining the sampling parameter values ​​of each problem to be inferred after preprocessing;

[0174] A processing unit 502 is configured to sequentially sample in the candidate word matrix using corresponding sampling strategies based on the multiple sampling parameter matrices to determine output candidate words for each of the questions to be inferred;

[0175] The sending unit 503 is used to write the output candidate words of each question to be inferred into the host.

[0176] Optionally, the multiple sampling parameter matrices are obtained in the following manner:

[0177] The sampling parameter values ​​of the problems to be inferred are obtained, the sampling parameter values ​​are preprocessed, and the sampling parameter values ​​belonging to the same sampling strategy among the sampling parameter values ​​of the problems to be inferred are combined into a sampling parameter matrix to obtain multiple sampling parameter matrices.

[0178] Optionally, the preprocessing includes one or more of the following:

[0179] Set missing sampling parameter values ​​to pre-configured default values;

[0180] Adding a random sampling parameter for adjusting the range of output candidate words, and setting the sampling parameter value of the random sampling parameter to a preset value or a random value;

[0181] The sampling parameter values ​​that are not within the pre-configured value interval are adjusted to be within the pre-configured value interval.

[0182] Optionally, the processing unit 502 is specifically configured to:

[0183] By at least one computing unit, the output candidate words of each problem to be inferred are sorted according to the output probability of each output candidate word in the candidate word matrix to obtain a sorting result, and the output candidate words with the top M output probabilities are selected from the sorting results of the output candidate words corresponding to each problem to be inferred, where M is a positive integer;

[0184] Based on the multiple sampling parameter matrices, corresponding sampling strategies are used to sequentially sample the top M output candidate words with output probabilities corresponding to the questions to be inferred, so as to determine the output candidate words for the questions to be inferred.

[0185] Optionally, the processing unit 502 is specifically configured to:

[0186] For the output candidate words of each of the problems to be inferred, the following operations are performed respectively:

[0187] Divide the output candidate words into N groups, where N is a positive integer less than or equal to the number of the computing units;

[0188] By using a plurality of computing units, the output probabilities of N groups of output candidate words are sorted respectively to obtain N output results, each of which includes the output candidate words with the top L output probabilities in a group of output candidate words, where L is a positive integer and the product of L and N is greater than or equal to M;

[0189] Each computing unit is controlled to obtain the output results of other computing units, and its own output results and the output results of other computing units are fully sorted to obtain a fully sorted result, and the output candidate words with the top M output probabilities are selected from the fully sorted result.

[0190] Optionally, the processing unit 502 is specifically configured to:

[0191] When multiple computing units do not have shared memory, control each computing unit to store its own output results into a temporary cache space;

[0192] Control each computing unit to obtain the output results of other computing units from the temporary cache space.

[0193] Optionally, the processing unit 502 is specifically configured to:

[0194] For a group of output candidate words, the following operations are performed respectively until all the output candidate words in the group of output candidate words are sorted, and the output candidate words with the top L output probabilities are selected from the candidate word list as the output results of the group of output candidate words:

[0195] Select a preset number of output candidate words from the group of output candidate words, perform full sorting on the output probabilities of the selected output candidate words to obtain an intermediate sorting result, and select the output candidate words with the top L output probabilities from the intermediate sorting result;

[0196] Writing the first L output candidate words into the first L positions or the last L positions in a pre-configured candidate word list, wherein the candidate word list includes 2L positions, wherein when not writing for the first time, the output probability arrangement order of the output candidate words written into the candidate word list is opposite to the output probability arrangement order of the output candidate words in the candidate word list;

[0197] The output candidate words in the candidate word list are arranged in a two-tone ascending order or a two-tone descending order, and output candidate words are continuously selected from the group of output candidate words for sorting.

[0198] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of another large model post-processing device, such as Figure 6 As shown, the post-processing device 600 of the large model includes:

[0199] An acquisition unit 601 is used to acquire a candidate word matrix and sampling parameter values ​​of each question to be inferred;

[0200] A processing unit 602 is used to preprocess the sampling parameter values ​​and combine the preprocessed sampling parameter values ​​to obtain multiple sampling parameter matrices, where the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy;

[0201] A sending unit 603 is used to write the candidate word matrix and the multiple sampling parameter matrices into a device;

[0202] The receiving unit 604 is used to read the output candidate words of each reasoning problem determined in the device, and the output candidate words of each reasoning problem are determined by the device based on the multiple sampling parameter matrices, using corresponding sampling strategies to sample in the candidate word matrix in sequence.

[0203] Optionally, the processing unit 602 is specifically configured to:

[0204] The sampling parameter values ​​are preprocessed, and the sampling parameter values ​​belonging to the same sampling strategy among the sampling parameter values ​​of the problems to be inferred are combined into a sampling parameter matrix to obtain a plurality of sampling parameter matrices.

[0205] Optionally, the preprocessing includes one or more of the following:

[0206] Set missing sampling parameter values ​​to pre-configured default values;

[0207] Adding a random sampling parameter for adjusting the range of output candidate words, and setting the sampling parameter value of the random sampling parameter to a preset value or a random value;

[0208] The sampling parameter values ​​that are not within the pre-configured value interval are adjusted to be within the pre-configured value interval.

[0209] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 7 As shown, it includes at least one processor chip 701 and a memory 702 connected to the at least one processor chip. The specific connection medium between the processor chip 701 and the memory 702 is not limited in the embodiment of the present application. Figure 7 For example, the processor chip 701 and the memory 702 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0210] In the embodiment of the present application, the memory 702 stores instructions that can be executed by at least one processor chip 701. The at least one processor chip 701 can execute the steps of the above-mentioned large model post-processing method by executing the instructions stored in the memory 702.

[0211] The processor chip 701 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, and realize data processing by running or executing instructions stored in the memory 702 and calling data stored in the memory 702. Optionally, the processor chip 701 may include one or more processing units, and the processor chip 701 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor chip 701. In some embodiments, the processor chip 701 and the memory 702 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0212] The processor chip 701 can be a general-purpose processor, such as a GPU, a GPGPU, a TPU, a CPU, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0213] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 702 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 702 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 702 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0214] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned large model post-processing method.

[0215] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned large model post-processing method.

[0216] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0217] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0218] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0219] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0220] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0221] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A post-processing method for a large model, characterized in that: include: Acquire a candidate word matrix and multiple sampling parameter matrices from a host, wherein the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy, and the multiple sampling parameter matrices are obtained by combining the sampling parameter values ​​of each problem to be inferred after preprocessing; Based on the multiple sampling parameter matrices, respectively use corresponding sampling strategies to perform sampling in the candidate word matrix in sequence to determine the output candidate words of each problem to be inferred; The output candidate words of each of the questions to be inferred are written into the host.

2. The method according to claim 1, characterized in that The multiple sampling parameter matrices are obtained in the following manner: The sampling parameter values ​​of the problems to be inferred are obtained, the sampling parameter values ​​are preprocessed, and the sampling parameter values ​​belonging to the same sampling strategy among the sampling parameter values ​​of the problems to be inferred are combined into a sampling parameter matrix to obtain multiple sampling parameter matrices.

3. The method according to claim 2, characterized in that The pretreatment includes one or more of the following: Set missing sampling parameter values ​​to pre-configured default values; Adding a random sampling parameter for adjusting the range of output candidate words, and setting the sampling parameter value of the random sampling parameter to a preset value or a random value; The sampling parameter values ​​that are not within the pre-configured value interval are adjusted to be within the pre-configured value interval.

4. The method according to claim 1, characterized in that: The method of sampling in the candidate word matrix in sequence using corresponding sampling strategies based on the multiple sampling parameter matrices to determine the output candidate words of each problem to be inferred includes: By at least one computing unit, the output candidate words of each problem to be inferred are sorted according to the output probability of each output candidate word in the candidate word matrix to obtain a sorting result, and the output candidate words with the top M output probabilities are selected from the sorting results of the output candidate words corresponding to each problem to be inferred, where M is a positive integer; Based on the multiple sampling parameter matrices, corresponding sampling strategies are used to sequentially sample the top M output candidate words with output probabilities corresponding to the questions to be inferred, so as to determine the output candidate words for the questions to be inferred.

5. The method according to claim 4, characterized in that The method of sorting the output candidate words of each problem to be inferred respectively according to the output probability of each output candidate word in the candidate word matrix by at least one computing unit to obtain a sorting result, and selecting the output candidate words with the top M output probabilities from the sorting results of the output candidate words corresponding to each problem to be inferred, comprises: For the output candidate words of each of the problems to be inferred, the following operations are performed respectively: Divide the output candidate words into N groups, where N is a positive integer less than or equal to the number of the computing units; By using a plurality of computing units, the output probabilities of N groups of output candidate words are sorted respectively to obtain N output results, each of which includes the output candidate words with the top L output probabilities in a group of output candidate words, where L is a positive integer and the product of L and N is greater than or equal to M; Each computing unit is controlled to obtain the output results of other computing units, and its own output results and the output results of other computing units are fully sorted to obtain a fully sorted result, and the output candidate words with the top M output probabilities are selected from the fully sorted result.

6. The method according to claim 5, characterized in that The controlling each computing unit to obtain the output results of other computing units includes: In the case where the multiple computing units do not have a shared memory, controlling each computing unit to store its own output result into a pre-configured temporary cache space; Each computing unit is controlled to obtain output results of other computing units from the temporary cache space.

7. The method according to claim 5, characterized in that The method uses a plurality of computing units to sort the output probabilities of N groups of output candidate words to obtain N output results, including: For a group of output candidate words, the following operations are performed respectively until all the output candidate words in the group of output candidate words are sorted, and the output candidate words with the top L output probabilities are selected from the candidate word list as the output results of the group of output candidate words: Selecting a preset number of output candidate words from the group of output candidate words, performing full sorting on the output probabilities of the selected output candidate words to obtain an intermediate sorting result, and selecting the output candidate words with the top L output probabilities from the intermediate sorting result; Writing the first L output candidate words into the first L positions or the last L positions in a pre-configured candidate word list, the candidate word list comprising 2L positions, wherein when not writing for the first time, the output probability arrangement order of the output candidate words written into the candidate word list is opposite to the output probability arrangement order of the output candidate words in the candidate word list; The output candidate words in the candidate word list are arranged in a two-tone ascending order or a two-tone descending order, and output candidate words are continuously selected from the group of output candidate words for sorting.

8. A post-processing method for a large model, characterized in that: The method comprises: Obtain the candidate word matrix and the sampling parameter values ​​of each question to be inferred; Preprocessing the sampling parameter values, and combining the preprocessed sampling parameter values ​​to obtain multiple sampling parameter matrices, where the sampling parameter values ​​in each sampling parameter matrix are sampling parameter values ​​of each problem to be inferred under the same sampling strategy; Writing the candidate word matrix and the plurality of sampling parameter matrices into a device; The output candidate words of each reasoning problem determined in the device are read, wherein the output candidate words of each reasoning problem are determined by the device based on the multiple sampling parameter matrices, respectively, by sampling in the candidate word matrix in sequence using corresponding sampling strategies.

9. The method according to claim 8, characterized in that The preprocessing of the sampling parameter values ​​and combining the preprocessed sampling parameter values ​​to obtain a plurality of sampling parameter matrices include: The sampling parameter values ​​are preprocessed, and the sampling parameter values ​​belonging to the same sampling strategy among the sampling parameter values ​​of the problems to be inferred are combined into a sampling parameter matrix to obtain a plurality of sampling parameter matrices.

10. The method according to claim 9, characterized in that The pretreatment includes one or more of the following: Set missing sampling parameter values ​​to pre-configured default values; Adding a random sampling parameter for adjusting the range of output candidate words, and setting the sampling parameter value of the random sampling parameter to a preset value or a random value; The sampling parameter values ​​that are not within the pre-configured value interval are adjusted to be within the pre-configured value interval.

11. A computer device comprising a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, characterized in that: When the processor chip executes the program, the steps of any one of the methods of claims 1 to 10 are implemented.

Images