Data processing method and device and storage medium

By sorting and selecting the first J output candidate words in descending order to generate the second candidate word matrix, the problem of high computational overhead in the sampling post-processing process of large pre-trained language models is solved, and the inference speed is improved.

CN118917415APending Publication Date: 2024-11-08SHANGHAI BIREN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410918385.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When a large pretrained language model uses joint sampling strategies during post-sampling processing, the calculation overhead is high, resulting in slow inference speed.

Method used

By obtaining the first candidate word matrix to be sampled and sorting it in descending order according to the output probability, selecting the first J output candidate words to generate the second candidate word matrix, reducing the matrix dimension, thereby reducing the calculation amount and overhead.

Benefits of technology

Improves inference speed and reduces the computational cost and overhead of post-processing operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917415A_ABST
    Figure CN118917415A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data processing method and device and a storage medium, and relates to the technical field of large models.The method comprises the steps that a to-be-sampled first candidate word matrix and sampling parameters set for all to-be-reasoned problems are obtained; a to-be-sampled first candidate word matrix and sampling parameters set for all to-be-reasoned problems are obtained, the dimension of the first candidate word matrix is N * M, and the sampling parameters at least comprise a parameter K when Top-K sampling is carried out on output candidate words; in the first candidate word matrix, according to the output probability, the output candidate words of each to-be-reasoned problem are arranged in a descending order, the first J output candidate words are selected from the sorting result of the output candidate words of each to-be-reasoned problem, and a second candidate word matrix is generated, j is determined based on a parameter K corresponding to the to-be-reasoned problem, the difference value between M and J is greater than a preset threshold value, and J is a positive integer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of large model technology, and in particular to a data processing method, device and storage medium. Background Art

[0002] Large model technology, especially large pre-trained language models, has become an important direction in the field of artificial intelligence. The reasoning performance of large models is an area of ​​great concern in current research.

[0003] Post-processing of large model sampling is an important part of large model reasoning. The reasoning process usually uses different sampling strategies to balance the diversity and accuracy of the output. Among them, the joint sampling strategy is one of the most commonly used sampling strategies, that is, the joint use of Top-K sampling, Top-P sampling and temperature sampling.

[0004] When the joint sampling strategy is actually used for sampling, the data dimension of the post-sampling processing operation of the large model can usually reach a mathematical matrix size of tens of thousands to hundreds of thousands. Each step of the calculation must process a large amount of data, which has a high computational overhead and leads to a slow inference speed. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, device and storage medium for reducing the computational overhead of the reasoning process and improving the reasoning speed.

[0006] On the one hand, an embodiment of the present application provides a data processing method, including:

[0007] Obtain a first candidate word matrix to be sampled and sampling parameters set for each problem to be inferred, wherein the first candidate word matrix contains output candidate words and corresponding output probabilities of each problem to be inferred, the dimension of the first candidate word matrix is ​​N×M, N is the number of problems to be inferred, M is the number of output candidate words for each problem to be inferred, and the sampling parameters at least include: parameters K when performing Top-K sampling on the output candidate words, M, N and K are all positive integers;

[0008] In the first candidate word matrix, the output candidate words of each question to be inferred are arranged in descending order according to the output probability, and the first J output candidate words are selected from the output candidate word sorting results of each question to be inferred to generate a second candidate word matrix, wherein the dimension of the second candidate word matrix is ​​N×J, and J is determined based on the parameter K corresponding to the question to be inferred, and the difference between M and J is greater than a preset threshold, and J is a positive integer.

[0009] On the one hand, an embodiment of the present application provides a data processing device, including:

[0010] An acquisition module is used to acquire a first candidate word matrix to be sampled and sampling parameters set for each problem to be inferred, wherein the first candidate word matrix contains output candidate words and corresponding output probabilities of each problem to be inferred, the dimension of the first candidate word matrix is ​​N×M, N is the number of problems to be inferred, M is the number of output candidate words for each problem to be inferred, and the sampling parameters at least include: a parameter K when performing Top-K sampling on the output candidate words, M, N and K are all positive integers;

[0011] A processing module is used to arrange the output candidate words of each problem to be inferred in descending order according to the output probability in the first candidate word matrix, and select the first J output candidate words from the output candidate word sorting results of each problem to be inferred to generate a second candidate word matrix, wherein the dimension of the second candidate word matrix is ​​N×J, J is determined based on the parameter K corresponding to the problem to be inferred, and the difference between M and J is greater than a preset threshold, and J is a positive integer.

[0012] Optionally, the device also includes: a sampling module, used to sample in the second candidate word matrix, determine the target candidate word for each question to be inferred, and randomly select a candidate word from the target candidate words as an output for the question to be inferred.

[0013] Optionally, there are multiple questions to be inferred, and the sampling parameters further include: a parameter P when performing Top-P sampling on the output candidate words, where P is greater than or equal to 0 and less than or equal to 1;

[0014] The sampling module is specifically used for:

[0015] In the second candidate word matrix, for each question to be inferred, the following operations are performed to obtain a third candidate word matrix: a parameter K corresponding to a question to be inferred is determined, and in the output candidate word ranking result corresponding to the question to be inferred, the output probabilities of the first K output candidate words are retained, and the output probabilities of the remaining JK output candidate words are adjusted to preset probabilities;

[0016] In the third candidate word matrix, normalizing the output probability of each output candidate word of the problem to be inferred is performed to obtain the normalized output probability of each output candidate word, thereby obtaining a fourth candidate word matrix;

[0017] In the fourth candidate word matrix, Top-P sampling is performed according to the ranking of the output candidate words of each question to be inferred and the parameter P corresponding to each question to be inferred, so as to determine the target candidate word of each question to be inferred.

[0018] Optionally, the sampling module is specifically used to:

[0019] Based on the output probabilities corresponding to the output candidate words in the fourth candidate word matrix and the ranking of the output candidate words of each question to be inferred, the cumulative probabilities corresponding to each output candidate word set are calculated to construct a probability matrix, wherein the cumulative probability corresponding to the i-th output candidate word set in the probability matrix is ​​the sum of the output probabilities of the first i output candidate words in the ranking of the output candidate words corresponding to each question to be inferred, where i is a positive integer;

[0020] For each question to be inferred, the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P are determined as the target candidate words.

[0021] Optionally, the sampling module is specifically used to:

[0022] In the probability matrix, for each output candidate word of the problem to be inferred, the following operations are performed respectively: determine the target cumulative probability whose cumulative probability is greater than or equal to the parameter P for the first time, adjust the cumulative probability greater than the target cumulative probability to the preset probability, and in the adjusted probability matrix, determine the output candidate word included in the output candidate word set corresponding to the maximum cumulative probability as the target candidate word.

[0023] Optionally, there are multiple problems to be inferred, and the sampling parameters further include: a parameter P when performing Top-P sampling on the output candidate words;

[0024] The sampling module is specifically used for:

[0025] For each problem to be inferred, perform the following operations in sequence:

[0026] Determine a parameter K corresponding to a question to be inferred, and take out the first K output candidate words from the sorted output candidate words corresponding to the question to be inferred in the second candidate word matrix;

[0027] Top-P sampling is performed based on the ranking of the first K output candidate words, the output probability corresponding to each output candidate word, and the parameter P corresponding to the one to-be-reasoned question to determine the target candidate word for the one to-be-reasoned question.

[0028] Optionally, J is determined in the following manner:

[0029] When there is one problem to be inferred, J is greater than or equal to a parameter K corresponding to the problem to be inferred; or

[0030] When there are multiple problems to be inferred, J is greater than or equal to the maximum value of the parameters K corresponding to the multiple problems to be inferred.

[0031] Optionally, the sampling parameters further include: a temperature parameter when performing temperature sampling on the output candidate words;

[0032] The sampling module is specifically used for:

[0033] The temperature parameter is used to adjust the output probability of each output candidate word in the second candidate word matrix, obtain the output probability of each output candidate word after temperature adjustment, obtain a new second candidate word matrix, and perform sampling in the new second candidate word matrix.

[0034] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, wherein the processor chip implements the steps of the above-mentioned data processing method when executing the program.

[0035] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned data processing method.

[0036] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned data processing method.

[0037] In an embodiment of the present application, the first candidate word matrix includes output candidate words and corresponding output probabilities for each problem to be inferred. Since the output candidate words for each problem to be inferred are usually pre-trained large models, the number of output candidate words may be tens of thousands or even hundreds of thousands based on the reasoning of the problem to be inferred. The number is recorded as M. After obtaining the first candidate word matrix to be sampled and the sampling parameters set for each problem to be inferred, the output candidate words for each problem to be inferred are first arranged in descending order according to the output probability of each output candidate word in the first candidate word matrix, and then the number J is determined based on the parameter K corresponding to each problem to be inferred. From the sorting results corresponding to each problem to be inferred, the first J output candidate words are selected to generate a second candidate word matrix. Since the difference between M and J is greater than the preset threshold, the preset threshold can make M much greater than J, that is, the matrix dimension of the second candidate word matrix is ​​greatly reduced compared with the first candidate word matrix.

[0038] The embodiment of the present application reduces the dimension of the first candidate word matrix so that during post-processing operations, compared with post-processing operations based on the first candidate word matrix, each step of calculation has a matrix scale of tens of thousands to hundreds of thousands. Post-processing operations are performed based on the second candidate word matrix, which can reduce the amount of calculation and computing overhead in the post-processing operations and the reasoning process, thereby improving the reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0041] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0042] Figure 3A A schematic diagram of a second candidate word matrix provided in an embodiment of the present application;

[0043] Figure 3B A schematic diagram of another second candidate word matrix provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a third candidate word matrix provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a probability matrix provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of another probability matrix provided in an embodiment of the present application;

[0047] Figure 7 A flowchart of a specific implementation process of a data processing method provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0051] It is understandable that in the following specific implementation methods of this application, problems and sampling parameters are involved. When the various embodiments of this application are applied to specific products or technologies, relevant licenses or consents need to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0052] The following is an introduction to some concepts involved in the embodiments of the present application.

[0053] 1. Token, or word unit, usually refers to a discrete text unit in a large language model. It can be a word, punctuation mark, number or other language elements, which are used as the basic unit for training and generating text.

[0054] 2. Top-K sampling is an optimization of greedy sampling (selecting the token with the highest probability from all tokens). Its parameter K represents the number of tokens to be selected. K can be set by the user and is usually a positive integer. When K=1, Top-K sampling is greedy sampling.

[0055] When sampling, the first K tokens are selected from the token probability descending order, and a random selection is made from the K tokens as the output. Although this sampling method is simple and efficient, it may lead to insufficient diversity in the generated text.

[0056] 3. Top-P sampling, which selects tokens by percentage from the token probability descending sort results. Its parameter P is equivalent to a cumulative probability threshold. P is greater than or equal to 0 and less than or equal to 1. P is usually set to a relatively high value.

[0057] When sampling, we start from the token with the highest probability in descending order, calculate the cumulative probability of the first L tokens, and then randomly select a token as output from the minimum set whose cumulative probability is greater than or equal to probability P (that is, the set including the first L tokens when the first L tokens are greater than or equal to P for the first time), where L is a natural number. This sampling strategy can increase the diversity of the text while maintaining a certain degree of accuracy.

[0058] For example, assuming that token1 is "girl" with a probability of 66%, token2 is "shoes" with a probability of 20%, token3 is "elephant" with a probability of 10%, and token4 is "watermelon" with a probability of 4%. If the parameter P of Top-P sampling is set to 0.85 (85%), then the cumulative probability corresponding to token1 is 66%, which is less than P. The cumulative probability of the set composed of token1 and token2 is 86%, which is greater than P for the first time. Then the sampling results of Top-P are token1 and token2.

[0059] 4. Temperature sampling, by adjusting the temperature parameter of the probability distribution, to control the randomness of the generated text. A higher temperature value will increase the sampling chance of low-probability tokens, thereby improving the diversity of the text.

[0060] 5. Joint sampling. In practical applications, Top-K sampling, Top-P sampling and temperature sampling are often used in combination to achieve a more balanced text generation effect.

[0061] The following is a brief introduction to the design concept of the embodiment of the present application:

[0062] Large model technology, especially large pre-trained language models, has become an important direction in the field of artificial intelligence. The reasoning performance of large models is an area of ​​great concern in current research.

[0063] Post-processing of large model sampling is an important part of large model reasoning. The reasoning process usually uses different sampling strategies to balance the diversity and accuracy of the output. Among them, the joint sampling strategy is one of the most commonly used sampling strategies, that is, the joint use of Top-K sampling, Top-P sampling and temperature sampling.

[0064] When the joint sampling strategy is actually used for sampling, the data dimension of the post-sampling processing operation of the large model can usually reach a mathematical matrix size of tens of thousands to hundreds of thousands. Each step of the calculation must process a large amount of data, which has a high computational overhead and leads to a slow inference speed.

[0065] In view of this, an embodiment of the present application provides a data processing method, device and storage medium, wherein a first candidate word matrix includes output candidate words and corresponding output probabilities for each problem to be inferred. Since the output candidate words for each problem to be inferred are usually pre-trained large models, the number of output candidate words may be tens of thousands or even hundreds of thousands based on the reasoning of the problem to be inferred, and the number is recorded as M. After obtaining the first candidate word matrix to be sampled and the sampling parameters set for each problem to be inferred, the output candidate words for each problem to be inferred are first arranged in descending order according to the output probability of each output candidate word in the first candidate word matrix, and then the number J is determined based on the parameter K corresponding to each problem to be inferred. From the sorting results corresponding to each problem to be inferred, the first J output candidate words are selected to generate a second candidate word matrix. Since the difference between M and J is greater than a preset threshold, the preset threshold can make M much greater than J, that is, the matrix dimension of the second candidate word matrix is ​​greatly reduced compared with the first candidate word matrix.

[0066] The embodiment of the present application reduces the dimension of the first candidate word matrix so that during post-processing operations, compared with post-processing operations based on the first candidate word matrix, each step of calculation has a matrix scale of tens of thousands to hundreds of thousands. Post-processing operations are performed based on the second candidate word matrix, which can reduce the amount of calculation and computing overhead in the post-processing operations and the reasoning process, thereby improving the reasoning speed.

[0067] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0068] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes any terminal device 110 among multiple terminal devices and any server 120 among multiple servers.

[0069] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers and other devices; a large model reasoning client can be installed on the terminal device, which can receive questions raised by users, perform reasoning, and give corresponding replies. The client can be software, or a web page, a small program, etc. The server 120 is the background service end of the large model reasoning client. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0070] It should be noted that the data processing method in the embodiment of the present application can be executed by the server 120. After the server 120 obtains the problem to be inferred input by the user through the terminal device 110 and the parameter K corresponding to each problem to be inferred during Top-K sampling in the post-sampling processing process, it first uses the pre-trained large model to infer each problem to be inferred, obtains the output candidate words and corresponding output probabilities of each problem to be inferred, and generates a first candidate word matrix. Then, according to the output probability of each output candidate word in the first candidate word matrix, the output candidate words of each problem to be inferred are arranged in descending order, and the number J is determined based on the parameter K corresponding to each problem to be inferred. From the sorting results corresponding to each problem to be inferred, the first J output candidate words are selected to generate a second candidate word matrix. Finally, sampling is performed in the second candidate word matrix to determine the target candidate word for each problem to be inferred, and a candidate word is randomly selected from the target candidate word as the output for the problem to be inferred, and displayed to the user through the terminal device 110.

[0071] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network, which may be a wired network or a wireless network.

[0072] It should be noted that Figure 1 What is shown is just an example. In fact, the number and communication method of terminal devices and servers are not limited. When there are multiple servers, multiple servers can form a blockchain, and the server is a node on the blockchain, which is not specifically limited in the embodiments of the present application.

[0073] The following describes the data processing method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0074] See also Figure 2 As shown, it is an implementation flow chart of a data processing method in an embodiment of the present application, with the server as the execution subject, and the specific implementation process of the method is as follows S201-S202:

[0075] S201: Obtain the first candidate word matrix to be sampled and the sampling parameters set for each problem to be inferred, wherein the first candidate word matrix contains the output candidate words of each problem to be inferred and the corresponding output probabilities, the dimension of the first candidate word matrix is ​​N×M, N is the number of problems to be inferred, M is the number of output candidate words for each problem to be inferred, and the sampling parameters include at least: the parameter K when performing Top-K sampling on the output candidate words, M, N and K are all positive integers.

[0076] It should be noted that the first candidate word matrix is ​​usually generated by a pre-trained large model (or a large language model) based on the reasoning of the problem to be reasoned, which includes the output candidate words and the corresponding output probabilities of each problem to be reasoned. The training process of the specific large model and the process of determining the output candidate words by reasoning of the large model are not specifically limited in the embodiments of the present application. The output candidate words mentioned in the embodiments of the present application can be words, punctuation marks, numbers or other language elements, which are not limited in the embodiments of the present application.

[0077] In specific implementation, the dimension of the first candidate word is determined by the number of questions to be reasoned and the number of output candidate words for each question to be reasoned. For example, for a reasoning process, if there are 3 questions to be reasoned and each question to be reasoned has 90,000 output candidate words, then the dimension of the first candidate word matrix is ​​3×90,000.

[0078] Among them, the problem to be inferred and the sampling parameters set for each problem to be inferred can be input by the user or generated in batches by other artificial intelligence tools, which is not limited in this embodiment of the present application.

[0079] Specifically, taking user input as an example, the user can input one question to be inferred at a time. For example, the user inputs a question to be inferred through a terminal device, "What do rabbits like to eat?", and sets the parameter K for Top-K sampling to 40. The user can also input multiple questions to be inferred at a time. For example, the user inputs three questions to be inferred through a terminal device, "What do rabbits like to eat?", "What fish are there in the ocean?", and "What is the smallest bird in the world?", and sets the sampling parameters of the three questions to be inferred through an array. For example, for the parameter K during Top-K sampling, it is set through the array [40, 50, 60], which means that the parameter K during Top-K sampling corresponding to the first question to be inferred ("What do rabbits like to eat?") is 40, the parameter K during Top-K sampling corresponding to the second question to be inferred ("What fish are there in the ocean?") is 50, and the parameter K during Top-K sampling corresponding to the third question to be inferred ("What is the smallest bird in the world?") is 60.

[0080] Of course, when the sampling parameters also include other sampling parameters, the user can also input other sampling parameters when inputting the problem to be inferred. Still taking the example of the user inputting three problems to be inferred in one input, assuming that in addition to Top-K sampling, Top-P sampling is also required, the parameters P for Top-P sampling of the three problems to be inferred can be set respectively through the array [0.9, 0.85, 0.95]. If temperature sampling is also required, the temperature parameters for temperature sampling of the three problems to be inferred can be set respectively through the array [0.9, 1.2, 1.5].

[0081] S202: In the first candidate word matrix, the output candidate words of each question to be inferred are arranged in descending order according to the output probability, and the first J output candidate words are selected from the sorted results of the output candidate words of each question to be inferred to generate a second candidate word matrix, wherein the dimension of the second candidate word matrix is ​​N×J, J is determined based on the parameter K corresponding to the question to be inferred, and the difference between M and J is greater than a preset threshold, and J is a positive integer.

[0082] It should be noted that the preset threshold can be set according to the number of candidate words in the first candidate word matrix and experience. In some possible implementations, the preset threshold can be set to a specific value less than the number of candidate words M. Specifically, taking the above example of the present application as an example, M is 90000, if J is a natural number between 1-X, then the preset threshold can be set to MX, for example, if J is a natural number between 1-100, then the preset threshold can be set to 89900.

[0083] During specific implementation, in the first candidate word matrix, when the output candidate words of each problem to be reasoned are arranged in descending order according to the output probability, the output candidate words of each problem to be reasoned are sorted separately. For example, if there are two problems to be reasoned and each problem to be reasoned has 90,000 output candidate words, then for the 90,000 output candidate words of the first problem to be reasoned, the output candidate words are arranged in descending order according to the output probability of each output candidate word, and for the 90,000 output candidate words of the second problem to be reasoned, the output candidate words are also arranged in descending order according to the output probability of each output candidate word.

[0084] After arranging the output candidate words of each problem to be reasoned in descending order, select the first J output candidate words from the sorted results of the output candidate words of each problem to be reasoned, and retain the selected first J output candidate words to obtain a second candidate word matrix. In other words, in the second candidate word matrix, there are J output candidate words for each problem to be reasoned.

[0085] In specific implementation, since the number of retained output candidate words needs to meet the needs of Top-K sampling, Top-K sampling requires at least K output candidate words. Therefore, in the first candidate word matrix, the first J output candidate words are selected to generate the second candidate word matrix. If there is one problem to be reasoned, J needs to be greater than or equal to the parameter K corresponding to the problem to be reasoned; if there are multiple problems to be reasoned, J needs to be greater than or equal to the maximum value of the parameters K corresponding to the multiple problems to be reasoned.

[0086] It should be noted that, in the case where there are multiple problems to be inferred, if J is determined to be greater than or equal to the maximum value of the parameters K corresponding to the multiple problems to be inferred, it can be ensured that the J output candidate words selected for each problem to be inferred include the K candidate words required to be selected when performing Top-K sampling on the output candidate words of the problem to be inferred.

[0087] For example, if Figure 3A As shown in the figure, if there are three questions to be inferred, in the first candidate word matrix, there are 90,000 output candidate words for each question to be inferred, and the parameters K corresponding to the three questions to be inferred are 40, 50, and 60, respectively. That is, when Top-K sampling is performed on the output candidate words of the three questions to be inferred, it is necessary to select the first 40 output candidate words, the first 50 output candidate words, and the first 60 output candidate words from the descending order of the output candidate words.

[0088] If J is determined to be 65, which is greater than the maximum value of 60 in the parameter K corresponding to the three problems to be reasoned, then in the first candidate word matrix, the output candidate words corresponding to each problem to be reasoned are arranged in descending order according to the output probability. Assuming that after the descending order, the output candidate words of the first problem to be reasoned are A1, A2, A3...A90000, the output candidate words of the second problem to be reasoned are B1, B2, B3...B90000, and the output candidate words of the third problem to be reasoned are C1, C2, C3...C90000, then in the descending order results of the output candidate words of each problem to be reasoned, the first 65 output candidate words are selected to generate the second candidate word matrix.

[0089] Since Top-K sampling selects the first 40 output candidate words, the first 50 output candidate words, and the first 60 output candidate words from the descending order of the output candidate words, and the second candidate word matrix has the first 65 output candidate words for each question to be inferred, these 65 output candidate words must include the output candidate words required for Top-K sampling.

[0090] Likewise, if Figure 3B As shown, if J is determined to be 60, which is equal to the maximum value 60 of the parameter K corresponding to the three questions to be inferred, then the first 60 output candidate words are selected from the descending order results of the output candidate words of each question to be inferred to generate the second candidate word matrix.

[0091] Since Top-K sampling selects the first 40 output candidate words, the first 50 output candidate words, and the first 60 output candidate words from the descending order of the output candidate words, and the second candidate word matrix has the first 60 output candidate words for each question to be inferred, these 60 output candidate words must include the first K output candidate words required for Top-K sampling.

[0092] The embodiment of the present application greatly reduces the dimension of the candidate word matrix by the above-mentioned operation of selecting output candidate words from the first candidate word matrix and generating the second candidate word matrix. For example, in the above-mentioned example, the dimension of the candidate word matrix is ​​reduced from 90,000 to 60, which is reduced by 1,500 times. When sampling is subsequently performed, sampling can be performed based on the second candidate word matrix, which greatly reduces the amount of calculation and computational overhead of each step, thereby improving the sampling speed and reasoning speed of the candidate words.

[0093] In some embodiments, J is greater than the maximum value of the parameters K corresponding to the multiple problems to be inferred, and the specific difference between the two can be set arbitrarily. For example, the difference between J and K is a natural number greater than or equal to 1 and less than 10. This is only used as an example and is not used for specific limitation. Of course, for the convenience of calculation, it is usually determined that J is equal to the maximum value of the parameters K corresponding to the multiple problems to be inferred. In the following embodiments of this application, J is taken as an example to illustrate the maximum value of the parameters K corresponding to the multiple problems to be inferred.

[0094] In some embodiments, after obtaining the second candidate word matrix, the embodiments of the present application can sample in the second candidate word matrix, determine the target candidate word for each question to be reasoned, and randomly select a candidate word from the target candidate words as the output for the question to be reasoned.

[0095] It should be noted that the embodiment of the present application does not specifically limit the specific selection method of randomly selecting candidate words from the target candidate words as the output for the problem to be inferred.

[0096] In the specific implementation, since the second candidate word matrix contains the first J output candidate words in the descending order of the output candidate words of each problem to be reasoned, and Top-K sampling requires the first K output candidate words in the descending order of the output candidate words of each problem to be reasoned, and J is greater than or equal to the maximum value of the parameter K corresponding to multiple problems to be reasoned, therefore, the output candidate words of some problems to be reasoned in the second candidate word matrix are greater than K, the second candidate word matrix has not yet completed Top-K sampling for each problem to be reasoned.

[0097] In addition, in actual applications, the situation where the first candidate word matrix and the second candidate word matrix only include one problem to be reasoned can be regarded as a special case, and its processing method is similar to the processing method of multiple problems to be reasoned. The following embodiment takes the processing method of multiple problems to be reasoned as an example to illustrate.

[0098] When sampling is performed in the second candidate word matrix in the embodiment of the present application, there may be the following two implementation methods to complete the sampling of the output candidate words for each problem to be inferred, which can be specifically summarized as a serial processing method and a parallel processing method, which are described below respectively.

[0099] Implementation method 1: The sampling parameters also include: a parameter P when performing Top-P sampling on the output candidate words, and a parallel processing method is adopted when sampling the output candidate words of multiple problems to be inferred.

[0100] In this case, in the second candidate word matrix, for each question to be inferred, the following operations are performed to obtain the third candidate word matrix:

[0101] Determine a parameter K corresponding to a problem to be inferred, retain the output probabilities of the first K output candidate words in the output candidate word ranking results corresponding to the problem to be inferred, and adjust the output probabilities of the remaining JK output candidate words to preset probabilities; wherein the preset probability is usually set to negative infinity or 0 to indicate that these output candidate words no longer participate in the calculation.

[0102] It should be noted that, for each question to be inferred, in the second candidate word matrix, the output probabilities of the first K output candidate words are retained, and after adjusting the output probabilities of the first K output candidate words and JK output candidate words to the preset probabilities, it is equivalent to that for each question to be inferred, only the first K output candidate words participate in the subsequent sampling, that is, the Top-K sampling of the output candidate words for each question to be inferred is completed.

[0103] For example, Figure 3B Taking the example shown as an example, the parameter K corresponding to the first question to be inferred is 40, the parameter K corresponding to the second question to be inferred is 50, and the parameter K corresponding to the third question to be inferred is 60. After adjusting the output probability of each output candidate word in the second candidate word matrix, the third candidate word matrix is ​​obtained.

[0104] like Figure 4 As shown in the third candidate word matrix, for the first question to be inferred, the output probabilities of the first 40 output candidate words are retained, and the output probabilities of the remaining 20 (41st to 60th) output candidate words are adjusted to negative infinity (i.e. Figure 4 -inf shown in ), thus completing the Top-K sampling of the output candidate words for the first problem to be inferred. In the subsequent sampling, only the first 40 output candidate words for the first problem to be inferred are sampled; for the second problem to be inferred, the output probabilities of the first 50 output candidate words are retained, and the output probabilities of the remaining 10 (51st to 60th) output candidate words are adjusted to negative infinity (i.e. Figure 4 -inf) shown in , thus completing the Top-K sampling of the output candidate words for the second problem to be reasoned. In the subsequent sampling, only the first 50 output candidate words for the second problem to be reasoned participate in the sampling; for the third problem to be reasoned, the output probabilities of the first 60 output candidate words are retained. Thus, the Top-K sampling of the output candidate words for the third problem to be reasoned is also completed. In the subsequent sampling, only the first 60 output candidate words for the third problem to be reasoned participate in the sampling.

[0105] In some embodiments, if the sampling also includes temperature sampling, after obtaining the third candidate word matrix, that is, after completing the Top-K sampling, the embodiment of the present application can also use the temperature parameters of the temperature sampling to adjust the output probability of each output candidate word in the third candidate word matrix. In this way, the temperature sampling is performed after the Top-K sampling, which can also reduce the calculation amount of the temperature sampling and improve the calculation speed of the reasoning process.

[0106] In specific implementation, after obtaining the third candidate word matrix, Top-K sampling of the output candidate words of each problem to be reasoned is completed. If Top-P sampling is performed at this time, the output probability of the output candidate words of each problem to be reasoned can be normalized in the third candidate word matrix to obtain the normalized output probability of each output candidate word and obtain the fourth candidate word matrix. Then, in the fourth candidate word matrix, Top-P sampling is performed according to the ranking of the output candidate words of each problem to be reasoned and the parameter P corresponding to each problem to be reasoned to determine the target candidate word for each problem to be reasoned.

[0107] It should be noted that in the related art, when performing Top-P sampling, it is also necessary to first sort the output candidate words in descending order. Therefore, in the related art, when performing joint sampling including Top-K sampling and Top-P sampling, Top-K sampling needs to perform a sorting operation, and Top-P sampling also needs to perform a sorting operation. The entire joint sampling process requires two sorting operations, and the sorting operation is relatively time-consuming, which also causes the existing joint sampling to be more time-consuming in calculation.

[0108] In the embodiment of the present application, when Top-P sampling is performed in the fourth candidate word matrix after completing Top-K sampling, since the output candidate words in the fourth candidate word matrix are already arranged in descending order, there is no need to perform a sorting operation again. Compared with the prior art that requires two sorting operations, the entire Top-K sampling and Top-P sampling process can reuse the results of the descending order, and only one sorting operation is required, thereby reducing the computational time of the sampling process.

[0109] In specific implementation, Top-P sampling is performed in the fourth candidate word matrix to determine the target candidate word for each question to be reasoned. Based on the output probability corresponding to each output candidate word in the fourth candidate word matrix and the ranking of the output candidate words for each question to be reasoned, the cumulative probability corresponding to each output candidate word set is calculated to construct a probability matrix, wherein the cumulative probability corresponding to the i-th output candidate word set in the probability matrix is ​​the sum of the output probabilities of the first i output candidate words in the ranking of the output candidate words corresponding to each question to be reasoned, and i is a positive integer.

[0110] For example, Figure 4Taking the example shown as an example, assuming that after the output probability normalization processing, the output probabilities of the first 40 output candidate words corresponding to the first question to be reasoned are D1...D40, the output probabilities of the first 50 output candidate words corresponding to the second question to be reasoned are E1...E50, and the output probabilities of the first 60 output candidate words corresponding to the third question to be reasoned are F1...F60.

[0111] like Figure 5 As shown, in the probability matrix, for the first problem to be inferred, the first output candidate word set only includes the output candidate word D1, and its corresponding cumulative probability is still D1. The second output candidate word set includes the output candidate word D1 and the output candidate word D2, and its corresponding cumulative probability is D1+D2. The third output candidate word set includes the output candidate word D1, the output candidate word D2 and the output candidate word D3, and its corresponding cumulative probability is D1+D2+D3. By analogy, the 39th output candidate word set includes the output candidate word D1, the output candidate word D2...the output candidate word D39, and its corresponding cumulative probability is D1+D2+...+D39. The 40th output candidate word set includes the output candidate word D1, the output candidate word D2...the output candidate word D40, and its corresponding cumulative probability is 1. The cumulative probabilities of subsequent output candidate word sets are all 1.

[0112] Similarly, for the second problem to be inferred, the first output candidate word set only includes output candidate word E1, and its corresponding cumulative probability is still E1. The second output candidate word set includes output candidate word E1 and output candidate word E2, and its corresponding cumulative probability is E1+E2. The third output candidate word set includes output candidate word E1, output candidate word E2 and output candidate word E3, and its corresponding cumulative probability is E1+E2+E3. By analogy, the 39th output candidate word set includes output candidate word E1, output candidate word E2...output candidate word E39, and its corresponding cumulative probability is E1+E2+…+E39. The 49th output candidate word set includes output candidate word E1, output candidate word E2...output candidate word E49, and its corresponding cumulative probability is E1+E2+…+E49. The 50th output candidate word set includes output candidate word E1, output candidate word E2...output candidate word E50, and its corresponding cumulative probability is 1. The cumulative probabilities of subsequent output candidate words are all 1.

[0113] For the third problem to be inferred, the first output candidate word set includes output candidate word F1, and its corresponding cumulative probability is still F1. The second output candidate word set includes output candidate word F1 and output candidate word F2, and its corresponding cumulative probability is F1+F2. The third output candidate word set includes output candidate word F1, output candidate word F2 and output candidate word F3, and its corresponding cumulative probability is F1+F2+F3. By analogy, the 39th output candidate word set includes output candidate word F1, output candidate word F2...output candidate word F39, and its corresponding cumulative probability is F1+F2+…+F39. The 49th output candidate word set includes output candidate word F1, output candidate word F2...output candidate word F49, and its corresponding cumulative probability is F1+F2+…+F49. The 60th output candidate word set includes output candidate word F1, output candidate word F2...output candidate word F60, and its corresponding cumulative probability is 1.

[0114] In specific implementation, Top-P sampling is to select candidate words included in the minimum set whose cumulative probability is greater than or equal to the probability P. Therefore, after obtaining the probability matrix, for each problem to be inferred, the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P are determined as the target candidate words.

[0115] Specifically, for each problem to be inferred, when the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P are determined as target candidate words, in the probability matrix, for each output candidate word of the problem to be inferred, the following operations are performed respectively: determine the target cumulative probability whose cumulative probability is greater than or equal to the parameter P for the first time, adjust the cumulative probability greater than the target cumulative probability to the preset probability, and in the adjusted probability matrix, determine the output candidate words included in the output candidate word set corresponding to the maximum cumulative probability as the target candidate words. Among them, the preset probability is usually set to negative infinity or 0 to indicate that these cumulative probabilities are no longer involved in the calculation.

[0116] For example, Figure 5 Taking the probability matrix shown as an example, assuming that the parameter P corresponding to the first problem to be inferred is 0.9, the parameter P corresponding to the second problem to be inferred is 0.8, and the parameter P corresponding to the third problem to be inferred is 0.95. Assuming that D1+D2+…+D10=0.91, which is greater than or equal to 0.9 for the first time, then for the first problem to be inferred, 0.91 is the target cumulative probability, E1+E2+…+E5=0.82, which is greater than or equal to 0.8 for the first time, then for the second problem to be inferred, 0.82 is the target cumulative probability, F1+F2+…+F12=0.96, which is greater than or equal to 0.95 for the first time, then for the third problem to be inferred, 0.96 is the target cumulative probability.

[0117] like Figure 6As shown in the figure, after determining the target cumulative probability corresponding to each problem to be inferred, in the probability matrix, the cumulative probability greater than the target cumulative probability is adjusted to the preset probability. For the first problem to be inferred, the cumulative probability after D1+D2+…+D10 is adjusted to negative infinity (i.e. Figure 6 -inf) shown in, for the second problem to be inferred, the cumulative probability after E1+E2+…+E5 is adjusted to negative infinity, and for the third problem to be inferred, the cumulative probability after F1+F2+…+F12 is adjusted to negative infinity.

[0118] At the same time, in the adjusted probability matrix, for each problem to be inferred, the output candidate word included in the output candidate word set corresponding to the maximum cumulative probability (that is, the target cumulative probability) can be determined as the target candidate word. Figure 6 Taking the figure as an example, at this time, among the cumulative probabilities corresponding to the first question to be reasoned, the maximum cumulative probability is D1+D2+…+D10=0.91, then the 10 output candidate words (D1, D2…D10) in the output candidate word set are the target candidate words for the first question to be reasoned. Similarly, the target candidate words for the second question to be reasoned can be determined, which are E1, E2, E3, E4 and E5, and the target candidate words for the third question to be reasoned are F1, F2…F12.

[0119] In the above embodiment, by adjusting the output probability or cumulative probability of some output candidate words to a preset probability, when determining the target candidate words of multiple problems to be reasoned through sampling, parallel processing of multiple problems to be reasoned can be achieved. Compared with the method of determining the target candidate words of the problems to be reasoned one by one, the sampling speed and the reasoning speed can be further improved.

[0120] Implementation method 2: The sampling parameters also include: parameter P when performing Top-P sampling on the output candidate words, and a serial processing method is adopted when sampling the output candidate words of multiple problems to be inferred.

[0121] In this case, when sampling is performed in the second candidate word matrix to determine the target candidate word for each question to be inferred, the following operations are performed in sequence for each question to be inferred:

[0122] Determine a parameter K corresponding to a problem to be inferred, take out the first K output candidate words from the ranking results of the output candidate words corresponding to the problem to be inferred in the second candidate word matrix, perform Top-P sampling based on the ranking of the first K output candidate words, the output probability corresponding to each output candidate word, and the parameter P corresponding to the problem to be inferred, and determine the target candidate word for the problem to be inferred.

[0123] For example, Figure 3BTaking the second candidate word matrix shown as an example, for each question to be inferred, the second candidate word matrix includes the top 60 output candidate words with the highest output probability.

[0124] First, determine that the parameter K corresponding to the first question to be reasoned is 40. Then, among the 60 output candidate words corresponding to the first question to be reasoned in the second candidate word matrix, take out the first 40 output candidate words to complete Top-K sampling, and then perform Top-P sampling based on the ranking of the first 40 output candidate words, the output probability corresponding to each output candidate word, and the parameter P corresponding to the first question to be reasoned, that is, determine the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P, and determine the output candidate words included in the minimum set as the target candidate words for the first question to be reasoned.

[0125] Then, it is determined that the parameter K corresponding to the second question to be reasoned is 50. Then, among the 60 output candidate words corresponding to the second question to be reasoned in the second candidate word matrix, the first 50 output candidate words are taken out to complete Top-K sampling. Then, Top-P sampling is performed based on the ranking of the first 50 output candidate words, the output probability corresponding to each output candidate word, and the parameter P corresponding to the second question to be reasoned, that is, the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P are determined, and the output candidate words included in the minimum set are determined as the target candidate words for the second question to be reasoned.

[0126] Finally, it is determined that the parameter K corresponding to the third question to be inferred is 60, then the 60 output candidate words contained in the second candidate word matrix are taken out to complete Top-K sampling, and then Top-P sampling is performed based on the sorting of these 60 output candidate words, the output probability corresponding to each output candidate word, and the parameter P corresponding to the third question to be inferred, that is, the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P are determined, and the output candidate words included in the minimum set are determined as the target candidate words for the third question to be inferred.

[0127] In some embodiments, sampling may also include temperature sampling, and the sampling parameters may also include: temperature parameters when performing temperature sampling on the output candidate words. In this embodiment, when sampling is performed in the second candidate word matrix, the temperature parameters may be used to first adjust the output probability of each output candidate word in the second candidate word matrix, and the output probability of each output candidate word after temperature adjustment may be obtained to obtain a new second candidate word matrix, and other sampling may be performed in the new second candidate word matrix.

[0128] Of course, as in the above-mentioned embodiment of the present application, temperature sampling can also be performed in the third candidate word matrix. It should be noted that whether the temperature sampling is performed in the second candidate word matrix or in the third candidate word matrix, compared with performing it in the first candidate word matrix, it can reduce the calculation amount of temperature sampling and improve the calculation speed of the reasoning process.

[0129] In order to better explain the embodiment of the present application, taking the joint sampling of output candidate words of multiple problems to be inferred, that is, Top-K sampling, Top-P sampling and temperature sampling as an example, the process of a data processing method provided by the embodiment of the present application is introduced in combination with a specific implementation scenario, such as Figure 7 As shown, the process of the method includes the following steps:

[0130] Step 701, obtain the first candidate word matrix to be sampled and the sampling parameters set for each problem to be inferred. The sampling parameters include: parameter K when Top-K sampling is performed on the output candidate words, parameter P when Top-P sampling is performed on the output candidate words, and temperature parameter T when temperature sampling is performed on the output candidate words.

[0131] Step 702, determining the maximum value J among the parameters K corresponding to the multiple problems to be inferred.

[0132] Step 703: In the first candidate word matrix, for each output candidate word of the question to be inferred, the output candidate words are arranged in descending order based on the output probability.

[0133] Step 704: In the first candidate word matrix, the first J output candidate words are selected from the output candidate word ranking results of each question to be inferred, and a second candidate word matrix is ​​generated.

[0134] Step 705, in the second candidate word matrix, for each question to be inferred, in the output candidate word ranking result corresponding to the question to be inferred, the output probabilities of the corresponding first K output candidate words are retained, and the output probabilities of the remaining JK output candidate words are adjusted to negative infinity to obtain a third candidate word matrix.

[0135] At this point, the Top-K sampling of the output candidate words for each question to be inferred is completed.

[0136] Step 706, using the temperature parameter T corresponding to each question to be inferred, respectively adjust the output probability of the output candidate word corresponding to each question to be inferred in the third candidate word matrix to obtain a new third candidate word matrix.

[0137] At this point, the temperature sampling of the output candidate words for each question to be inferred is completed.

[0138] Step 707: In the new third candidate word matrix, normalize the output probability of each output candidate word of the problem to be inferred, obtain the normalized output probability of each output candidate word, and obtain the fourth candidate word matrix.

[0139] Step 708, in the fourth candidate word matrix, based on the output probability corresponding to each output candidate word and the ranking of the output candidate words of each problem to be inferred, the cumulative probability corresponding to each output candidate word set is calculated to construct a probability matrix.

[0140] Among them, the cumulative probability corresponding to the i-th output candidate word set in the probability matrix is ​​the sum of the output probabilities of the first i output candidate words in the output candidate word ranking corresponding to each question to be inferred, and i is a positive integer.

[0141] Step 709, in the probability matrix, for each output candidate word of the problem to be inferred, determine the target cumulative probability whose cumulative probability is greater than or equal to the parameter P for the first time, adjust the cumulative probability greater than the target cumulative probability to negative infinity, and in the adjusted probability matrix, determine the output candidate word included in the output candidate word set corresponding to the maximum cumulative probability as the target candidate word.

[0142] At this point, the Top-P sampling of the output candidate words for each question to be inferred is completed.

[0143] Step 710: Randomly select a candidate word from the target candidate words of each question to be inferred as the output for the question to be inferred.

[0144] Based on the same technical concept, the present application embodiment provides a structural diagram of a data processing device, such as Figure 8 As shown, the data processing device 800 includes:

[0145] The acquisition module 801 is used to acquire a first candidate word matrix to be sampled and sampling parameters set for each problem to be inferred, wherein the first candidate word matrix contains output candidate words for each problem to be inferred and corresponding output probabilities, the dimension of the first candidate word matrix is ​​N×M, N is the number of problems to be inferred, M is the number of output candidate words for each problem to be inferred, and the sampling parameters at least include: a parameter K when performing Top-K sampling on the output candidate words, M, N and K are all positive integers;

[0146] Processing module 802 is used to arrange the output candidate words of each problem to be reasoned in descending order according to the output probability in the first candidate word matrix, and select the first J output candidate words from the sorted results of the output candidate words of each problem to be reasoned to generate a second candidate word matrix, wherein the dimension of the second candidate word matrix is ​​N×J, J is determined based on the parameter K corresponding to the problem to be reasoned, and the difference between M and J is greater than a preset threshold, and J is a positive integer.

[0147] Optionally, the data processing device further includes: a sampling module 803, which is used to perform sampling in the second candidate word matrix, determine the target candidate word for each question to be inferred, and randomly select a candidate word from the target candidate words as an output for the question to be inferred.

[0148] Optionally, there are multiple questions to be inferred, and the sampling parameters further include: a parameter P when performing Top-P sampling on the output candidate words, where P is greater than or equal to 0 and less than or equal to 1;

[0149] The sampling module 803 is specifically used for:

[0150] In the second candidate word matrix, for each question to be inferred, the following operations are performed to obtain a third candidate word matrix: a parameter K corresponding to a question to be inferred is determined, and in the output candidate word ranking result corresponding to a question to be inferred, the output probabilities of the first K output candidate words are retained, and the output probabilities of the remaining JK output candidate words are adjusted to preset probabilities;

[0151] In the third candidate word matrix, the output probability of each output candidate word of the problem to be inferred is normalized to obtain the normalized output probability of each output candidate word, thereby obtaining a fourth candidate word matrix;

[0152] In the fourth candidate word matrix, Top-P sampling is performed according to the ranking of the output candidate words of each question to be inferred and the parameter P corresponding to each question to be inferred, and the target candidate word of each question to be inferred is determined.

[0153] Optionally, the sampling module 803 is specifically configured to:

[0154] Based on the output probabilities corresponding to the output candidate words in the fourth candidate word matrix and the ranking of the output candidate words for each question to be inferred, the cumulative probabilities corresponding to each output candidate word set are calculated to construct a probability matrix, wherein the cumulative probability corresponding to the i-th output candidate word set in the probability matrix is ​​the sum of the output probabilities of the first i output candidate words in the ranking of the output candidate words corresponding to each question to be inferred, where i is a positive integer;

[0155] For each question to be inferred, the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P are determined as the target candidate words.

[0156] Optionally, the sampling module 803 is specifically used for:

[0157] In the probability matrix, for each output candidate word of the problem to be inferred, the following operations are performed respectively: determine the target cumulative probability whose cumulative probability is greater than or equal to the parameter P for the first time, adjust the cumulative probability greater than the target cumulative probability to the preset probability, and in the adjusted probability matrix, determine the output candidate word included in the output candidate word set corresponding to the maximum cumulative probability as the target candidate word.

[0158] Optionally, there are multiple questions to be inferred, and the sampling parameters further include: a parameter P when performing Top-P sampling on the output candidate words;

[0159] The sampling module 803 is specifically used for:

[0160] For each problem to be inferred, perform the following operations in sequence:

[0161] Determine a parameter K corresponding to a question to be inferred, and take out the first K output candidate words from the sorted output candidate words corresponding to a question to be inferred in the second candidate word matrix;

[0162] Top-P sampling is performed based on the ranking of the first K output candidate words, the output probability corresponding to each output candidate word, and the parameter P corresponding to a question to be inferred, and a target candidate word for a question to be inferred is determined.

[0163] Optionally, J is determined in the following manner:

[0164] When there is only one problem to be inferred, J is greater than or equal to the parameter K corresponding to the problem to be inferred; or

[0165] When there are multiple problems to be inferred, J is greater than or equal to the maximum value of the parameters K corresponding to the multiple problems to be inferred.

[0166] Optionally, the sampling parameters further include: a temperature parameter when performing temperature sampling on the output candidate words;

[0167] The sampling module 803 is specifically used for:

[0168] The temperature parameter is used to adjust the output probability of each output candidate word in the second candidate word matrix, obtain the output probability of each output candidate word after temperature adjustment, obtain a new second candidate word matrix, and perform sampling in the new second candidate word matrix.

[0169] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Fig. 9 As shown, it includes at least one processor chip 901 and a memory 902 connected to the at least one processor chip. The specific connection medium between the processor chip 901 and the memory 902 is not limited in the embodiment of the present application. Fig. 9For example, the processor chip 901 and the memory 902 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0170] In the embodiment of the present application, the memory 902 stores instructions that can be executed by at least one processor chip 901, and the at least one processor chip 901 can perform the steps of the above-mentioned data processing method by executing the instructions stored in the memory 902.

[0171] Among them, the processor chip 901 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and data processing can be realized by running or executing instructions stored in the memory 902 and calling data stored in the memory 902. Optionally, the processor chip 901 may include one or more processing units, and the processor chip 901 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor chip 901. In some embodiments, the processor chip 901 and the memory 902 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0172] The processor chip 901 can be a general-purpose processor, such as a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0173] The memory 902 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 902 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RandomAccess Memory, RAM), a static random access memory (Static RandomAccess Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 902 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 902 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0174] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned data processing method.

[0175] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned data processing method.

[0176] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0180] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0181] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: include: Obtain a first candidate word matrix to be sampled and sampling parameters set for each problem to be inferred, wherein the first candidate word matrix contains output candidate words and corresponding output probabilities of each problem to be inferred, the dimension of the first candidate word matrix is ​​N×M, N is the number of problems to be inferred, M is the number of output candidate words for each problem to be inferred, and the sampling parameters at least include: parameters K when performing Top-K sampling on the output candidate words, M, N and K are all positive integers; In the first candidate word matrix, the output candidate words of each question to be inferred are arranged in descending order according to the output probability, and the first J output candidate words are selected from the output candidate word sorting results of each question to be inferred to generate a second candidate word matrix, wherein the dimension of the second candidate word matrix is ​​N×J, and J is determined based on the parameter K corresponding to the question to be inferred, and the difference between M and J is greater than a preset threshold, and J is a positive integer.

2. The method according to claim 1, characterized in that The method further comprises: Sampling is performed in the second candidate word matrix to determine a target candidate word for each question to be inferred, and a candidate word is randomly selected from the target candidate words as an output for the question to be inferred.

3. The method according to claim 2, characterized in that There are multiple problems to be inferred, and the sampling parameters further include: a parameter P when performing Top-P sampling on the output candidate words, where P is greater than or equal to 0 and less than or equal to 1; The sampling in the second candidate word matrix to determine the target candidate word for each question to be inferred includes: In the second candidate word matrix, for each question to be inferred, the following operations are performed to obtain a third candidate word matrix: a parameter K corresponding to a question to be inferred is determined, and in the output candidate word ranking result corresponding to the question to be inferred, the output probabilities of the first K output candidate words are retained, and the output probabilities of the remaining JK output candidate words are adjusted to preset probabilities; In the third candidate word matrix, normalizing the output probability of each output candidate word of the problem to be inferred is performed to obtain the normalized output probability of each output candidate word, thereby obtaining a fourth candidate word matrix; In the fourth candidate word matrix, Top-P sampling is performed according to the ranking of the output candidate words of each question to be inferred and the parameter P corresponding to each question to be inferred, so as to determine the target candidate word of each question to be inferred.

4. The method according to claim 3, characterized in that In the fourth candidate word matrix, Top-P sampling is performed according to the ranking of the output candidate words of each question to be inferred and the parameter P corresponding to each question to be inferred, and the target candidate word of each question to be inferred is determined, including: Based on the output probabilities corresponding to the output candidate words in the fourth candidate word matrix and the ranking of the output candidate words of each question to be inferred, the cumulative probabilities corresponding to each output candidate word set are calculated to construct a probability matrix, wherein the cumulative probability corresponding to the i-th output candidate word set in the probability matrix is ​​the sum of the output probabilities of the first i output candidate words in the ranking of the output candidate words corresponding to each question to be inferred, where i is a positive integer; For each question to be inferred, the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P are determined as the target candidate words.

5. The method according to claim 4, characterized in that For each problem to be inferred, determining the output candidate words included in the minimum set whose cumulative probability is greater than or equal to the parameter P as the target candidate words includes: In the probability matrix, for each output candidate word of the problem to be inferred, the following operations are performed respectively: determine the target cumulative probability whose cumulative probability is greater than or equal to the parameter P for the first time, adjust the cumulative probability greater than the target cumulative probability to the preset probability, and in the adjusted probability matrix, determine the output candidate word included in the output candidate word set corresponding to the maximum cumulative probability as the target candidate word.

6. The method according to claim 2, characterized in that There are multiple problems to be inferred, and the sampling parameters also include: parameter P when performing Top-P sampling on the output candidate words; The sampling in the second candidate word matrix to determine the target candidate word for each question to be inferred includes: For each problem to be inferred, perform the following operations in sequence: Determine a parameter K corresponding to a question to be inferred, and take out the first K output candidate words from the sorted output candidate words corresponding to the question to be inferred in the second candidate word matrix; Top-P sampling is performed based on the ranking of the first K output candidate words, the output probability corresponding to each output candidate word, and the parameter P corresponding to the one to-be-reasoned question to determine the target candidate word for the one to-be-reasoned question.

7. The method according to any one of claims 1 to 6, characterized in that The J is determined in the following manner: When there is one problem to be inferred, J is greater than or equal to a parameter K corresponding to the problem to be inferred; or When there are multiple problems to be inferred, J is greater than or equal to the maximum value of the parameters K corresponding to the multiple problems to be inferred.

8. The method according to any one of claims 1 to 6, characterized in that The sampling parameters also include: temperature parameters when performing temperature sampling on the output candidate words; The sampling in the second candidate word matrix includes: The temperature parameter is used to adjust the output probability of each output candidate word in the second candidate word matrix, obtain the output probability of each output candidate word after temperature adjustment, obtain a new second candidate word matrix, and perform sampling in the new second candidate word matrix.

9. A computer device comprising a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, characterized in that: When the processor chip executes the program, the steps of any one of the methods of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 8.