Numerical task big language prediction model construction method, prediction method and related products

Abstract

The invention provides a numerical task big language prediction model construction method, a prediction method and a related product, and relates to the technical field of numerical prediction. The large language prediction model construction method comprises the steps of obtaining a sample data set corresponding to a numerical task; each sample in the sample data set comprises input information and target output; inputting the input information into a forward process of the large language prediction model to obtain an output text of the forward process; the output text is a text formatted based on a cue word template, and the cue word template is used for specifying an index interval corresponding to the numerical value text; extracting numerical text output in the output text based on the index interval corresponding to the numerical text; performing numerical meaning alignment operation on the numerical text output, and calculating the loss between the output after the alignment operation and the target output; and training a big language prediction model based on loss. According to the method, the loss of the value meaning is avoided, and the prediction precision of the value prediction task is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of numerical prediction, in particular to a large language prediction model construction method for numerical tasks, a prediction method and related products. BACKGROUND

[0002] In recent years, large language models (LLMs) have gradually broken through the boundaries of natural language processing (NLP) and shown significant potential in the field of numerical prediction, thanks to their powerful sequence modeling and context understanding capabilities.

[0003] When processing numerical prediction tasks, in order to utilize the language processing capabilities of large language models, numerical training data is often converted into text form for training as language data, and the loss function used is cross entropy. For example, when predicting weather temperatures, temperature data is often converted into text form, and when predicting stock trends, stock price data is converted into text form. Although this method preserves the text processing capabilities of the model, it loses numerical meaning information, which seriously affects the accuracy of numerical prediction. SUMMARY

[0004] The present application provides a large language prediction model construction method for numerical tasks, a prediction method and related products, to solve the defect that existing large language models lose numerical meaning information when predicting numerical tasks, and to improve the accuracy of numerical task prediction.

[0005] The present application provides a large language prediction model construction method for numerical tasks, comprising the following steps.

[0006] Obtain a sample data set corresponding to a numerical prediction task; each sample in the sample data set includes input information and target output; Input the input information into the forward process of a large language prediction model to obtain the output text of the forward process; the output text is a text formatted based on a prompt word template, and the prompt word template is used to specify the index interval corresponding to the numerical text; Extract the numerical text output in the output text based on the index interval corresponding to the numerical text; Perform a numerical meaning alignment operation on the numerical text output and calculate the loss between the output after the alignment operation and the target output; Train the large language prediction model based on the loss.

[0007] According to the large language prediction model construction method for numerical tasks provided by the present application, the numerical text output is subjected to a numerical meaning alignment operation, and the loss between the output after the alignment operation and the target output is calculated, including: The numerical output is parsed based on a floating-point number composition rule to obtain a numerical output. The numerical loss between the numerical output and the target output is calculated.

[0008] According to the numerical task large language prediction model construction method provided by the application, the numerical output is parsed based on a floating-point number composition rule to obtain a numerical output, which includes: The numerical output is parsed based on the following formula: Wherein, represents the numerical output; represents the positive or negative sign in the numerical text output; represents the bit mark, The rule of is that for the text before the decimal point in the numerical text output, from back to front, the unit corresponds to , the ten corresponds to , and so on. For the text after the decimal point in the numerical text output, from front to back, the first digit after the decimal point corresponds to , the second digit after the decimal point corresponds to , and the third digit after the decimal point corresponds to , and so on. represents the numerical value corresponding to the text in the numerical text output.

[0009] According to the numerical task large language prediction model construction method provided by the application, the numerical meaning of the numerical text output is aligned, and the loss between the output after the alignment operation and the target output is calculated, which includes: Based on the difference in the size of the numerical value at different positions, a first weight is assigned to each word in the numerical text output; The loss between the numerical text output and the target output is calculated based on the first weight.

[0010] According to the numerical task large language prediction model construction method provided by the application, the numerical meaning of the numerical text output is aligned, and the loss between the output after the alignment operation and the target output is calculated, which also includes: Based on the gap between the word and its corresponding target output, a second weight is assigned to each word in the numerical text output; The loss between the numerical text output and the target output is calculated based on the first weight, which includes: Based on the first weight and the second weight, the comprehensive weight of each word in the numerical text output is determined; ​Calculate a loss between the numerical text output and the target output based on the comprehensive weight of each word in the numerical text output.

[0011] According to the numerical task large language prediction model construction method provided by the application, a second weight is given to each word in the numerical text output based on the gap between the word and its corresponding target output, which comprises: A second weight is given to each word in the numerical text output based on Gaussian distribution; wherein the Gaussian distribution mean of each word is the numerical value of the corresponding bit in the target output.

[0012] According to the numerical task large language prediction model construction method provided by the application, the input information is input into the forward process of the large language prediction model, and the output text of the forward process is obtained, which comprises: The input information is input into the forward process of the large language prediction model, and the prediction distribution of each word in the output is obtained; the prediction distribution represents a score vector based on the dictionary; For each word, the prediction value of the word is calculated based on the score vector of the word and the dictionary using weighted average method, and the output text is obtained.

[0013] The application also provides a numerical task prediction method, which comprises: Obtain a numerical prediction task; the numerical prediction task comprises input information and a problem to be predicted; The input information and the problem to be predicted are input into the large language prediction model described above to obtain a numerical prediction output.

[0014] The application also provides a numerical task large language prediction model construction device, which comprises the following modules: A data set acquisition module is used to acquire a sample data set corresponding to a numerical prediction task; each sample in the sample data set comprises input information and a target output; An output text acquisition module is used to input the input information into the forward process of the large language prediction model to obtain the output text of the forward process; the output text is a text formatted based on a prompt word template, and the prompt word template is used to specify the index interval corresponding to the numerical text; A numerical text output extraction module is used to extract the numerical text output in the output text based on the index interval corresponding to the numerical text; An alignment operation module is used to perform numerical meaning alignment operation on the numerical text output and calculate the loss between the output after alignment operation and the target output; A model training module is used to train the large language prediction model based on the loss.

[0015] The application also provides a numerical task prediction device, which comprises the following modules: a numerical prediction task acquisition module configured to acquire a numerical prediction task; the numerical prediction task comprising input information and a problem to be predicted; a numerical prediction output module configured to input the input information and the problem to be predicted into a large language prediction model constructed by the large language prediction model construction method, to obtain a numerical prediction output.

[0016] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the numerical task large language prediction model construction method or the numerical task prediction method according to any one of the above when executing the program.

[0017] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the numerical task large language prediction model construction method or the numerical task prediction method according to any one of the above.

[0018] The application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the numerical task large language prediction model construction method or the numerical task prediction method according to any one of the above.

[0019] The numerical task large language prediction model construction method and the prediction method provided by the application can avoid the loss of numerical meaning of the large language prediction model by performing numerical meaning alignment operation on the output text of the forward process of the large language prediction model and calculating the loss between the output after the alignment operation and the target output, and train the model parameters based on the loss, thereby improving the prediction accuracy of the numerical prediction task based on the large language prediction model. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 is a flowchart of the numerical task large language prediction model construction method provided by the application.

[0022] Figure 2 is one of the flowcharts of the numerical meaning alignment operation provided by the embodiments of the application.

[0023] Figure 3 is another of the flowcharts of the numerical meaning alignment operation provided by the embodiments of the application.

[0024] Figure 4 is a weight assignment principle based on a Gaussian distribution provided by an embodiment of the application.

[0025] Figure 5 is a flow diagram of a loss calculation method provided by an embodiment of the application.

[0026] Figure 6 is a flow diagram of a determination method of a digital word prediction value provided by an embodiment of the application.

[0027] Figure 7 is a flow diagram of a numerical task prediction method provided by an embodiment of the application.

[0028] Figure 8 is a structural diagram of a numerical task large language prediction model construction device provided by an embodiment of the application.

[0029] Figure 9 is a structural diagram of a numerical task prediction device provided by an embodiment of the application.

[0030] Figure 10 is a structural diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0032] In processing numerical prediction tasks such as stock price prediction and environmental temperature prediction, in order to obtain better and more accurate prediction results, we hope to take advantage of the context understanding ability of large language models, but when using large language models to process numerical prediction tasks, the following problems are encountered: first, tokenize the numerical value through the embedding module to align to the large language model space, perform calculation on the large model space, and perform reverse embedding at the output end to obtain numerical output, typical methods such as GPT4TS, the problem of this kind of method is that it can only process pure numerical data and cannot take advantage of the language processing ability of large language models. Second, convert numerical training data into text form as language data for training, and use cross-entropy as the loss function, this method retains the text processing ability of the model, but there is a meaning alignment problem in the processing of numerical values, for example, the temperature prediction value y=38 degrees, the numerical part is converted into the following character sequence after being converted into a string: ["3", "8"], but in numerical representation, each digit (called token in large language models (such as LLM), hereinafter also referred to as "word") has different weights, and the weight of the unit digit is 10 times higher than that of the tenth digit. When the large language model is processed, the weight of each character is the same; the same digit of the numerical value has different meanings between numbers, such as the error between 3 and 4 is small, while the error between 3 and 9 is large, and the large language model only considers whether the result is label 3 when processing, and there is no difference between predicting classes 4 and 9. The embodiments of the present application fine-tune the loss calculation of the large language model for processing numerical prediction tasks, that is, align the numerical meaning of the numerical text output of the forward process of the large language model, which realizes the improvement of the prediction accuracy of the numerical task while retaining the language reasoning ability of the large language model.

[0033] The following will be described in combination with Figures 1-7 The numerical task large language prediction model construction method and the numerical task prediction method of the present application are described. The numerical task large language prediction model construction method and the numerical task prediction method can be applied to terminal devices or servers and other electronic devices. Among them, the terminal device can include a mobile phone, a computer, a tablet computer, a smart terminal, etc.; the server can include a standalone server, a cluster server, or a cloud server, etc. The numerical task large language prediction model construction method or the numerical task prediction method can also be applied to the numerical task large language prediction model construction device or the numerical task prediction device provided in the terminal device or the server and other electronic devices, and the numerical task large language prediction model construction device or the numerical task prediction device can be realized by software, hardware or a combination of both.

[0034] Figure 1 The flowchart of the numerical task large language prediction model construction method provided by the present application is shown in Figure 1 The method comprises the following steps 101 to 105.

[0035] Step 101, obtaining a sample data set corresponding to a numerical task; each sample in the sample data set includes input information and target output.

[0036] Step 102, inputting the input information into a forward process of a large language prediction model to obtain output text of the forward process; the output text is text formatted based on a prompt word template, and the prompt word template is used to specify an index interval corresponding to a numerical text.

[0037] Specifically, the input information of the sample, the problem to be predicted, and the output format requirement can be formatted into text according to a preset prompt word template, and the forward process of the large language prediction model is run to obtain the output text. O and each word score vector V (i.e., the output value of the last layer of the model before entering the Softmax calculation probability), assuming that the number of the dictionary of the large language prediction model is N, and the length of the output text is O |, then . From the output text O , according to the format specified by the prompt word template F, the index interval corresponding to the output numerical part is extracted s : e ], then the corresponding text is O[s:e], denoted as (which represents a slice, O from s to e directly). Here is an example to facilitate understanding, assuming that the specification of the output value in the template F is "output value={}", and the output text O is "output value=-3.14", then s =5, e =9, O [ s : e ] is "-3.14".

[0038] Taking stock price prediction as an example, the input information can be historical prices, stock market indexes, stock fundamental information, and analyst evaluations, and the output target can be the end-of-period price or yield of the stock in a certain period in the future. Taking electricity consumption prediction as an example, the input information can be historical electricity consumption, environmental temperature, and weather, and the output target can be electricity consumption in a certain period in the future.

[0039] Step 103, extracting the numerical text output in the output text based on the index interval corresponding to the numerical text.

[0040] It should be noted that the numerical text output here refers to a text form, but it is a numerical content.

[0041] Step 104, perform a numerical meaning alignment operation on the numerical text output, and calculate the loss between the output after the alignment operation and the target output.

[0042] Step 105, train the parameters of the large language prediction model based on the loss. Specifically, the CE (Cross-Entropy) loss, that is, the cross-entropy loss, can be used, which is mainly used to measure the difference between two different probability distributions of the same random variable. The smaller the cross-entropy loss, the closer the two probability distributions, which means that the prediction result is more accurate.

[0043] It can be understood that the embodiment of the present application can be based on a pre-trained large language prediction model.

[0044] The embodiment of the present application avoids the loss of numerical meaning of the large language prediction model by performing a numerical meaning alignment operation on the output text of the forward process of the large language prediction model, thereby improving the prediction accuracy of the numerical prediction task based on the large language prediction model.

[0045] In one example embodiment, referring to Figure 2 , the numerical meaning alignment operation of step 103 can be implemented by the following steps 201 to step 202.

[0046] Step 201, perform numerical meaning analysis on the numerical text output based on a floating-point number composition rule to obtain a numerical output.

[0047] Specifically, the numerical meaning analysis on the numerical text output can be performed according to the following formula: wherein, represents the numerical output; represents a positive sign or a negative sign in the numerical text output; represents a bit marker, The rule for is that for the text before the decimal point in the numerical text output, from back to front, the unit place corresponds to , the ten's place corresponds to , and so on. For the text after the decimal point in the numerical text output, from front to back, the first digit after the decimal point corresponds to , the second digit after the decimal point corresponds to , the third digit after the decimal point corresponds to , and so on. represents the numerical value corresponding to the text in the th place in the numerical text output.

[0048] For example, the numerical meaning analysis corresponding to a numerical text output floating-point number "-ef.cd" can be as follows: .

[0049] Step 202, calculate the numerical loss between the numerical output and the target output.

[0050] It should be noted that the target output at this time is not in text form, but in numerical form. When calculating the loss, it is a loss between numbers, and the size of the loss can reflect the numerical meaning. That is, if the numerical meaning alignment operation provided by the embodiment of the present application is not used, the numerical training data will be converted into text form by using the context understanding ability of the large language prediction model. Thus, the existing large language prediction model has no difference in training the model when the output text and the label are different, regardless of how much the difference is. That is, it loses the numerical meaning information in the training process. The embodiment of the present application converts the numerical text part in the output text of the large language prediction model forward process from the text form to the numerical form before calculating the loss. Thus, the calculated loss is a loss with numerical meaning, which avoids losing the numerical meaning information as in the prior art.

[0051] In the embodiment, the specific calculation process of the loss can be as follows: 1) Format check loss (Format Loss), define a format correctness indication function: The loss when the format is incorrect is: Wherein, is a format error penalty coefficient, which can be set to be relatively large to return a relatively large loss , is a regular cross-entropy loss.

[0052] 2) Numerical prediction loss (Numerical Loss) When the format is correct, parse the output numerical part , and calculate the loss with the target value : Wherein, is an error function (such as mean squared error), is a predicted value, and examples of commonly used numerical error functions are: Mean Squared Error (MSE) Formula: ​MSE is the average of the squared errors between predicted and actual values. It is sensitive to outliers because larger errors are penalized more after being squared. MSE helps the model converge quickly during training but can be affected by outliers.

[0053] Mean Absolute Error (MAE) Formula: MAE is the average of the absolute errors between predicted and actual values. Compared to MSE, MAE is not sensitive to outliers because it only considers the absolute value of errors. MAE is more robust in the presence of outliers.

[0054] It should be noted that the above mean square error and mean absolute error are only two examples, and other suitable numerical error functions can also be used, and the present application does not limit this.

[0055] 2.1 Symbolic Expectation If The expected value of a symbol (such as -) is: where is the predicted probability of the symbol character.

[0056] 2.2 Numeric Expectation For each numeric character position The expected value is: where is the score vector of the numeric part of the model output.

[0057] 2.3 Numerical Resolution Decompose into integer and decimal parts to calculate the predicted value: where and are the number of bits for the integer and decimal parts, respectively.

[0058] 3) Complete Loss Function Combine the format loss and numerical loss, and optionally add a regularization term: That is: where is the vector of all parameters of the model, is the loss of the non-numeric part.

[0059] In one example embodiment, referring to Figure 3 , the numerical meaning alignment operation of step 103 can also be implemented through steps 301 to 302.

[0060] Step 301, based on the difference in the size of the numerical values represented on different bits, a first weight is given to the word on each bit in the numerical text output.

[0061] Step 302, based on the first weight, the loss between the numerical text output and the target output is calculated.

[0062] It should be noted that, as mentioned above, the numerical training data is converted into text form by utilizing the contextual understanding ability of the large language prediction model, so that the existing large language prediction model loses the numerical meaning information when calculating the loss. Among them, the loss of numerical meaning information includes: in numerical representation, the weight of each bit is different, and the weight of the unit place is 10 times higher than that of the tenth place, while the large language prediction model treats each character as the same weight. The embodiment assigns different weights to different bits of text or characters to solve the problem of loss of numerical meaning information of different bits by the large language prediction model.

[0063] Further, when setting the first weight, the weight corresponding to the high bit number is greater than the weight corresponding to the low bit number, for example, the first weight of the word corresponding to the tenth bit is greater than the first weight of the word corresponding to the unit bit, and similarly, the first weight of the word corresponding to the hundredth bit is greater than the first weight of the word corresponding to the tenth bit.

[0064] Specifically, the first weight of the word corresponding to the tenth bit can be set to be 10 times the first weight of the word corresponding to the unit bit, and similarly, the first weight of the word corresponding to the hundredth bit can be set to be 10 times the first weight of the word corresponding to the tenth bit, and the first weight of the word corresponding to the unit bit can be set to be 10 times the first weight of the word corresponding to the first bit after the decimal point, and so on. For example, refer to the table below.

[0065] The above is only a simple example of the specific setting method of the first weight, of course, other specific methods can also be used to set the first weight, as long as the principle that the weight corresponding to the high bit number is greater than the weight corresponding to the low bit number is followed, and the present application does not limit this.

[0066] In one example embodiment, based on the above embodiment, it further includes: a second weight is given to each word in the numerical text output based on the gap between the word and its corresponding target output.

[0067] In addition to the different numerical value meaning information of different bits, the numerical value meaning information also includes different numerical value sizes on the same bit, and the text processing manner of the large language prediction model also loses this part of the numerical value meaning. The purpose of setting the second weight in this embodiment is to solve the problem of the loss of this part of the numerical value meaning.

[0068] It can be understood that when the second weight is set, the greater the distance from the target output, the smaller the weight. For example, a word in the first numerical text output is 3, the word in the second numerical text output is 5, and the word corresponds to 6 in the target output (i.e., the label). Therefore, the second weight given to the word in the second numerical text output is greater than the second weight given to the word in the first numerical text output.

[0069] Further, the second weight can be given to each word in the numerical text output based on a Gaussian distribution; wherein the Gaussian distribution mean of each word is the numerical value of the corresponding bit in the target output. For details, see Figure 4 , the mean of the Gaussian distribution is the label number, and the standard deviation adjusts the width of the distribution. For example, the numerical label is 4, and the second weights of all numbers from 1 to 9 are set according to the Gaussian distribution in Figure 4

[0070] Referring to Figure 5 , the step 302 of the previous embodiment specifically includes the following steps 501 to 502.

[0071] Step 501, based on the first weight and the second weight, determining the comprehensive weight of each word in the numerical text output.

[0072] Step 502, based on the comprehensive weight of each word in the numerical text output, calculating the loss between the numerical text output and the target output.

[0073] In one example embodiment, referring to Figure 6 , the step 102 can be specifically implemented by the following steps 601 to 602.

[0074] Step 601, inputting the input information into the forward process of the large language prediction model to obtain the prediction distribution of each word in the output; the prediction distribution represents a score vector based on a dictionary.

[0075] Step 602, for each word, based on its score vector, calculating the predicted value of the word in a weighted average manner to obtain the output text.

[0076] Specifically, each word in the text output can be predicted according to the following formula: wherein,​ represent the prediction value of the i th word, the probability of the i th word being a numerical value .

[0077] Due to the principle of output of a large language prediction model, it outputs a score vector corresponding to the i th output word. In order to fully preserve information, if it is a number, it is not directly used, but the score given by the model for each number is calculated . Here we only consider the output number, so is taken as a slice corresponding to the numbers 0~9 , . Based on the recalculated numerical value, a new numerical value sequence is obtained.

[0078] In a large model, the forward process refers to the process of input data starting from the input layer of the model, passing through the calculation and transformation of each hidden layer in turn, and finally reaching the output layer and producing output. This is a forward information transmission process. When a prompt or question is input to a large model, after the forward process, the model will generate corresponding text as an answer. This generated text is the output text. For example, when we input "Yesterday's temperature was 32 degrees, the temperature this morning is 26 degrees, and the weather is sunny now, what will the temperature be at noon today?", it outputs "The temperature at noon today may be 31 degrees" after forward calculation, which is the output text. The score vector of each word represents the score of the word that may appear at each position during the generation of the output text by the model. Each element in the score vector corresponds to a word in the vocabulary, and its numerical value represents the size of the possibility that the model considers the word to appear at the current position. For example, when generating the sentence "The temperature at noon today may be 31 degrees", for the "1" position, the model will give a score vector, in which the score of the "1" word may be higher, and the scores of some irrelevant words are lower. The embodiment optimizes the numerical value part of the output text, which is not the traditional method of taking the highest element in the score vector as the word at that position, but rather a weighted average of the numbers that may appear at that position and their probabilities, and the weighted average value is taken as the word at that position. For example, the score vector for a certain position indicates that the probability of 5 at that position is 0.5, the probability of 4 is 0.2, the probability of 6 is 0.1, the probability of 3 is 0.1, and the probability of 7 is 0.1. According to the traditional processing method, the word at that position will be determined as 5, which will lose part of the information. To avoid this loss, the embodiment calculates as follows: 5x0.5+4x0.3+3x0.1+2x0.1=4.2≈4, and finally takes 4 as the word at that position.

[0079] In this embodiment, the specific calculation process of the loss can be as follows: A weight sequence W is generated for each word of the sequence S, the non-numeric part is all 1, and the numeric part is assigned a value according to its position in Y. For example:

[0080] = wherein, is the first weight of the s th token, is the second weight of the s th token, W s is the comprehensive weight of the s th token.

[0081] A label vector L is generated for each word of the sequence S, the non-numeric part is all hard labels, and the numeric part is generated by using a Gaussian kernel with a mean value of Si corresponding to the numeric value and a variance of a preset value σ to generate a weight, and using Softmax for normalization as a soft label.

[0082] wherein, W s is the comprehensive weight of the s th token, I(s) is a function that is 0 when the s th token is a number, and 1 otherwise.

[0083] Standard cross-entropy loss (hard label) For hard labels (One-Hot encoding), the cross-entropy loss is: wherein is the probability predicted by the model, is the logits.

[0084] Soft label cross-entropy loss When the label is a probability distribution (soft label), the loss function is directly extended as: In an example embodiment, when calculating the loss value L= , to increase the stability of gradient calculation, the loss value can be normalized. As can be seen from the above calculation process, all operations are derivable operations, and the loss value can be passed to each output digit score (The same applies to the + / - characters.) Therefore, the entire model can be trained end-to-end.

[0085] The present invention also provides a numerical task prediction method, see Figure 7 The method is implemented by following steps 701 to 702.

[0086] Step 701: Obtain a numerical prediction task; the numerical prediction task includes input information and a problem to be predicted.

[0087] Step 702: Input the input information and the problem to be predicted into the large language prediction model trained in the above embodiment of the present invention to obtain a numerical prediction output.

[0088] For example, in stock price prediction, input information could include historical prices, market indices, stock fundamentals, and analyst reviews. The problem to be predicted could be the stock's end-of-period price or rate of return for a specific period in the future. For example, in electricity consumption prediction, input information could include historical electricity consumption, ambient temperature, and weather conditions. The problem to be predicted could be electricity consumption for a specific period in the future.

[0089] The embodiment of the present invention obtains a numerical prediction task based on a large language prediction model that has undergone a numerical meaning alignment operation, and the prediction results are more accurate.

[0090] The following describes the numerical task large language prediction model construction device and the numerical task prediction device provided by the present invention. The numerical task large language prediction model construction device and the numerical task prediction device described below can be referenced to each other with the numerical task large language prediction model construction method and the numerical task prediction method described above.

[0091] See also Figure 8 ,The numerical task large language prediction model construction device includes the ,following modules.

[0092] The data set acquisition module 801 is used to acquire a sample data set corresponding to the numerical prediction task; each sample in the sample data set includes: input information and target output; An output text acquisition module 802 is configured to input the input information into a forward process of a large language prediction model to obtain output text of the forward process; the output text is formatted based on a prompt word template, wherein the prompt word template is used to specify an index interval corresponding to a numerical text; A numerical text output extraction module 803 is configured to extract the numerical text output from the output text based on an index interval corresponding to the numerical text; An alignment operation module 804 is used to perform a numerical meaning alignment operation on the numerical text output and calculate the loss between the output after the alignment operation and the target output; The model training module 805 is configured to train the large language prediction model based on the loss.

[0093] Referring to Figure 9 The numerical task prediction device includes the following modules.

[0094] The numerical prediction task acquisition module 901 is configured to acquire a numerical prediction task; the numerical prediction task includes input information and a problem to be predicted. The numerical prediction output module 902 is configured to input the input information and the problem to be predicted into the large language prediction model of the embodiment of the application to obtain numerical prediction output.

[0095] Figure 10 An example of an entity structure diagram of an electronic device is shown in Figure 10 As shown, the electronic device can include a processor 1010, a communications interface 1020, a memory 1030, and a communications bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communications bus 1040. The processor 1010 can invoke the logical instructions in the memory 1030 to execute the numerical task large language prediction model construction method or the numerical task prediction method.

[0096] The numerical task large language prediction model construction method includes: acquiring a sample data set corresponding to a numerical prediction task; each sample in the sample data set includes input information and a target output; inputting the input information into a forward process of a large language prediction model to obtain an output text of the forward process; the output text is a text formatted based on a prompt word template, and the prompt word template is used to specify an index interval corresponding to a numerical text; extracting a numerical text output in the output text based on the index interval corresponding to the numerical text; performing an alignment operation on the numerical text output in terms of numerical meaning, and calculating a loss between the output after the alignment operation and the target output; training parameters of the large language prediction model based on the loss.

[0097] The numerical task prediction method includes: acquiring a numerical prediction task; the numerical prediction task includes input information and a problem to be predicted; inputting the input information and the problem to be predicted into the large language prediction model constructed by the large language prediction model construction method of the embodiment of the application to obtain numerical prediction output.

[0098] Further, the logic instructions in the memory 1030 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0099] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the numerical task large language prediction model construction method or the numerical task prediction method provided by the above-mentioned methods.

[0100] The numerical task large language prediction model construction method comprises: obtaining a sample data set corresponding to a numerical prediction task; each sample in the sample data set comprises: input information and target output; inputting the input information into a forward process of a large language prediction model to obtain an output text of the forward process; the output text is a text formatted based on a prompt word template, and the prompt word template is used to specify an index interval corresponding to a numerical text; extracting a numerical text output in the output text based on the index interval corresponding to the numerical text; performing an alignment operation on the numerical text output in terms of numerical meaning, and calculating a loss between the output after the alignment operation and the target output; and training parameters of the large language prediction model based on the loss.

[0101] The numerical task prediction method comprises: obtaining a numerical prediction task; the numerical prediction task comprises input information and a to-be-predicted question; inputting the input information and the to-be-predicted question into the large language prediction model constructed by the large language prediction model construction method described in the embodiments of the present application to obtain a numerical prediction output.

[0102] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the numerical task large language prediction model construction method or the numerical task prediction method provided by the above-mentioned methods.

[0103] The numerical task large language prediction model construction method comprises: obtaining a sample data set corresponding to a numerical prediction task; each sample in the sample data set comprises: input information and target output; inputting the input information into a forward process of a large language prediction model to obtain output text of the forward process; the output text is a text formatted based on a prompt word template, and the prompt word template is used to define an index interval corresponding to a numerical text; extracting a numerical text output in the output text based on the index interval corresponding to the numerical text; performing an alignment operation on the numerical text output in terms of numerical meaning, and calculating a loss between the output after the alignment operation and the target output; training parameters of the large language prediction model based on the loss.

[0104] The numerical task prediction method comprises: obtaining a numerical prediction task; the numerical prediction task comprises input information and a to-be-predicted question; inputting the input information and the to-be-predicted question into a large language prediction model constructed by the large language prediction model construction method to obtain a numerical prediction output.

[0105] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0106] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0107] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a large language prediction model for numerical tasks, characterized in that: include: Get the sample data set corresponding to the numerical task; Each sample in the sample data set includes: input information and target output; Inputting the input information into a forward process of a large language prediction model to obtain output text of the forward process; the output text is text formatted based on a prompt word template, wherein the prompt word template is used to specify an index interval corresponding to the numerical text; Extracting the numerical text output in the output text based on the index interval corresponding to the numerical text; Performing a numerical alignment operation on the numerical text output, and calculating a loss between the output after the alignment operation and the target output; The large language prediction model is trained based on the loss.

2. The method for constructing a large language prediction model for numerical tasks according to claim 1, characterized in that: Performing a numerical alignment operation on the numerical text output, and calculating a loss between the output after the alignment operation and the target output, including: Analyzing the numerical meaning of the numerical text output based on floating-point number composition rules to obtain a numerical output; A numerical loss between the numerical output and the target output is calculated.

3. The method for constructing a large language prediction model for numerical tasks according to claim 2, characterized in that: The numerical meaning of the numerical text output is parsed based on floating-point number composition rules, including: The numerical meaning of the numerical text output is parsed according to the following formula: in, Indicates numerical output; Indicates the positive or negative sign in the output of numerical text; Indicates the bit mark, The representation rule is: for the text before the decimal point in the numerical text output, from back to front, the ones place corresponds to , the tens digit corresponds to , hundreds place corresponds to , and so on, for the text after the decimal point in the numerical text output, from front to back, the first decimal point corresponds to , the second decimal place corresponds to , the third decimal place corresponds to , and so on; Indicates that the numerical text is output The numeric value corresponding to the text in the bit.

4. The method for constructing a large language prediction model for numerical tasks according to claim 1, characterized in that: Performing a numerical alignment operation on the numerical text output, and calculating a loss between the output after the alignment operation and the target output, including: Assigning a first weight to each word in the numerical text output based on the difference in magnitude of the numerical value represented at different bits; A loss between the numerical text output and a target output is calculated based on the first weight.

5. The method for constructing a large language prediction model for numerical tasks according to claim 4, characterized in that: Performing a numerical alignment operation on the numerical text output and calculating a loss between the output after the alignment operation and the target output, further comprising: assigning a second weight to each word in the numerical text output based on a distance between the word and its corresponding target output; Calculating the loss between the numerical text output and the target output based on the first weight includes: determining a comprehensive weight of each word in the numerical text output based on the first weight and the second weight; Based on the comprehensive weight of each word in the numerical text output, the loss between the numerical text output and the target output is calculated.

6. The method for constructing a large language prediction model for numerical tasks according to claim 5, characterized in that: Assigning a second weight to each word in the numerical text output based on a distance between the word and its corresponding target output, comprising: A second weight is assigned to each word in the numerical text output based on Gaussian distribution; wherein the Gaussian distribution mean of each word is the numerical value of the corresponding bit in the target output.

7. The method for constructing a large language prediction model for numerical tasks according to claim 1, characterized in that: Inputting the input information into a forward process of a large language prediction model to obtain output text of the forward process includes: Inputting the input information into the forward process of a large language prediction model to obtain a predicted distribution of each word in the output; the predicted distribution represents a score vector based on the dictionary; For each word, based on the word's score vector and the dictionary, a weighted average method is used to calculate the word's prediction value to obtain the output text.

8. A numerical task prediction method, characterized in that: include: Obtaining a numerical prediction task; the numerical prediction task includes input information and a problem to be predicted; The input information and the problem to be predicted are input into the large language prediction model according to any one of claims 1 to 7 to obtain a numerical prediction output.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, it implements the method for constructing a large language prediction model for numerical tasks as described in any one of claims 1 to 7, or the method for predicting numerical tasks as described in claim 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a large language prediction model for a numerical task as described in any one of claims 1 to 7 or the method for predicting a numerical task as described in claim 8 is implemented.

Images