Large language model numerical value type input word segmentation optimization method and system and application
By constructing a quantitative numerical table and expanding it to the tokenizer vocabulary table of large language model, the problem of word segmentation too long when processing numerical type features is solved, and more efficient training and more feature processing capabilities are achieved.
Patent Information
- Application Number
- CN202510400579.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-01
- Publication Date
- 2025-08-12
AI Technical Summary
When large language models deal with numerical type features, word segmentation devices divide each numerical into multiple tokens, resulting in an increase in input length, high training cost and limited number of acceptable features, especially in data-intensive fields such as financial risk control.
The quantized numerical table is constructed and expanded to the tokenizer vocabulary table. The numerical values between [0,1] are quantified by dynamic tree quantization method. After normalization, the numerical features are input tokenizer for word segmentation, reducing the number of tokens after word segmentation.
It effectively reduces the length of input word segmentation, saves training resource consumption, increases the number of numerical features that can be processed by large language models, and is suitable for numerical intensive industries such as finance.
Smart Images
Figure HDA0005339683500000011 
Figure HDA0005339683500000012
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence and word segmentation optimization technology, and relates to a large language model numerical type input word segmentation optimization method, system and application. Background Art
[0002] In recent years, large language models have developed rapidly and have shown great application value and potential in various data-intensive industries. For example, in the financial industry, large models can be used for financial knowledge retrieval and question answering, company financial report interpretation, market forecasting, and risk control decision-making assistance. [1] Among them, credit assessment and fraud detection are very important parts of risk control, and how to combine them with large language models is also being explored. [2] . Through data platforms such as Kaggle's public datasets, we can find that many credit assessment and fraud detection datasets contain a large number of numerical inputs. In some datasets, each sample may contain hundreds of numerical features. When applying large language models to deal with such problems, since the large language model word segmenter training mainly uses text inputs such as English or Chinese for training, when the numerical value is directly input into the word segmenter of the large language model, each numerical type will be divided into multiple words corresponding to multiple index values. When there is a lot of floating-point data input, the input length and training cost will be greatly increased. Therefore, most of the public datasets such as credit assessment and fraud detection that are currently processed by large language models contain relatively few numerical features, and at most only contain dozens of numerical features. However, in real risk control scenarios, the number of numerical features contained in real data is usually far more than dozens.
[0003] For example, datasets commonly used in credit assessment and fraud detection in the financial industry typically include many numerical features. When using large language models to solve datasets with many such numerical features, the presence of many numerical values in the input data significantly increases the length of the input after word segmentation. Furthermore, large language models have a limited acceptable context length. For example, the Llama series of large models accept input word segmentation lengths of 1024 and 2048. Therefore, increasing word segmentation length not only increases training and inference costs, but also limits the number of features that large models can accept when solving problems such as credit assessment and fraud detection. Summary of the Invention
[0004] In order to address the deficiencies in the prior art, the present invention aims to provide a large language model numerical input word segmentation optimization method, system and application.
[0005] In response to the problems existing in the above-mentioned prior art, the present invention proposes a large language model numerical input word segmentation optimization method. The method constructs a quantized numerical table and expands the quantized numerical table into the tokenizer vocabulary. Then, the input numerical type features are preprocessed and normalized, quantized to the numerical values in the quantized numerical table, and then input into the tokenizer for word segmentation. Because the original large language model tokenizer is usually trained using text type data, a numerical type input is usually divided into multiple word segments, especially floating point type data. Moreover, in data-intensive industries, such as the financial industry, data sets are usually in the form of tables. When processing data, feature data preprocessing, normalization, etc. are usually performed. The normalized data range is between [0, 1]. Therefore, the present invention quantizes the numerical values between [0, 1] into a quantized numerical table and expands the quantized numerical table into the tokenizer vocabulary of the large language model. After the normalized feature data is quantized and input into the tokenizer, it will no longer be segmented into multiple tokens. Each numerical data corresponds to only one token.
[0006] The construction of the quantization value table in the present invention can select the appropriate quantization bit width and the size of different value tables according to the specific range of data in the actual scenario, the requirements for data accuracy, the size of the data set, and the computing power resources. For example, the present invention will statistically analyze the distribution interval of the normalized data, use traditional machine learning methods to test the accuracy that needs to be retained, and select the corresponding quantization vocabulary size according to the accuracy requirements.
[0007] In a specific implementation process, the selection of the bit width for constructing the quantized numerical table can be determined based on the accuracy requirements of the relevant data set, as well as the sensitivity and computing power resources. The bit width will determine the size of the numerical table. The larger the bit width, the higher the accuracy. The larger the size of the numerical table, the larger the vocabulary of the model, the larger the weight dimension that needs to be expanded in the model embedding layer, and the more computing power required. The degree of influence of different accuracies on the results can be judged through experiments (such as the machine learning method used in the present invention, which can quickly judge the accuracy requirements of the data set), and the appropriate bit width can be selected based on the degree of time and resource consumption.
[0008] During the implementation of the method of the present invention, different methods can be used to construct the quantization value table, and different sizes of the quantization value table can be used according to specific data conditions and requirements.
[0009] The large language model numerical input word segmentation optimization method of the present invention includes the following steps:
[0010] Step 1: quantize the values between [0, 1] using the dynamic tree quantization method and construct a quantization value table;
[0011] Step 2: Expand the quantized value table constructed in step 1 to the large language model tokenizer vocabulary;
[0012] Step 3: Based on the expanded tokenizer vocabulary, expand the weight dimension of the embedding layer of the large language model and randomly initialize the weight of the expanded part;
[0013] Step 4: Process and quantify the numerical features in the input data set;
[0014] Step 5: Input the processed and quantified numerical type features into the expanded tokenizer for word segmentation to reduce the total number of tokens after word segmentation.
[0015] In step 1, the value is represented as a sign bit, an exponent bit, and a mantissa bit, and the interval (0.1, 1) is recursively bisected using a binary tree, and the mantissa is encoded using a binary tree path to reduce the mean relative error; and / or,
[0016] A dynamic exponent mechanism is introduced to represent the exponent 10^(-n) with n leading 0 bits; the first bit set to 1 is a flag indicating that the following bits are part of a binary tree.
[0017] In one embodiment, the quantization value table uses a 14-bit quantization bit width and includes dynamic exponents of 0 to 13 bits;
[0018] Traverse the exponent bits from 0 to 13 and calculate the number of mantissas corresponding to each exponent bit; evenly divide the (0.1,1) interval to obtain the mantissa interval list, and obtain the final mantissa list by calculating the average of adjacent numbers in the mantissa interval list; generate representable values based on different exponent bits and mantissa lists and add them to the quantization value table; traverse all exponent bit situations and expand the interval endpoint values of 0 and 1 to the quantization value table.
[0019] Of the 14-bit quantization bit width, 1 bit is used to represent the sign, and the remaining 13 bits are used to represent the exponent and mantissa. The mantissa is used to represent the binary tree path of the interval (0.1, 1), and the tree is split into two parts through the split path. Each child node represents the starting and ending points of the interval.
[0020] The segmented intervals are selected through the tree path to ensure that the relative error in a smaller numerical range is small.
[0021] In the specific implementation process of the present invention, the size and construction method of the quantitative value table can be determined according to the specific data conditions and requirements in the implementation.
[0022] In step 2, the quantized value table constructed in step 1 is added to the original tokenizer vocabulary to expand the original tokenizer vocabulary. Each numerical data after quantization corresponds to only one token.
[0023] In step 3, the weight dimension of the embedding layer of the large language model is expanded; the expanded weight dimension is the same as the expanded tokenizer vocabulary size dimension; and / or,
[0024] In step 3, random initialization is performed using methods including Xavier uniform distribution, Xavier normal distribution, Kaiming uniform distribution, Kaiming normal distribution, uniform distribution, and normal distribution to facilitate subsequent training;
[0025] In a specific embodiment, the weight of the expansion part is initialized using Kaiming normal distribution.
[0026] In step 4, the numerical features in the data set are processed, including preprocessing such as missing value filling, outlier deletion, replacement, or retention, and normalization using methods such as Min-Max.
[0027] In this invention, outliers are observations that differ significantly from the majority of data points. These values may arise due to various reasons, including but not limited to data entry errors, extreme market events, unusual transactions, system errors, or unique market conditions. They can be identified using statistical methods such as Z-scores or IQR (interquartile range) methods, or through boxplots, scatterplots, histograms, and machine learning. Outliers can be deleted, replaced, or retained, depending on the specific circumstances of the dataset.
[0028] Specifically, in one embodiment, missing values in the input data can be supplemented by the median of the feature column, and outliers can be replaced by the mean or a fixed value of the feature column. When extreme data values, such as those that are too large or too small, appear in the input data set, they are treated as outliers and handled according to the outlier processing method.
[0029] Step 4 may specifically include the following sub-steps:
[0030] Step 4.1: Distinguish and filter the features in the input dataset based on numerical and categorical features.
[0031] Step 4.2: Process the filtered numerical type features and map them to the interval [0, 1].
[0032] Step 4.3: Match the processed values in step 4.2 with the constructed quantization value table, find the quantization value with the smallest error, and replace the quantized numerical features.
[0033] In the specific implementation process of the present invention, only the numerical type features are preprocessed and normalized, and the category type features are directly segmented by the original word segmenter.
[0034] In the present invention, the input dataset of the large language model needs to be constructed using a processed dataset. The construction process includes: constructing a prefix description task and output results according to the task type of the dataset, and guiding prompt generation.
[0035] The word segmentation optimization method in the present invention is applicable to all numerically intensive industries, including the financial field, and is particularly suitable for processing tabular data with a large number of numerical features, and has wide application value.
[0036] The present invention also provides a word segmentation optimization system for implementing the above-mentioned word segmentation optimization method, wherein the word segmentation optimization system comprises: a quantization value table construction module, a word table expansion module, a model weight adjustment module, a numerical feature preprocessing module, and a word segmentation processing module;
[0037] The quantization value table construction module is used to quantize the values in the [0,1] interval based on the dynamic tree quantization method, and construct a quantization value table to represent values of different precisions through a binary tree path;
[0038] The vocabulary expansion module is used to expand the quantized value table to the tokenizer vocabulary of the large language model, so that each quantized value corresponds to a unique token and is compatible with the original vocabulary;
[0039] The model weight adjustment module is used to adjust the embedding layer weights of the large language model to make its dimensions consistent with the size of the expanded tokenizer vocabulary, and to perform random initialization and training optimization on the newly added embedding layer weights;
[0040] The numerical feature preprocessing module is used to screen, normalize and quantize the numerical type features in the input data, and map the normalized values to the optimal values in the quantization value table;
[0041] The word segmentation processing module is used to perform word segmentation processing on the quantized numerical input after the expansion of the tokenizer to reduce the number of tokens of the numerical type input and optimize the computational efficiency of the large language model in the numerical processing task.
[0042] The present invention also provides a hardware system for implementing the above-mentioned word segmentation optimization method, the hardware system comprising: a memory and a processor; a computer program is stored on the memory, and when the computer program is executed by the processor, the above-mentioned word segmentation optimization method is implemented.
[0043] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned word segmentation optimization method is implemented.
[0044] The present invention also provides the application of the above-mentioned word segmentation optimization method, the above-mentioned word segmentation optimization system, the above-mentioned hardware system, or the above-mentioned computer-readable storage medium in data-intensive fields to optimize the number of numerical type input word segments, reduce input length and resource consumption, etc.
[0045] The beneficial effects of the present invention include: the existing large language model word segmenter is usually trained by text type data during training. When facing numerical type input, especially floating point type data, a numerical value will be divided into multiple tokens. For example, many public data sets in data-intensive fields such as financial risk control contain many features, some of which include hundreds of features, most of which may be numerical type features. When these numerical features are input into the large language model tokenizer, each numerical value will be divided into multiple tokens, so the input length after tokenizer word segmentation becomes several times the number of original features. When the input numerical features are large, the input length will be greatly increased, which will not only increase the training time and the consumption of training resources, but also make it impossible for a data set containing a large number of features to input all features into the large language model due to the limited context length acceptable to the large language model. The present invention constructs a quantized numerical table and expands it to the large language model tokenizer vocabulary, normalizes and quantizes the numerical features, makes the data correspond to the data in the quantized numerical table, so that each numerical feature corresponds to only one token after passing through the tokenizer, avoiding the problem caused by the numerical features being divided into multiple tokens. In the experiment of the present invention, the original data set had 3,000 numerical features. After word segmentation, the maximum feature length exceeded 10,000 and could not be fully input into the model. After optimization, the word segmentation length was only 3,000 and could be fully input into the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0047] Figure 1 This is a flowchart of the numerical type word segmentation optimization method for large language models.
[0048] Figure 2 It is a flowchart for constructing a quantitative value table. DETAILED DESCRIPTION
[0049] The present invention is further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for those specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.
[0050] The present invention mainly provides a method for optimizing word segmentation of numerical type input of a large language model, and belongs to the field of artificial intelligence and word segmentation optimization technology. In order to reduce the number of word segments of numerical type data in the input, the method of the present invention constructs a quantization numerical table by selecting an appropriate quantization bit width; then expands the quantization numerical table into the tokenizer vocabulary; expands the weight of the embedding layer of the large language model to a size corresponding to the size of the tokenizer vocabulary; quantizes the numerical type data in the data set as needed; and uses the processed data set to fine-tune the embedding layer and other layers. The technical solution in the present invention is particularly aimed at data sets containing a large number of numerical types, such as data-intensive industries. Using the word segmentation optimization method of the present invention can greatly reduce the input word segmentation length, avoid exceeding the model input length limit and save computing costs. Specifically, the method of the present invention proposes a word segmentation optimization scheme for the input of a large language model with a large number of numerical type data. After optimization, each numerical type data will only be mapped into one token, effectively reducing the length of the input after word segmentation, saving training and inference costs. In addition, the acceptable text length of the large language model is limited, and this method can also enable the large language model to accept a larger number of numerical type features.
[0051] The innovation of this invention mainly lies in proposing a method for reducing the word segmentation length of a large language model for numerical input in a specific application scenario with a large number of numerical inputs. Taking into account the characteristic that numerical type features can be quantized and mapped, a quantized numerical table is constructed to map the input, and the tokenizer vocabulary of the large language model is expanded and fine-tuned. This effectively solves the problem of excessive input word segmentation length in application scenarios with a large number of numerical inputs, such as financial data.
[0052] The large language model numerical type input word segmentation optimization method of the present invention first quantizes the numerical values between [0,1] to construct a quantization numerical table. Considering that uniform quantization will lead to relatively large numerical errors in the interval, the present invention adopts a dynamic tree quantization method. [3]This method allows smaller values within an interval to be represented using fewer values. The implementation is as follows: First, the number of bits to be used is determined, with one bit reserved for the sign of the value and the remaining bits used for the exponent and mantissa. The mantissa bits are then used to represent a binary tree with the interval (0.1, 1), with the tree partitioned according to the path through the tree; the child nodes thus represent the start and end points of the interval in the partition. This method allows the mantissa to cover a wider range of numbers, thereby reducing the mean relative error. Furthermore, additional bits can be used for the exponent, and a dynamic exponent can be introduced, where n leading zero bits represent the exponent 10^(-n); the first bit set to 1 is a flag indicating that the following bits are part of a binary tree. While this format loses the ability to represent large exponents and a bit is lost to the binary tree, it allows the ability to approximate numbers with large absolute values with less error, while retaining the ability to approximate numbers with small absolute values and few significant digits.
[0053] like Figure 2 As shown, in the specific implementation process, the data is normalized, and the accuracy of the data to be retained is tested using a machine learning method, and the quantization bit width and the size of the quantization vocabulary are determined according to the accuracy.
[0054] In the process of constructing a specific quantization value table, a 14-bit quantization bit width is used, where the number of dynamic exponent bits is 0 to 13. The specific process is as follows:
[0055] Step a: traverse the number of exponent digits from 0 to 13, and calculate the number of representable mantissas according to the number of exponent digits;
[0056] Step b, evenly divide the interval (0.1, 1) according to the number of mantissas calculated in step a, and obtain a list of intervals of mantissas;
[0057] Step c, obtaining the mean of two adjacent numbers in the mantissa interval list in step b to obtain a mantissa list;
[0058] Step d: Based on the current exponent bit and the mantissa list obtained in step c, all values that can be represented by the current exponent bit value interval are obtained and added to the quantized value table;
[0059] Step e: traverse all possible cases of the number of exponent bits and expand the two interval endpoints of 0 and 1 into the quantization value table to obtain the final quantization value;
[0060] In a specific embodiment of the present invention, the size of the quantization value table is 16384.
[0061] After completing the construction of the quantization value table, expand the quantization value table into the tokenizer vocabulary corresponding to the large language model;
[0062] In a specific embodiment of the present invention, the Llama2 large language model is used, the original tokenizer vocabulary size is 128256, the quantization value table size is 16384, and the expanded tokenizer vocabulary size is 144640.
[0063] Based on the expanded tokenizer vocabulary size, the embedding layer weight dimensions of the large language model are expanded to the corresponding dimension of the tokenizer vocabulary size, and the weights of the expanded part are randomly initialized. The original Llama2 embedding layer weight dimension is 128256*4096, and the expanded dimension is 144640*4096.
[0064] The reason for random initialization is that the weight dimension of the embedding is expanded. The expanded part needs to be initialized for subsequent training. Common initialization methods include: Xavier uniform distribution, Xavier normal distribution, Kaiming uniform distribution, Kaiming normal distribution, uniform distribution, normal distribution, etc.
[0065] In a specific embodiment of the present invention, Kaiming normal distribution is used for initialization because the original weight data distribution table is close to normal distribution.
[0066] Perform data processing on the data set used as input. The specific process is as follows:
[0067] Step i: First, distinguish the features in the data set according to numerical type features and category type features, and filter out numerical type features;
[0068] Step ii: perform basic preprocessing and normalization on the numerical type features in step i.
[0069] In the present invention, the preprocessing mainly includes processing such as missing value supplementation and outlier replacement, and the normalization method includes the Min-Max normalization method.
[0070] In a specific embodiment, missing values in the input data are replaced by the median of the column where the feature is located by default, and outliers are replaced by the mean or a fixed value of the column where the feature is located; if extreme data such as too small or too large appears, it is treated as an outlier.
[0071] Step iii: quantize the normalized numerical features in step ii. The specific process is to first obtain the absolute error between the numerical features to be quantized and each number in the quantization numerical table, and then replace the numerical feature to be quantized with the value corresponding to the position with the minimum absolute error in the quantization numerical table.
[0072] In the present invention, the processed and quantified data sets are used to construct [2] The input dataset that meets the requirements of the large language model is as follows:
[0073] Step I: Construct a prefix describing the task and optional output results based on the task type of the current dataset;
[0074] Step II: Based on whether the dataset gives the specific meaning of each feature, decide whether to add a description of the meaning of each feature to the prompt. If it does, each feature description includes the feature meaning explanation and feature value; otherwise, it only includes the feature name and feature value.
[0075] Example 1
[0076] This example was experimented on the Kaggle public dataset LendingClub. The specific results are as follows:
[0077] (1) First, the data set is preprocessed, which mainly includes feature type screening, missing value filling, screening out numerical type features, normalizing according to feature dimensions, and then quantizing according to the quantization value table. The original numerical type features in a certain sample include [329.9, 0.1016, 329.56, 10200.0, 11870.0, 0.0, 0.0, 0.0, 7.0, 0.797, 16.0, 695.0, 699.0, 95 000.0], and the corresponding values after normalization are [0.2046389132744998, 0.18886292834890966, 0.008211035266640639, 0.23589743589743586, 0.02822091723924776, 0.0, 0.0, 0.0, 0.10714285714285715, 0.69912280 7017543, 0.15053763440860216, 0.200000000000000018, 0.199999999999999973, 0.1302497504688991], the corresponding values after quantization are [0.20464476943016052, 0.18882447481155396, 0.008209228515625, 0.23 584595322608948,0.02822631597518921,0,0,0,0.10719604790210724,0.699139416217804,0.15059202909469604,0.20003050565719604,0.20003050565719604,0.13026732206344604].
[0078] (2) Constructing prompts. The task of the LendingClub dataset is a binary classification task of credit assessment, so the prefix description constructed on this dataset is "Assess the client's loan status based on the following loan records from Lending Club. Respond with only 'good' or 'bad', and do not provide any additional information. For instance, 'The client has astable income, no previous debts, and owns a property.' should be classified as 'good'." This dataset gives the specific meaning of each feature, so the description of the feature includes the meaning of each feature.Taking the sample in (1) as an example, the feature text description format is: "The client has attributes as follows: The state of Installment is 329.9. The state of Loan Purpose is credit_card. The state of Loan Application Type is Individual. The state of InterestRate is 0.1016. The state of Last Payment Amount is 329.56. The state of LoanAmount is 10200.0. The state of Revolving Balance is 11870.0. The state of Delinquency In 2years is 0.0. The state of Inquiries In 6Months is 0.0. The state of Mortgage Accounts is 0.0. The state of Grade is B. The state of OpenAccounts is 7.0. The state of Revolving Utilization Rate is 0.797. The state of Total Accounts is 16.0. The state of Fico Range Low is 695.0. The state of Fico Range High is 699.0. The state of Address State is AZ. The state of Employment Length is 10+years. The state of Home Ownership is RENT. The state of Verification Status is Not Verified. The state of Annual Income is 95000.0."。
[0079] (3) Taking the sample in (2) as an example, the original tokenizer is used to segment the text input consisting of unnormalized and unquantized numerical data, and the segmentation length is 311. The tokenizer with the expanded vocabulary is used to segment the text input consisting of quantized numerical features, and the segmentation length is 280. If only numeric data is used as input, the prompts are "329.9, 0.1016, 329.56, 10200.0, 11870.0, 0.0, 0.0, 0.0, 7.0, 0.797, 16.0, 695.0, 699.0, 95000.0" and "0.20464476943016052, 0.18882447481155396, 0.008209228515625, 0.23584595322608948, 0.0 The word segmentation lengths of the two are 59 and 28 respectively. It can be seen that the word segmentation length is reduced by 31 when the input data only contains 14 numerical features.
[0080] Example 2
[0081] The present invention provides a training and inference-friendly word segmentation optimization method for large financial models. This method constructs a quantized numerical table, expands the large language model tokenizer vocabulary, and quantizes the numerical type features of the input data. This ensures that each numerical type input feature corresponds to only one token after word segmentation. This effectively improves the problem of the input length increasing exponentially with the number of numerical type features due to the original tokenizer dividing the numerical type input into multiple tokens. This can not only reduce the resource consumption of training and inference, but also enable the large language model to accept a larger number of numerical type features.
[0082] The present invention is mainly aimed at some data sets in actual scenarios that include a large number of features, which may be as high as thousands, and most of which are numerical inputs. Due to the current context length of large language models, it is impossible to input all features into the model or the resource consumption is greatly increased due to the long input length. Through the method of the present invention, the large language model can accept a larger number of features as input to meet the needs of actual scenarios. Because originally all features could not be used as input, after using this method, all features can be input into the model.
[0083] References
[0084] [1]Xie Q, Han W, Zhang
[0085] [2]Feng D, Dai Y, Huang J, et al. Empowering many, biasing a few: Generalist credit scoring through large language models [J]. arXiv preprintarXiv:2310.00566, 2023.
[0086] [3]Dettmers T.8-bit approximations for parallelismin deep learning[J].arXiv preprint arXiv:1511.04561,2015.
[0087] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.
[0088] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0089] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0091] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.
[0092] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.
[0093] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.
Claims
1. A large language model numerical input word segmentation optimization method, characterized in that: The word segmentation optimization method comprises the following steps: Step 1: quantize the values between [0, 1] using the dynamic tree quantization method and construct a quantization value table; Step 2: Expand the quantized value table constructed in step 1 to the large language model tokenizer vocabulary; Step 3: Based on the expanded tokenizer vocabulary, expand the weight dimension of the embedding layer of the large language model and randomly initialize the weight of the expanded part; Step 4: Process and quantify the numerical features in the input data set; Step 5: Input the processed and quantified numerical type features into the expanded tokenizer for word segmentation to reduce the total number of tokens after word segmentation.
2. The word segmentation optimization method according to claim 1, wherein: In step 1, the value is represented as a sign bit, an exponent bit, and a mantissa bit, and the interval (0.1, 1) is recursively bisected using a binary tree, and the mantissa is encoded using a binary tree path to reduce the mean relative error; and / or, A dynamic exponent mechanism is introduced, with n leading 0 bits representing the exponent 10^(-n); the first bit set to 1 indicates that the following bits are part of a binary tree.
3. The word segmentation optimization method according to claim 2, wherein: The quantization value table uses a 14-bit quantization bit width and includes dynamic exponents from 0 to 13 bits; Traverse the exponent bits from 0 to 13 and calculate the number of mantissas corresponding to each exponent bit; evenly divide the interval (0.1, 1) to obtain the mantissa interval list, and calculate the average of adjacent numbers in the mantissa interval list to obtain the final mantissa list; based on different exponent bits and mantissa lists, generate representable values and add them to the quantized value table; Traverse all the cases of exponent digits and expand the interval endpoint values of 0 and 1 into the quantization value table.
4. The word segmentation optimization method according to claim 1, wherein: In step 2, the quantized value table constructed in step 1 is added to the original tokenizer vocabulary to expand the original tokenizer vocabulary. Each numerical data after quantization corresponds to only one token.
5. The word segmentation optimization method according to claim 1, wherein: In step 3, the weight dimension of the embedding layer of the large language model is expanded; the expanded weight dimension is the same as the expanded tokenizer vocabulary size dimension; and / or, Random initialization is performed through methods including Xavier uniform distribution, Xavier normal distribution, Kaiming uniform distribution, Kaiming normal distribution, uniform distribution, and normal distribution.
6. The word segmentation optimization method according to claim 1, wherein: In step 4, the numerical features in the data set are processed, including missing value filling, outlier deletion or replacement or retention preprocessing, and normalization including Min-Max method; and / or, Missing values in the input data are supplemented by the median of the feature column, and outliers are replaced by the mean or fixed value of the feature column. When extreme data values appear in the input data set, they are treated as outliers and handled according to the outlier handling method. and / or, Step 4 includes the following sub-steps: Step 4.1: Distinguish and filter the features in the input dataset based on numerical and categorical features. Step 4.2: Process the filtered numerical type features and map them to the interval [0, 1]. Step 4.3: Match the processed values in step 4.2 with the constructed quantization value table, find the quantization value with the smallest error, and replace the quantized numerical features.
7. A word segmentation optimization system for implementing the word segmentation optimization method according to any one of claims 1 to 6, characterized in that: The word segmentation optimization system includes: a quantization value table construction module, a word table expansion module, a model weight adjustment module, a numerical feature preprocessing module, and a word segmentation processing module; The quantization value table construction module is used to quantize the values in the [0,1] interval based on the dynamic tree quantization method, and construct a quantization value table to represent values of different precisions through a binary tree path; The vocabulary expansion module is used to expand the quantized value table to the tokenizer vocabulary of the large language model, so that each quantized value corresponds to a unique token and is compatible with the original vocabulary; The model weight adjustment module is used to adjust the embedding layer weights of the large language model to make its dimensions consistent with the size of the expanded tokenizer vocabulary, and to perform random initialization and training optimization on the newly added embedding layer weights; The numerical feature preprocessing module is used to screen, normalize and quantize the numerical type features in the input data, and map the normalized values to the optimal values in the quantization value table; The word segmentation processing module is used to perform word segmentation processing on the quantized numerical input after the expansion of the tokenizer to reduce the number of tokens of the numerical type input and optimize the computational efficiency of the large language model in the numerical processing task.
8. A hardware system for implementing the word segmentation optimization method according to any one of claims 1 to 6, characterized in that: The hardware system includes: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the word segmentation optimization method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the word segmentation optimization method according to any one of claims 1 to 6 is implemented.
10. Application of the word segmentation optimization method according to any one of claims 1 to 6, the word segmentation optimization system according to claim 7, the hardware system according to claim 8, or the computer-readable storage medium according to claim 9 in optimizing the number of word segments of numerical input in data-intensive fields, reducing input length and resource consumption.