A method and system for constructing an ultra-high-order n-gram language model based on an additivity hash count table

Abstract

The application discloses a method and system for constructing an ultra-high-order N-gram language model based on an additivity hash counting table, and belongs to the technical field of natural language processing. The method maps a variable-length context into a fixed bit width state identifier through a rolling hash mechanism, constructs a sparse statistical matrix composed of a context hash value, a word token identifier and a co-occurrence frequency triple, and realizes low time complexity context matching and probability prediction. The model supports independent training of multiple sources of corpus, completes knowledge fusion through linear weighting, has a completely combinable model superposition characteristic, introduces a rollback strategy in the reasoning stage to improve robustness, and trims entries with a frequency lower than a preset threshold before deployment to compress the model volume. N-order language modeling is realized under low resource consumption, various natural language processing tasks can be supported, the model can independently and efficiently run on a general central processing unit device without special acceleration hardware, deployment cost and energy consumption are significantly reduced, and the model constitutes a lightweight alternative scheme for large language models.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method and system for constructing a high-order N-gram language model based on an additive hash counting table. This invention is applicable to achieving efficient and composable language understanding and generation in hardware environments with limited computing resources, and can be deployed on general computing devices or private servers. Background Technology

[0002] Traditional N-gram language models are based on the Markov assumption and utilize the prior... Predicting the conditional probability of the next word from a given word, although simple in structure and easy to implement, results in a large number of parameters that increase with the order of the word. This exponential growth leads to the "curse of dimensionality" problem. For example, when the vocabulary size is 50,000, a 5-gram model theoretically needs to store... The combination of these parameters far exceeds the capacity of existing storage systems. Therefore, in practical applications, the order is usually limited to 1. This severely restricts the ability to model context.

[0003] In recent years, large-scale language models based on the Transformer architecture have achieved breakthrough progress in many natural language processing tasks by realizing the modeling of long-distance contextual dependencies through self-attention mechanisms. However, such models have the following inherent defects: (1) huge computational resource consumption, with training and inference processes relying on hundreds of GB or even TB of GPU memory capacity, as well as server clusters composed of multiple high-end GPUs or tensor processors; (2) significant energy consumption, with the training process of a single model consuming thousands of kilowatt-hours of electricity and huge carbon emissions; (3) high model fine-tuning costs, with new domain knowledge needing to be achieved through full parameter fine-tuning or complex parameter efficient fine-tuning techniques, and different models lack explicit algebraic combination capabilities; (4) poor interpretability, with prediction results difficult to trace back to specific training data.

[0004] While existing research optimizes the efficiency of large language models through techniques such as model compression (e.g., quantization, pruning, knowledge distillation), approximate retrieval (e.g., locality-sensitive hashing), and key-value caching mechanisms, these methods are still based on deep neural network frameworks and fail to fundamentally solve core problems such as high computational complexity, lack of composability, and weak interpretability. Therefore, a new language modeling paradigm is urgently needed that can overcome the order limitations of traditional N-gram models, circumvent the resource bottlenecks of large models, and simultaneously possess composability, updability, and interpretability. Summary of the Invention

[0005] This invention aims to solve the following three key technical problems existing in the prior art:

[0006] Question 1: How to achieve ultra-high-order language modeling without a neural network architecture?

[0007] Traditional N-ary models cannot be extended to higher orders due to the exponential growth of the number of parameters. However, this invention uses rolling hash compression technology to map variable-length contexts to fixed-width state identifiers, constructing a sparse statistical matrix and achieving theoretically infinite-order modeling capability.

[0008] Question 2: How to achieve knowledge fusion of multiple language models without retraining?

[0009] Large-scale language models lack explicit algebraic combinatorics. This invention proposes an additivity hash counting table structure, which allows statistical tables trained on different corpora to be directly superimposed through linear weighting, achieving modular combination and preference control.

[0010] Question 3: How to complete the language understanding and generation tasks of large-scale language models on resource-constrained devices?

[0011] This invention constructs a pure lookup table-based reasoning process, avoiding floating-point matrix operations, achieving millisecond-level response on a general-purpose central processing unit, and supporting model pruning and compression to meet deployment requirements.

[0012] To address the aforementioned problems, this invention proposes a method for constructing ultra-high-order N-gram language models based on an additivity hash counting table. Its core concept lies in replacing fitting with statistics, explicit storage with hashing, and gradient fine-tuning with additivity, thereby constructing a lightweight, transparent, and modular language modeling system. The specific steps are as follows:

[0013] Step 1: Obtain the text corpus and perform preprocessing and word segmentation, converting all text into word sequence;

[0014] Step 2: Training the distributed hash counting table: Define a rolling hash function to map a word sequence of arbitrary length to a 32-bit integer state identifier. Construct a two-dimensional sparse counting table based on the rolling hash function. Each entry in the table includes a context hash value, word identifier, and co-occurrence frequency. A multi-granularity context queue mechanism is used to maintain the rolling hash queue in parallel.

[0015] Step 3: Perform overflow sorting through a combined training and sorting merging mechanism using both internal and external memory;

[0016] Step 4: Use multiple independent corpora to train independently to obtain corresponding sub-count tables, and then weight and fuse all sub-count tables to obtain a global count table;

[0017] Step 5: Compress the model using frequency pruning and variable-length encoding;

[0018] Step 6: In the inference phase, receive the user input context sequence, query the counter table based on the current context hash value to obtain the distribution of candidate lemmas, and generate subsequent text by combining the backoff mechanism and decoding strategy.

[0019] Furthermore, the rolling hash function adopts the Rabin-Kap type rolling hash algorithm, and its recursive formula is:

[0020]

[0021] in, Represents a rolling hash function. For the k-th word element, It is a prime number base. This is a function that maps tokens to integer codes. Indicates the modulus to be taken .

[0022] Furthermore, the parallel maintenance of the rolling hash queue using a multi-granularity context queue mechanism is as follows:

[0023] The two-dimensional sparse counting table Each entry is a triple It means that among them For context hash value, For word identifiers, V represents the set of lexical terms, indicating co-occurrence frequency.

[0024] For the input sentence ,in For the Lth word element, Parallel maintenance of sentence length rolling hash state ,in Indicates from the first Subsequence starting with a lexical unit The hash values ​​correspond to context prefixes starting from different starting positions at each generation position. , will the next word The frequency of each value is accumulated into the hash buckets corresponding to all levels, i.e., execution... ,in .

[0025] Furthermore, the overflow sorting through the internal and external memory collaborative training and sorting merging mechanism is specifically as follows:

[0026] Based on the pre-allocated open addressing hash table in the device memory, a quadratic probing method is used to resolve collisions; the frequency is dynamically accumulated in memory, and an overflow is triggered when the load factor exceeds a threshold; upon overflow, all valid triples are extracted and sorted by radix: Main sequence The radix sort is performed in multiple rounds, with 16 bits of data processed in each round.

[0027] The sorting results are written to a temporary file and finally merged into a globally sorted array using a min-heap-driven multi-way merge algorithm.

[0028] Furthermore, the reasoning stage is specifically as follows:

[0029] Step 6.1: Lookup-driven autoregressive text generation: Receive initial context The initial state is calculated using rolling hashing. ; at every step Based on the current hash value Query count table Obtain the distribution of candidate lexical units; use a decoding strategy to select the next lexical unit. Output the tokens and update the hash state;

[0030] Step 6.2: Rollback Mechanism: When the counter table If it does not exist, iteratively remove the context first word to generate a reduced-order context hash value. Query the counting table Until it hits, among them Indicates before removing the original context sequence The rolling hash value of the remaining sequence after 1000 words. The number of backsteps, when When the length equals the original context length, it falls back to a unary syntax distribution;

[0031] Introducing a backoff coefficient calibration function Weighting the distribution at each level to satisfy The probability estimation formula is:

[0032]

[0033] in, For the i-th word element, Let be the summation variable, representing the vocabulary. Any word in The smoothing parameter is used; the total probability is the product of the conditional probabilities at each step; the distributions at each level are passed through the backoff coefficient function. calibration.

[0034] Furthermore, the frequency pruning and the variable-length encoding are specifically as follows:

[0035] Frequency clipping: Set threshold Remove all tables that satisfy the count. Entries;

[0036] Variable-length encoding: Files are stored according to frequency range. Use 1 byte, Use 2 bytes, Use 4 bytes Using 4-byte floating-point numbers, the total length of the table entries is 7–10 bytes.

[0037] This invention also provides a high-order N-gram language model system based on an additive hash counting table, used to implement the above method, deployed on a general-purpose central processing unit hardware platform, capable of independently completing natural language processing tasks without the acceleration of a graphics processor or tensor processor, including:

[0038] The rolling hash calculation module is used to convert variable-length input contexts into fixed-width status identifiers;

[0039] A sparse counting table storage module is used to store triples in the form of an ordered array;

[0040] The table lookup and probability estimation module is used to obtain the distribution of candidate word elements based on binary search;

[0041] The fallback mechanism module is used to handle context-based degraded queries in case of a miss.

[0042] The decoding and generation module is used to perform autoregressive text generation.

[0043] The present invention has the following outstanding advantages compared with the prior art:

[0044] Order scalability: Theoretically supports infinite-order N-ary modeling, breaking through the limitation of traditional methods where n≤5;

[0045] Resource efficiency: The inference process only requires table lookup operations and can run on a general-purpose CPU;

[0046] Full composability: linear fusion of multiple models, supporting modular development and incremental updates;

[0047] High interpretability: Each prediction can be traced back to a specific context hash value and statistical frequency;

[0048] Low training cost: The training cost of a corpus of tens of billions of word units is only 1 / 100 of that of a large model;

[0049] Real-time responsiveness: Average response latency <200ms, meeting the requirements for real-time interaction;

[0050] Privacy and security: Supports fully offline deployment; data does not need to be uploaded to the cloud.

[0051] Environmental friendliness: Carbon emissions are reduced by three orders of magnitude compared to larger models. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the overall process of constructing the ultra-high-order N-gram language model according to the present invention.

[0053] Figure 2A schematic diagram illustrating rolling hash state updates and multi-granularity context queue maintenance.

[0054] Figure 3 The flowchart shows the collaborative training and multi-way merging of the sparse counting table's internal and external memory.

[0055] Figure 4 This is a logical block diagram of the autoregressive generation and backoff mechanism in the inference stage.

[0056] Figure 5 This is a schematic diagram of the model pruning and variable-length encoding storage structure. Detailed Implementation

[0057] To make the technical solution, structural features and achieved technical effects of the present invention clearer and more complete, the technical solution of the present invention will now be described and explained in detail with reference to the accompanying drawings and specific embodiments.

[0058] Example 1:

[0059] A method for constructing a high-order N-gram language model based on an additive hash counting table, such as... Figure 1 As shown, the overall process includes: the corpus preprocessing stage converts the original text into a sequence of word IDs; the training stage constructs a multi-granularity counting table through rolling hash calculation and uses an internal and external memory collaborative mechanism to process large-scale data; the fusion stage merges multiple specialized models through linear weighting; the pruning and compression stage reduces storage volume through frequency thresholds and variable-length encoding; and the inference stage completes text generation through table lookup operations combined with a rollback mechanism.

[0060] Includes the following steps:

[0061] Step 1: Obtain the text corpus and perform preprocessing and word segmentation, converting all text into word sequence;

[0062] Step 2: Training the distributed hash counter table. For example... Figure 2 As shown, for those containing The system maintains the input sentence with 100 words in parallel. A rolling hash state queue. Each queue corresponds to a context window with a different starting position, and the hash value is updated in real time using a recursive formula. The hash states at all levels are simultaneously incremented in the corresponding bucket of the counting table to form a multi-level contextual representation.

[0063] Step 2.1: Design of rolling hash function and state compression mechanism;

[0064] Define a rolling hash function Maps arbitrary length word sequences to 32-bit integer state identifiers, where V * This represents the set of word sequences. The Rabin-Kap rolling hash algorithm is preferred, and its recursive formula is:

[0065]

[0066] in It is a prime number base, and its range is 1. Preferably, it is 31 or 101; This is a function that maps tokens to integer codes; the modulus is taken as... This mechanism fully utilizes the 32-bit processor registers to improve computational efficiency. Its key advantage lies in the fact that the hash value maintains a fixed bit width regardless of changes in context length, and the update operation has a time complexity of O(n log n). This enables the model theory to support infinite-order N-ary modeling.

[0067] Step 2.2: Construct a multi-granularity sparse counting table and maintain a multi-granularity rolling hash queue in parallel.

[0068] Based on the hash mechanism described above, a two-dimensional sparse counting table is constructed. Each entry in the table is a triple. It means that among them For context hash value, For word identifiers, V represents the set of lexical terms, indicating co-occurrence frequency. Unlike traditional N-gram models that only count fixed-length prefixes, this invention employs a multi-granularity context queue mechanism during the training phase:

[0069] For the input sentence ,in For the Lth word element, Sentence length; parallel maintenance rolling hash state ,in Indicates from the first Subsequence starting with a lexical unit The hash values ​​correspond to context prefixes starting from different positions. At each generation position... , will the next word The frequency of each value is accumulated into the hash buckets corresponding to all levels, i.e., execution... ,in For example, in the generation location , will the next word The frequency of each value is accumulated and added to the corresponding hash buckets at all levels:

[0070]

[0071]

[0072]

[0073]

[0074] This design establishes a multi-layered contextual representation in the hash space, enabling the model to naturally support a fallback mechanism: when a higher-order context is not found, a lower-order distribution can be queried directly without additional parameter fitting. Simultaneously, all levels share a unified hash space and table structure, ensuring that additivity is maintained.

[0075] Step 3: Perform overflow sorting through a combined training and sorting merging mechanism using both internal and external memory;

[0076] Considering that a complete counting table may exceed memory capacity, this invention adopts a combined internal and external memory training strategy:

[0077] (a) Based on the pre-allocated open addressing hash table in the device memory, a quadratic probing method is used to resolve collisions;

[0078] (b) The frequency is dynamically accumulated in memory, and an overflow is triggered when the load factor exceeds the threshold (75%);

[0079] (c) When overflow occurs, extract all valid triples and perform radix sort: by Main sequence The radix sort is performed in multiple rounds, with 16 bits of data processed in each round.

[0080] (d) Write the sorting results to a temporary file, and finally merge them into a globally ordered array using a min-heap driven multi-way merge algorithm.

[0081] This mechanism supports the efficient construction of TB-level models in external storage without loading all data into memory, and the training process can be scaled up to hundreds of billions of word units.

[0082] The triples are stored using an ordered array structure, ordered by Main sequence The order is arranged to support the reasoning phase through binary search. A table lookup operation with time complexity. This is to count the total number of items in the table.

[0083] like Figure 3 As shown, an overflow process is triggered when the load factor of the memory hash table reaches 75%. The system will then process the triples in memory. After extraction, perform radix sorting, by 32 bits. 16-bit value The values ​​are hierarchically sorted to generate temporary files. Finally, a min-heap-driven multi-way merge algorithm is used to merge multiple temporary files into a globally ordered array, supporting binary search.

[0084] Step 4: Independent training and weighted fusion strategy for multi-source corpora;

[0085] This invention supports training counting tables from multiple independent corpora. ( ), such as general corpus Instruction fine-tuning table Domain-specific tables Since all sub-tables have the same dimensional structure, they can be seamlessly merged into a global counting table through element-wise addition. :

[0086]

[0087] The symbols in the formula are defined as follows: This is the merged global count table; The number of independent corpora; For the first The weighting coefficients corresponding to each corpus; For the first Local counting tables obtained independently from each corpus, and... They have the same hash space and lexical dimension.

[0088] Among them, the weighting coefficient The relative importance of various knowledge types can be adjusted manually or through validation set optimization. For example, the weight of the instruction corpus can be set to... Interval values, general corpus weights set to This allows the model to more readily generate responses that conform to instructions, mimicking the effects of reinforcement learning without requiring complex gradient updates. This feature also endows the model with full composability, supporting modular development and feature customization.

[0089] Step 5: Model trimming and compression storage strategy:

[0090] like Figure 5 As shown, for storage-constrained environments, this invention provides a two-level compression mechanism:

[0091] (1) Frequency clipping: Set a threshold Remove all that meet the criteria. The entries, experiments show that when It can compress the volume by more than 99% without causing a performance drop of no more than 3% on the test set;

[0092] (2) Variable-length encoding: Files are stored according to frequency range. Use 1 byte, Use 2 bytes, Use 4 bytes Using 4-byte floating-point numbers, the total length of the table entries is 7–10 bytes.

[0093] This strategy allows models trained with hundreds of billions of words to be compressed from 32TB to 420GB, making them suitable for deployment on high-memory servers.

[0094] Step 6: During the inference phase, receive the user input context sequence and query the counter table based on the current context hash value. Obtain the distribution of candidate lexical units, and combine a backtracking mechanism and decoding strategy to generate subsequent text. For example... Figure 4 As shown, after receiving user input during the inference phase, the initial context hash value is first calculated. , In each generation step, the system first queries the counter table entry corresponding to the higher-order context. If a match is not found, a fallback mechanism is activated to calculate the lower-order hash value. (Before removal) (The sequence hash of each lexical term) continues until a hit is found or the unary syntax distribution is reverted. Upon a hit, the next lexical term is selected using a decoding strategy, and the hash state is updated to proceed to the next round of generation.

[0095] Step 6.1: Contextualization and Autoregressive Generation Mechanism: Providing a lookup-driven autoregressive text generation method:

[0096] (1) Receive initial context The initial state is calculated using rolling hashing. ;

[0097] (2) In each step Based on the current hash value Query count table Obtain the distribution of candidate lexical elements;

[0098] (3) Use a decoding strategy to select the next word. These include: greedy search, beam search, random sampling, or Top-k sampling;

[0099] (4) Output the tokens and update the hash state: ;

[0100] (5) Repeat steps (2)–(4) until the termination condition is met: encountering the end marker, reaching the length limit, or detecting a repeating pattern.

[0101] This process is entirely based on table lookup operations, with a single-step response time of less than 1 millisecond, making it suitable for real-time interactive scenarios.

[0102] Step 6.2: Backoff mechanism and probability estimation method.

[0103] To address misses caused by low-frequency context, this invention designs a prefix removal backoff mechanism: when If it does not exist, iteratively remove the context first word to generate a reduced-order context hash value. Query Until it hits. (Among them) Indicates the process The reduced-order context hash value obtained after the step-back operation, i.e., before removing the original context sequence. The rolling hash value of the remaining sequence after 1000 words. The number of backsteps, when When the length equals the original context length, fall back to a unary syntax distribution. .

[0104] Introducing a backoff coefficient calibration function Weighting the distribution at each level to satisfy The probability estimation formula is:

[0105]

[0106] in Let be the summation variable, representing the vocabulary. Any word element in the vocabulary (dummy variables when performing normalization calculations by traversing the entire vocabulary). The smoothing parameter has a range of values. The total probability is the product of the conditional probabilities at each step. The distributions at each level can be determined using the backoff coefficient function. calibration, Preferred is about the number of backtracking steps It is an exponentially decaying function.

[0107] This embodiment also provides a high-order N-gram language model system based on an additive hash counting table. The system is deployed on a general-purpose central processing unit (CPU) hardware platform and can independently complete natural language processing tasks without the need for a graphics processor or tensor processor acceleration. It includes:

[0108] The rolling hash calculation module is used to convert variable-length input contexts into fixed-width status identifiers;

[0109] A sparse counting table storage module is used to store triples (h,t,n) in the form of an ordered array;

[0110] The table lookup and probability estimation module is used to obtain the distribution of candidate word elements based on binary search;

[0111] The fallback mechanism module is used to handle context-based degraded queries in case of a miss.

[0112] The decoding and generation module is used to perform autoregressive text generation;

[0113] Example 2: Overall system construction process.

[0114] This embodiment describes the complete process from corpus preparation to system deployment, and verifies the feasibility of the present invention in a real environment.

[0115] Step 1: Corpus preprocessing and word segmentation.

[0116] The general text corpus tiiuae / falcon-refinedweb (total 1.68TB) and the instruction fine-tuning corpus stanford_alpaca_52K (52,000 instruction-output pairs) were selected. The Falcon model vocabulary (vocabulary size) was used. The tokenizer converts all text into a sequence of token IDs.

[0117] Step 2: Training the distributed counting table.

[0118] The training was performed in parallel on 16 PC servers equipped with Intel Xeon Gold 6248R processors (24 cores, 3.00 GHz) and 256 GB of memory. Each server processed approximately 100 GB of corpus subsets, maintaining an open-addressable hash table in memory (initial capacity). (Hundreds of millions of items). Adopted As a base of prime numbers As a smoothing parameter This serves as a pruning threshold. During training, overflow sorting is triggered when the load factor reaches 75%, generating temporary files. On average, a single server takes approximately 12 days to process 100GB of data, and training a complete corpus takes approximately 15 days.

[0119] Step 3: Global merging and weighted fusion.

[0120] The local count tables of 16 servers are merged using the min-heap multi-way merge algorithm to generate... (Approximately 32TB in size). The merge process uses radix sort, processing 16 bits per round, for a total of 4 rounds to complete the 48-bit key-value pair (32 bits). +16-bit The sorting and merging process was completed on a management node equipped with 512GB of memory and took approximately 48 hours.

[0121] Independently trained on the stanford_alpaca_52K corpus (Approximately 800MB in size), weighted by percentage Weighted fusion, i.e., execution .

[0122] Step 4: Model trimming and compression

[0123] Prune the merged count table: remove all frequencies. The number of entries ranges from approximately Down to A variable-length encoding strategy is employed: approximately 99.3% of the entries are frequency-sensitive. 0.6% of entries are stored as 1 byte. The first byte is stored as 2 bytes; the rest is stored as 4 bytes. The final model size is compressed to 420GB, with a compression rate of 98.7%.

[0124] Step 5: Inference Deployment and Performance Testing

[0125] The model was deployed on a single server equipped with an AMD EPYC 7763 processor (64 cores, 2.45GHz) and 512GB of DDR4-3200 memory. Model loading time was 28 seconds, and memory usage remained stable between 480–500GB. A reasonable response rate of 95% was achieved on the AlpacaEval instruction compliance test set, with an average response time of 200ms per request. The system supports context windows of up to 4096 words and maintains stable performance even with more than 20 consecutive dialogue rounds.

[0126] Example 3: Lightweight Deployment of Edge Devices

[0127] This example illustrates the deployment scheme of the model in an environment with extremely limited resources.

[0128] A subset of the Common Crawl corpus (100GB) was selected to train the base model, with the vocabulary limited to 32,000 words. Aggressive pruning was performed after training, with K=5, reducing the model size to 12GB. On a MacBook Pro equipped with an Apple M2 chip (8-core CPU, 16GB unified memory), memory mapping was used to load model fragments on demand. Although the initial token latency increased to approximately 800ms, subsequent generation speed remained at 150ms / token, supporting offline text generation and basic question-answering functions, with peak memory usage not exceeding 14GB.

[0129] Example 4: Application of Multilingual Model Combination

[0130] This embodiment verifies the composability of the model.

[0131] Training English counting tables separately (Based on The Pile corpus) and programming language table (Based on GitHubCodeSearchNet). The two tables are 180GB and 90GB in size respectively. A multi-language code assistant model is constructed through weighted fusion: The weights are allocated based on the size of each corpus and the importance of the task. The fused model achieves a 13.2% pass rate on the HumanEval programming test set, demonstrating its ability to combine multi-domain knowledge without retraining.

[0132] The core innovation of this invention lies in the synergistic effect of the hash compression mechanism and the additivity statistical architecture: rolling hash maps the theoretically infinite parameter space to... The fixed-dimensional sparse matrix enables ultra-high-order modeling; the multi-granularity context queue explicitly encodes backoff paths at each level during training, eliminating the need for dynamic calculation of reduced-order distributions during inference; the internal and external memory coordination mechanism combining open-addressable hashing and radix sort solves the memory bottleneck of large-scale corpus training; linear additivity transforms model combination into a simple element-wise weighted summation, avoiding the complexity and instability of gradient backpropagation. These four elements together constitute a complete closed-loop system encompassing training, fusion, inference, and pruning.

[0133] This invention is applicable to the following technical scenarios:

[0134] Private AI Assistant: Deployed within the enterprise intranet to process sensitive business data;

[0135] Mobile terminal application: Integrated into smartphones to achieve offline text generation;

[0136] Edge computing nodes: Perform local semantic understanding on IoT gateways;

[0137] In the fields of scientific research and education: it serves as a research platform for interpretable language models.

[0138] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for constructing a high-order N-gram language model based on an additive hash counting table, characterized in that, Includes the following steps: Step 1: Obtain the text corpus and perform preprocessing and word segmentation, converting all text into word sequence; Step 2: Training the distributed hash counting table: Define a rolling hash function to map a word sequence of arbitrary length to a 32-bit integer state identifier. Construct a two-dimensional sparse counting table based on the rolling hash function. Each entry in the table includes a context hash value, word identifier, and co-occurrence frequency. A multi-granularity context queue mechanism is used to maintain the rolling hash queue in parallel. Step 3: Perform overflow sorting through a combined training and sorting merging mechanism using both internal and external memory; Step 4: Use multiple independent corpora to train independently to obtain corresponding sub-count tables, and then weight and fuse all sub-count tables to obtain a global count table; Step 5: Compress the model using frequency pruning and variable-length encoding; Step 6: In the inference phase, receive the user input context sequence, query the counter table based on the current context hash value to obtain the distribution of candidate lemmas, and generate subsequent text by combining the backoff mechanism and decoding strategy.

2. The method for constructing a high-order N-gram language model based on an additive hash counting table according to claim 1, characterized in that, The rolling hash function adopts the Rabin-Kap type rolling hash algorithm, and its recursive formula is as follows: in, Represents a rolling hash function. For the k-th word element, It is a prime number base. This is a function that maps tokens to integer codes. Indicates the modulus to be taken .

3. The method for constructing a high-order N-gram language model based on an additive hash counting table according to claim 2, characterized in that, The parallel maintenance of the rolling hash queue using a multi-granularity context queue mechanism is as follows: The two-dimensional sparse counting table Each entry is a triple It means that, among them For context hash value, For word identifiers, V represents the set of lexical terms, indicating co-occurrence frequency. For the input sentence ,in For the Lth word element, Parallel maintenance of sentence length rolling hash state ,in Indicates from the first Subsequence starting with a word The hash values ​​correspond to context prefixes starting from different starting positions at each generation position. , will the next word The frequency of each value is accumulated into the hash buckets corresponding to all levels, i.e., execution... ,in .

4. The method for constructing a high-order N-gram language model based on an additive hash counting table according to claim 3, characterized in that, The overflow sorting process, which utilizes a combined internal and external memory training and sorting merging mechanism, is described in detail below: Based on the pre-allocated open addressing hash table in the device memory, a quadratic probing method is used to resolve collisions; the frequency is dynamically accumulated in memory, and an overflow is triggered when the load factor exceeds a threshold; upon overflow, all valid triples are extracted and sorted by radix: by Main sequence The radix sort is performed in multiple rounds, with 16 bits of data processed in each round. The sorting results are written to a temporary file and finally merged into a globally sorted array using a min-heap-driven multi-way merge algorithm.

5. The method for constructing a high-order N-gram language model based on an additive hash counting table according to claim 4, characterized in that, The reasoning stage is as follows: Step 6.1: Lookup-driven autoregressive text generation: Receive initial context The initial state is calculated using rolling hashing. ; at every step Based on the current hash value Query count table Obtain the distribution of candidate lexical units; use a decoding strategy to select the next lexical unit. Output the tokens and update the hash state; Step 6.2: Rollback Mechanism: When the counter table If it does not exist, iteratively remove the context first word to generate a reduced-order context hash value. Query the counting table Until it hits, among them Indicates before removing the original context sequence The rolling hash value of the remaining sequence after 1000 words. The number of backsteps, when When the length equals the original context length, it falls back to a unary syntax distribution; Introducing a backoff coefficient calibration function Weighting the distribution at each level to satisfy The probability estimation formula is: in, For the i-th word element, Let be the summation variable, representing the vocabulary. Any word in the word, The smoothing parameter is used; the total probability is the product of the conditional probabilities at each step; the distributions at each level are passed through the backoff coefficient function. calibration.

6. The method for constructing a high-order N-gram language model based on an additive hash counting table according to claim 5, characterized in that, The frequency pruning and the variable-length encoding are as follows: Frequency pruning: Set a threshold Remove all tables that satisfy the count. Entries; Variable-length encoding: Files are stored according to frequency range. Use 1 byte, Use 2 bytes, Use 4 bytes Using 4-byte floating-point numbers, the total length of the table entries is 7–10 bytes.

7. The method for constructing a high-order N-gram language model based on an additive hash counting table according to claim 6, characterized in that, Step 4 is as follows: Multiple independent corpora, including a general corpus, an instruction fine-tuning table, and a domain-specific table, are fused together into a global counting table by element-wise addition. All sub-tables have the same hash space and lexical dimension. : ； in, The number of independent corpora; For the first The weighting coefficients corresponding to each corpus; For the first Local counting tables obtained by independently training on each corpus.

8. A high-order N-gram language model system based on an additive hash counting table, characterized in that, The system is used to implement the method as described in any one of claims 1-7, deployed on a general-purpose central processing unit hardware platform, and capable of independently completing natural language processing tasks without the acceleration of a graphics processor or tensor processor, including: The rolling hash calculation module is used to convert variable-length input contexts into fixed-width status identifiers; A sparse counting table storage module is used to store triples in the form of an ordered array; The table lookup and probability estimation module is used to obtain the distribution of candidate word elements based on binary search; The fallback mechanism module is used to handle context-based degraded queries in the event of a miss. The decoding and generation module is used to perform autoregressive text generation.

Images