A method for generating a pinyin bucket word library of a four-level index

Abstract

The present application relates to the technical field of electronic digital data processing, and discloses a four-level index pinyin bucket word library generation method, acquires a Chinese word library text, extracts the first Chinese character and the first pinyin of each line of words as a classification key, and generates a triple; the triple is classified into a corresponding pinyin bucket according to the pinyin, the words in the bucket are sorted in descending order of word frequency, an independent Chinese character block is generated for each unique Chinese character, and the words are solidified according to the word length by using a first character multiplexing mechanism; a four-level static index of an initial letter statistical area and a pinyin index area and a Chinese character word index area and a word storage area is sequentially constructed, each index area uses fixed-length entries and absolute offset addressing; a metadata area containing a file length double-semantic field is constructed, a high-frequency-first deterministic truncation is performed on the word library according to a preset threshold, each data area is spliced and a data integrity check value is appended, and an embedded binary word library is generated. The problems of high storage redundancy and uncontrollable memory are solved, and the purposes of deterministic analysis, resource adaptation and high security are achieved.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a method for generating a four-level indexed Pinyin bucket dictionary. Background Technology

[0002] In human-computer interaction devices such as smartwatches, walkie-talkies, in-vehicle infotainment systems, and industrial HMIs, Chinese input methods need to run stably in environments with extremely small RAM (e.g., less than 64KB), no MMU, no operating system, or only a lightweight RTOS. In actual engineering, the pinyin index structure often uses a Trie tree or hash table, requiring dynamic allocation of memory nodes at runtime to construct matching paths; the word storage area generally adopts the format of "2-byte length field + GB2312 encoding", that is, each word entry starts with a uint16_t representing the number of Chinese characters, followed by the corresponding number of GB2312 encodings, with no terminator at the end; in addition, ASIL-B level software units should have deterministic memory behavior - their RAM usage must be a statically declared value at compile time. Currently, most embedded Chinese input method dictionary files do not contain any integrity verification fields; the loading process is a pure Flash address reading without verifying data validity; once a bit flip, OTA interruption, or media wear occurs, the engine will still perform subsequent parsing, resulting in garbled candidate words, abnormal UI rendering, or MCUHardFault; therefore, there is an urgent need for an input method system-level solution that does not rely on dynamic memory, does not require a length field, has all addresses determined by compile-time formulas, and has a built-in streaming integrity verification mechanism.

[0003] For example, Chinese patent CN101183369A discloses an embedded electronic dictionary lexicon structure, providing the following technical solution: It consists of a dictionary tree layer, an index stream layer, and an information stream layer, wherein the dictionary tree layer is the first layer and resides in memory, the index stream layer and the information stream layer are the second and third layers, respectively; the dictionary tree layer is a tree-shaped index area for quickly locating words; the index stream layer provides a list of words closest to the input word to ultimately locate the word; the information stream layer stores the specific explanation information of the word; the tree nodes of the dictionary tree layer point to their corresponding index nodes in the index stream layer; and the words in the index nodes of the index stream layer point to the explanation information block of that word in the information stream layer. This invention's dictionary lexicon structure is reasonable and effective, with strong independence between layers, fast and practical search, significantly reducing the processor's computational load, reducing the number of matching operations during queries, greatly improving query speed and efficiency, and saving storage space. However, the aforementioned embedded electronic dictionary database structure, which employs a dynamic tree structure, variable-length index stream, and relative offset, cannot achieve first-character reuse, word-length partitioning, or absolute offset addressing. This results in high storage redundancy, numerous parsing branches, poor cross-platform compatibility, and a lack of integrity verification functionality. Summary of the Invention

[0004] This invention solves the problems of high storage redundancy, uncertain parsing, uncontrollable memory, and lack of integrity verification in the prior art. It proposes a four-level indexed method for generating a Pinyin bucket dictionary, achieving deterministic parsing, resource adaptation, and high security.

[0005] Furthermore, this invention aims to provide a fully static, zero dynamic memory dependency, truncation-enabled, and integrity-verified method for generating a pinyin dictionary. By reusing the first character, using fixed-length entries, and absolute offsets, it achieves a binary dictionary file with controllable size and cross-platform compatibility.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for generating a four-level indexed Pinyin bucket dictionary includes: Obtain the Chinese dictionary text, extract the first Chinese character and the first pinyin of each word in each line as the classification key, and generate triplet; The triplets are categorized into corresponding pinyin buckets according to their pinyin, and the words in the buckets are sorted in descending order of word frequency. An independent Chinese character block is generated for each unique Chinese character, and the words are solidified by partitioning according to word length using the first character reuse mechanism. A four-level static index is constructed sequentially, consisting of an initial letter statistics area, a pinyin index area, a Chinese character and word index area, and a word storage area. Each index area uses fixed-length entries and absolute offset addressing. Construct a metadata area containing a dual semantic field of file length, perform deterministic truncation of the lexicon based on a preset threshold prioritizing high frequencies, concatenate the data areas and append data integrity verification values ​​to generate an embedded binary lexicon.

[0007] Using the first Chinese character and the first pinyin as the classification key solves the problem of missing candidate words for polyphonic characters. The first character reuse mechanism greatly compresses the dictionary storage space. The design of four-level static index, fixed-length entries, and absolute offset addressing realizes deterministic access without dynamic memory and branch jumps, which is suitable for resource-constrained embedded environments without an operating system or a lightweight RTOS. The high-frequency priority deterministic truncation makes the dictionary size accurately controllable. The design of dual semantic fields for file length and integrity check value ensures the security and reliability of dictionary loading and meets the functional safety requirements of high reliability scenarios.

[0008] Preferably, the triplet includes words, pinyin, and word frequency. The pinyin uses lowercase ASCII characters without tone. Polyphonic characters are distinguished by separators for multiple pronunciations. When classifying, only the first pinyin is taken as the key. Different pronunciations of the same Chinese character are classified into the corresponding pinyin bucket.

[0009] The standardization of lowercase ASCII characters without tone marks is adapted to the minimalist character processing capabilities of embedded platforms, reducing parsing overhead. The design of using separators to distinguish polyphonic characters with multiple pronunciations and taking only the first pinyin as the classification key ensures that different pronunciations of the same Chinese character can be accurately classified into the corresponding pinyin bucket. This solves the problem of candidate word omission and mismatch caused by polyphonic characters in traditional solutions, ensuring that users can obtain a complete and accurate set of candidate words by inputting any pinyin.

[0010] Preferably, the sorting of words in the bucket by word frequency in descending order specifically includes: the words in the Pinyin bucket are arranged in descending order by word frequency using a stable sorting algorithm, and the input order in the original text is maintained when the word frequencies are the same; when fixing words by word length, the words are divided into four sub-regions according to the number of Chinese characters, including two-character words, three-character words, four-character words, and phrases, and different sub-regions are stored in physical isolation.

[0011] The stable sorting algorithm ensures the reproducibility and auditability of the word library generation results. The descending order of word frequency ensures that high-frequency and commonly used candidate words can be obtained on the first screen after the user inputs the pinyin, improving input efficiency. The word is divided into four sub-areas and physically isolated for storage, which allows the embedded system to adopt the optimal configuration for different word lengths, avoids runtime branch judgment, and achieves deterministic access latency.

[0012] As a preferred embodiment, the method of using the first character reuse mechanism to partition and solidify words by word length specifically includes the following: for two-character words, the sub-region stores only the GBK encoding of the second Chinese character of the word; for three-character words, the sub-region stores only the encoding of the second and third Chinese characters; for four-character words, the sub-region stores only the encoding of the second to fourth Chinese characters. No additional terminator is added to any sub-region, except for the phrase sub-region which uses a double-byte illegal encoding as the termination mark.

[0013] The design of omitting the first Chinese character eliminates coding redundancy, compresses storage space, and significantly reduces the footprint of embedded devices. The design of removing the terminator in the fixed-length sub-region and retaining only the double-byte illegal code as the termination mark in the phrase sub-region not only achieves accurate and fast calculation of word addresses through the fixed-length formula, but also ensures reliable parsing of variable-length phrases, avoiding the uncertainty of execution time caused by terminator scanning, and further improving the determinism and real-time performance of index access.

[0014] Preferably, the initial letter statistics area is a continuous storage space of fixed length, with its storage units corresponding one-to-one with the 26 English letters, and each storage unit records the number of non-empty pinyin bins under the corresponding initial letter.

[0015] Preferably, the pinyin index area consists of several consecutive entries of fixed length, with the total number of entries equal to the total number of non-empty pinyin buckets; each entry includes, in sequence, a fixed-length pinyin string storage space, a field for the total number of Chinese characters corresponding to the pinyin, and a field for the starting offset of the Chinese character entry in the Chinese character word index area. All entries are arranged in ascending order according to the pinyin ASCII code, and empty characters are added if the pinyin string storage space is insufficient.

[0016] Fixed-length consecutive entries enable pure byte comparison in the pinyin matching process, without branching or dynamic parsing, and with fully controllable execution time. Each entry integrates the pinyin string, the total number of Chinese characters, and the starting offset field, enabling direct one-step mapping from the pinyin string to the list of Chinese characters. The design of sorting by pinyin ASCII code in ascending order supports binary search to further improve matching efficiency, and the standard of padding with empty characters ensures the consistency of pinyin comparison results across platforms, avoiding matching errors caused by inconsistent string lengths.

[0017] Preferably, the Chinese character word index area consists of several consecutive entries of fixed length, the total number of entries being equal to the total number of unique Chinese characters in the word database. Each entry includes, in sequence, a fixed-length Chinese character GB code field and a relative address field of the independent Chinese character block corresponding to that character in the word storage area. All entries are arranged in ascending order according to the Chinese character GB code.

[0018] Fixed-length consecutive entries achieve a deterministic mapping from Chinese character encoding to word storage area addresses, allowing address jumps to be completed in a single fixed-length read. Each entry integrates the GB code of the Chinese character with the relative address of the corresponding Chinese character block, establishing a binding between the Chinese character and its exclusive word set. This ensures that when a user selects a Chinese character, only the exclusive candidate words for that character are returned, avoiding semantic pollution across Chinese characters. The design of ascending order of the GB code of the Chinese characters supports fast location of matching Chinese character entries, further improving index parsing efficiency.

[0019] Preferably, the deterministic truncation of the lexicon based on a preset threshold specifically includes truncating the words in the corresponding sub-regions of each Chinese character block by setting a maximum threshold for the number of words of different word length categories, retaining only the words with the first threshold number after descending word frequency, and synchronously updating the corresponding counting field and index entries after truncation.

[0020] By configuring thresholds for different word length categories, the Flash storage resource constraints of different embedded platforms can be accurately adapted, enabling predictable and controllable dictionary size. The high-frequency priority truncation rule compresses the dictionary size while preserving the most frequently used high-frequency candidate words, ensuring that the input experience is not degraded. The design of synchronously updating the count field and index entries after truncation ensures the global consistency of the dictionary index chain, avoiding problems such as address misalignment and parsing errors caused by truncation, and ensuring the stability and reliability of dictionary operation.

[0021] Preferably, the metadata area is a fixed-length contiguous storage space, using big-endian encoding with the most significant byte first, and sequentially storing a fixed-length format magic number field, an ASCII format version number field, a file length field, and the absolute offset fields of the four index areas; the file length field simultaneously carries the dual semantics of valid data length and the starting address of the integrity check value.

[0022] The design of fixed-length contiguous storage space enables single fixed-length reads during dictionary loading, with zero parsing overhead. The big-endian encoding design allows the same dictionary file to be directly loaded on heterogeneous endian and big-endian platforms without the need for field conversion, greatly improving cross-platform compatibility.

[0023] Preferably, the data integrity check value is a fixed-length binary data generated by a standard hash digest algorithm and is physically stored at the end of the binary dictionary file. The calculation scope of the data integrity check value covers all valid data in the metadata area and the four major index areas except for the check value.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows.

[0025] 1. This invention significantly eliminates coding redundancy, greatly compresses the dictionary size, and significantly reduces Flash usage in embedded devices through a storage design that reuses the first character and employs hierarchical termination. Simultaneously, it supports deterministic truncation prioritizing high-frequency elements, precisely adapting to the storage resource constraints of different platforms.

[0026] 2. This invention constructs a four-level static index architecture, employing fixed-length entries and absolute offset addressing to achieve deterministic address jumps without branching or loop scanning, reducing end-to-end word lookup latency. Words within a bucket are arranged in a stable descending order of frequency, allowing users to obtain high-frequency candidate words on the first screen after input, significantly improving input efficiency.

[0027] 3. This invention achieves dual-gateway verification of format validity and compatibility through magic numbers and version numbers, and realizes full data integrity verification with file length as the linchpin. It can accurately identify data corruption and tampering, ensuring reliable boot. The full-field big-endian encoding design allows the same dictionary to be directly loaded across heterogeneous SoC platforms. The address-independent feature supports OTA differential upgrades, resulting in high security and stability. Attached Figure Description

[0028] Figure 1 This is an overall flowchart of a four-level indexed Pinyin bucket dictionary generation method according to the present invention.

[0029] Figure 2 This is a flowchart of the high-frequency priority truncation part of the four-level indexed pinyin bucket dictionary generation method of the present invention. Detailed Implementation

[0030] See Figure 1 and Figure 2 As shown, a method for generating a four-level indexed pinyin bucket dictionary includes: Obtain the Chinese dictionary text, extract the first Chinese character and the first pinyin of each word in each line as the classification key, and generate triplet; The triplets are categorized into corresponding pinyin buckets according to their pinyin, and the words in the buckets are sorted in descending order of word frequency. An independent Chinese character block is generated for each unique Chinese character, and the words are solidified by partitioning according to word length using the first character reuse mechanism. A four-level static index is constructed sequentially, consisting of an initial letter statistics area, a pinyin index area, a Chinese character and word index area, and a word storage area. Each index area uses fixed-length entries and absolute offset addressing. Construct a metadata area containing a dual semantic field of file length, perform deterministic truncation of the lexicon based on a preset threshold prioritizing high frequencies, concatenate the data areas and append data integrity verification values ​​to generate an embedded binary lexicon.

[0031] In this embodiment, the present invention comprises the following steps: Step 1: Input source acquisition and triple generation First, obtain a UTF-8 encoded Chinese dictionary text file. Each line of the text file is formatted as "words". <tab>Words in pinyin <tab>The "word frequency" function uses lowercase ASCII characters without tones for word pinyin. For polyphonic characters, multiple pronunciations are separated by vertical lines. When classifying words, only the first pinyin before the vertical line is used as the classification key to ensure that different pronunciations of the same Chinese character can be classified into the corresponding pinyin bucket, avoiding the omission of candidate words. For each line of the text file, the first Unicode Chinese character of the word is extracted. Combined with the first pinyin extracted from that line and the word frequency, a "word-pinyin-word frequency" triple is generated to complete the preprocessing of the original data.

[0032] Step 2: Classify Pinyin by Bin and Sort by Word Frequency in Descending Order Based on the generated triples, a hash table is built using Pinyin as the key. All triples are categorized into corresponding Pinyin buckets according to their Pinyin keys, with each bucket corresponding to a unique Pinyin string. For all words within each Pinyin bucket, a stable sorting algorithm is used to sort them in descending order by word frequency, with words having higher frequencies appearing first. When multiple words have the same frequency, their input order in the original Chinese dictionary text file is maintained to ensure the reproducibility and auditability of the generated results. After sorting, for each unique Chinese character appearing in each Pinyin bucket, an independent character block is generated. Subsequently, words are physically isolated and stored in partitions based on their length.

[0033] Step 3: Word zoning and solidification under the initial character reuse mechanism For each independent Chinese character block, a first-character reuse mechanism is adopted. Words are divided into four sub-regions based on the number of characters: 2-character words, 3-character words, 4-character words, and phrases. These four sub-regions are physically contiguous, without padding, and completely physically isolated. The specific rules are as follows: The Chinese character block pre-count field is set as follows: At the beginning of each Chinese character block, an 8-byte pre-count field is set. All count fields are stored in big-endian order with the most significant byte first. The fields are as follows: 2-byte unsigned integer N, which records the total number of 2-character words corresponding to the Chinese character; 2-byte unsigned integer M, which records the total number of 3-character words corresponding to the Chinese character; 2-byte unsigned integer K, which records the total number of 4-character words corresponding to the Chinese character; and 2-byte unsigned integer L, which records the total number of phrases corresponding to the Chinese character.

[0034] Storage rules for reusing the first character of each sub-area: 2-character word sub-region: Stores only the GB2312 national standard Chinese character encoding of the second Chinese character of the word. Each word occupies a fixed 2 bytes, without adding any additional terminator. The total length of the sub-region is N×2 bytes. The starting address of the storage of the i-th 2-character word in this sub-region is calculated by a fixed formula: Word_OFF+8+i×2, where Word_OFF is the starting absolute offset of the word storage area. 3-character word sub-area: Stores only the GB2312 national standard Chinese character encoding of the second and third Chinese characters of the word. Each word occupies a fixed 4 bytes, without adding any additional terminator. The total length of the sub-area is M×4 bytes. The starting address of the storage of the j-th 3-character word in this sub-area is calculated by a fixed formula: Word_OFF+8+N×2+j×4. 4-character word sub-region: Stores only the GB2312 national standard Chinese character encoding of the second to fourth Chinese characters of the word. Each word occupies a fixed 6 bytes, without adding any additional terminator. The total length of the sub-region is K×6 bytes. The starting address of the storage of the k-th 4-character word in this sub-region is calculated by a fixed formula: Word_OFF+8+N×2+M×4+k×6. Phrase Sub-region: Stores only the GB2312 encoding of the remaining Chinese characters in a word excluding the first character, using a double-byte 0x0000 (an illegal Chinese character code point in the GB2312 encoding system) as the termination marker; the starting address of the l-th phrase in this sub-region is calculated using a fixed formula: Word_OFF+8+N×2+M×4+K×6+phrase_len_sum[l]; Where phrase_len_sum[0]=0, , phrase_len[t] is the total number of GB2312 encoded bytes of the remaining Chinese characters in the t-th phrase, excluding the 0x0000 terminator at the end.

[0035] Alignment rules: The total length of each Chinese character block (including the 8-byte pre-count field + all contents of the 4 sub-regions) is forced to be an integer multiple of 4 bytes. If the original length is less than an integer multiple of 4 bytes, it is padded with 0x00 bytes at the end of the phrase sub-region to ensure the determinism of bus access. All Chinese character blocks are finally linearly concatenated in ascending order according to the GB2312 national standard encoding of their first Chinese character to form a complete word storage area.

[0036] Step 4: Constructing a four-level static index structure Based on the aforementioned preprocessing and solidification results, a four-level static index is constructed sequentially, consisting of an initial letter statistics area, a pinyin index area, a Chinese character and word index area, and a word storage area. All index areas use fixed-length entries and absolute offset addressing. The specific construction rules are as follows: Construction of the first letter statistics area: The first letter statistics area is a fixed 32-byte contiguous storage space, with subscripts 0 to 25 corresponding to the English letters a to z, and subscripts 26 to 31 being reserved fields; each storage unit is a 1-byte unsigned integer, recording the number of non-empty pinyin buckets under the corresponding first letter. If there is no valid pinyin entry under a certain first letter, the value of the corresponding storage unit is 0, realizing the first letter existence judgment with O(1) time complexity.

[0037] The Pinyin index area is constructed as follows: The Pinyin index area consists of several fixed 16-byte consecutive entries. The total number of entries is equal to the total number of all non-empty Pinyin bins. All entries are arranged in ascending order according to the ASCII codes of the Pinyin. Each entry contains three parts in a fixed order: The first part is a 10-byte Pinyin string storage bit, which stores the lowercase ASCII character without tone corresponding to the Pinyin. If the part is less than 10 bytes, it is padded with 0x00 null characters on the right. The second part is a 2-byte unsigned integer field, which records the total number of Chinese characters corresponding to the Pinyin. The third part is a 4-byte unsigned integer field, which records the starting offset of the Chinese character entry corresponding to the Pinyin in the Chinese word index area.

[0038] The Chinese character and word index area is constructed as follows: The Chinese character and word index area consists of several fixed 6-byte consecutive entries. The total number of entries is equal to the total number of unique Chinese characters in the dictionary. All entries are arranged in ascending order according to the GB2312 national standard encoding of the Chinese characters. Each entry contains two parts in a fixed order: the first part is a 2-byte field that stores the GB2312 national standard encoding of the corresponding Chinese character; the second part is a 4-byte unsigned integer field that stores the relative address of the independent Chinese character block corresponding to the Chinese character in the word storage area.

[0039] Word storage area: This refers to the complete storage area organized by Chinese character blocks after being assembled in the aforementioned steps, serving as the endpoint of the fourth-level static index.

[0040] Step 5: Deterministic truncation with high frequency priority After the word partitioning is finalized and before the index structure is finally locked, a deterministic truncation operation prioritizing high frequencies is performed. The specific rules are as follows: the upper limit of the number of 2-character words, 3-character words, 4-character words, and phrases is set by preset threshold parameters MAX_N, MAX_M, MAX_K, and MAX_L, respectively; for the corresponding sub-region within each Chinese character block, only the words with the first threshold number after descending word frequency are retained, and all words exceeding the threshold are truncated and removed; after the truncation operation is completed, the values ​​of the preceding count fields N, M, K, and L of the corresponding Chinese character block are updated synchronously, and the corresponding entries and offsets in the Pinyin index area and the Chinese character word index area are updated synchronously to ensure that the index structure and the stored content are completely consistent, and to achieve precise control over the size of the dictionary.

[0041] Step 6: Metadata area construction and integrity verification encapsulation After completing the index structure construction and truncation, a fixed 32-byte metadata area is constructed. The metadata area uses big-endian encoding with the most significant byte first, and its structure is strictly fixed, containing the following 5 fields in order: 4-byte magic number identifier: located in the metadata area Offset0-3, it is fixed to write the hexadecimal code 0x434A434B corresponding to the ASCII string "CJCK", and is used for the legality identification of the dictionary format; 8-byte ASCII version number: located in the metadata area Offset4-11, in the format "XX.XX.XX" consisting of 8 ASCII characters, without a string terminator, and the character set is limited to ASCII '0'-'9' and '.', used for version compatibility matching; 4-byte file length field: Located in the metadata area Offset12-15, its value is defined as the total length of the final generated dictionary file minus 16 bytes, where the 16 bytes include the aforementioned magic number identifier, ASCII version number and this field itself; this value has a dual semantic meaning: first, it defines the total length of the valid data in the dictionary, and second, it serves as the starting address offset for subsequent integrity verification values; The 16-byte absolute offset group for the four major index areas: located in the metadata area Offset16-31, in order are 4 bytes A_Z_OFF, 4 bytes P_Y_OFF, 4 bytes H_Z_OFF, and 4 bytes Word_OFF, respectively corresponding to the big-endian 32-bit unsigned absolute address of the first letter statistics area, the pinyin index area, the Chinese character word index area, and the word storage area relative to the starting address of the dictionary file.

[0042] After the metadata area is constructed, a data integrity check value is generated: using the MD5 hash digest algorithm conforming to RFC1321, the hash digest of all valid data (including the metadata area and all content of the four-level static index) from Offset=0 to the value of the file length field is calculated, generating a fixed 16-byte binary check value; finally, the metadata area, initial letter statistics area, pinyin index area, Chinese character word index area, word storage area, and 16-byte integrity check value are concatenated in sequence, where the integrity check value is physically stored at the end of the final binary dictionary file, and its calculation range is clearly defined not to include the check value itself, thus completing the final generation of the embedded binary dictionary file.

[0043] In another embodiment, the present invention includes the following parts: First-letter pinyin anchoring module: Input: A UTF-8 encoded TXT file, with each line formatted as "words". <tab>Words in pinyin <tab>"Word frequency" (e.g., for "重庆" in Chongqing\tchong|qing\t8500, and "重工" in Zhonggong\tzhong|gong\t6200); The "Pinyin of word" field is the standard pinyin defined for the word, in lowercase, without tones, and in ASCII characters; when there are multiple pronunciations, they are separated by a vertical bar delimiter (e.g., chong|qing), and only the first pinyin before the | is used for classification (e.g., chong|qing → chong; zhong|gong → zhong).

[0044] For each line of words, perform the following: First Chinese character extraction: Take the first Unicode Chinese character (skipping punctuation marks, numbers, and English letters); Pinyin anchoring: Use the first pinyin before the | in the "Pinyin of word" field as the hash key; Compatibility for polyphonic characters: The same Chinese character (e.g., "重") can be naturally distributed in the zhong bucket and the chong bucket due to different words it belongs to; when the user enters any pinyin, all the standardized words under the corresponding bucket can be hit; Output: A stream of (pinyin, word, word frequency) triples, where the pinyin field is always equal to the first pinyin before the |.

[0045] Pinyin bucket word frequency descending classification module Construct a hash table dict[pinyin]=[], and insert all the triples according to the pinyin key; For all the words in each pinyin bucket, sort them in descending order according to their word frequency values, so that the word with the highest word frequency is ranked first, the second highest follows, and so on; When the word frequencies of multiple words are the same, maintain their relative positions in the order of their appearance in the original TXT (stable sorting) to ensure that the generated results are repeatable and auditable; This design enables the loader to locate the corresponding bucket once according to the pinyin string entered by the user (e.g., zhong), and obtain the set of candidate words with the highest word frequency, meeting the requirements of embedded real-time performance.

[0046] Four-dimensional partition solidification and implicit termination module For each pinyin bucket, traverse the sorted list and perform a hard partition according to the number of Chinese characters in the word: len == 2 → Write to the two-character sub-region; len == 3 → Write to the three-character sub-region; len == 4 → Write to the four-character sub-region; len >= 5 → Write to the phrase sub-region; Storing words of different lengths separately can enable the embedded system to use the optimized DMA burst length for each length category (e.g., a two-character block is fixed at 2×N bytes), avoiding runtime judgment and branch jumps.

[0047] For each word written: Write its GB2312 encoding into the corresponding sub-region in byte order; No 0x000x00 terminator is added to the sub-regions for 2-character words, 3-character words, and 4-character words (because their lengths are uniquely determined by the prefixed count field); Each phrase in the phrase sub-region must end with 0x000x00 (because the length is variable, and 0x0000 is an illegal Chinese character code in the GB2312 encoding space and can be safely used as a termination marker); The total length of each Chinese character block (including the 8-byte count field + all sub-regions) is an integer multiple of 4 bytes. Pad with zeros at the end of the Chinese character block if it is insufficient; At the same time, increment the counters: N++ (for 2-character words), M++ (for 3-character words), K++ (for 4-character words), L++ (for phrases), to form the prefixed 8-byte count field for each Chinese character block (N, M, K, L are all double-byte unsigned integers with the high byte in front); All count fields are local values for the corresponding Chinese character block and are not shared across blocks; different Chinese characters under the same pinyin bucket (such as "美" and "每") each have independent N / M / K / L fields.

[0048] Four-level static index structure dynamic construction module According to the classification result, generate the data area required by the embedded static index architecture: A_Z_OFF_TABLE

[32] : Traverse a~z, count the non-empty number of each pinyin bucket (for example, if the zhong bucket exists, then A_Z_OFF_TABLE['z'-'a'] = 1; if both the mei and men buckets exist, then A_Z_OFF_TABLE['m'-'a'] = 2). It is strictly 32 bytes, each byte is uint8_t, and the value ranges from 0 to 255.

[0049] The basis of this design: Each entry in MemForPY_lable has a fixed length of 16 bytes; the loader locates a certain letter range through the prefix sum formula: c_cnt = input_c - 'a' - 1 (how many letters are there between the first letter of the input pinyin and 'a'. For example, if the input is "mei", then c_cnt ='m' - 'a' - 1 = 11); there are 11 letters before m; c_pinyin_sum = (Calculate the sum of the number of pinyin buckets starting from all letters before the input letter); Offset_c_start = P_Y_OFF + c_pinyin_sum × 16; Offset_c_start is the start address of the pinyin bucket corresponding to the first character of the input pinyin, thus achieving O(1) time scanning; MemForPY_label: Creates a 16-byte entry for each non-empty pinyin bucket, where: bytes[0..9] = ASCII pinyin right padding 0x00 (e.g., zhong\0...); bytes[10..11] = Total number of words in this bucket (high-order byte first); bytes[12..15] = the starting offset of the first Chinese character in the MemPY2HZ_label (high-order byte first); MemPY2HZ_label: Iterates through all words and generates a 6-byte entry for each Chinese character. bytes[0..1] = GB2312 encoding of Chinese characters (high-order byte first); bytes[2..5] = the relative address of the Chinese character in MemForWord (high-order byte first); All entries are sorted in ascending order according to the GB2312 encoding of Chinese characters; All offsets are calculated using absolute addresses (e.g., MemForPY_label starting address = 32 + 32 = 64) to ensure compatibility with the metadata area.

[0050] The storage capacity constraint and dynamic truncation module, designed to adapt to resource-constrained embedded platforms (such as MCUFlash with 256KB / 512KB), adds a mechanism to perform deterministic truncation on each sub-region according to a preset threshold after the step of partitioning and solidifying words by word length. This ensures that the total size of the final BIN file is strictly controllable. User-configurable thresholds: Four upper limit values ​​can be specified via command-line parameters or configuration files: MAX_N (maximum number of 2-character words), MAX_M (maximum number of 3-character words), MAX_K (maximum number of 4-character words), and MAX_L (maximum number of phrases); the default value is 0, indicating no limit. These correspond to the preset upper limit thresholds for the number of words in different word length categories.

[0051] Deterministic truncation rules (lossless high-frequency priority): If the current number of 2-character words N > MAX_N, then only the first MAX_N' words are retained (the first MAX_N items after sorting by word frequency in descending order). If M > MAX_M, retain the first MAX_M terms; If K > MAX_K, retain the first MAX_K terms; If L > MAX_L, retain the first MAX_L entries (in descending order of word frequency).

[0052] Global consistency guarantee: After truncation, the first 8 bytes of the MemForWord count field are recalculated and updated to ensure that its value is strictly consistent with the actual number of words written to the sub-region; only Chinese character entries involving the retained words are generated in MemPY2HZ_label. Output volume predictability: Given a threshold, the maximum volume of the final BIN file can be accurately estimated by a deterministic formula.

[0053] Trusted metadata encapsulation module Calculate the starting offsets of the four main regions (32-bit unsigned absolute addresses with the most significant byte first, relative to the starting address of the BIN file): A_Z_OFF=32; P_Y_OFF=32+32=64; H_Z_OFF=P_Y_OFF+N_total×16; Word_OFF=H_Z_OFF+M_total×6; Construct a 32-byte metadata area: Offset0-3: magic number "CJCK" (0x43,0x4A,0x43,0x4B); Offset4-11: ASCII version number "00.00.01" (0x30,0x30,0x2E,0x30,0x30,0x2E,0x30,0x31). Offset12-15: File length (total BIN length - 16, high-order byte first); Offset16-31: Four major offsets (A_Z_OFF / P_Y_OFF / H_Z_OFF / Word_OFF, with the most significant byte first); The MD5 checksum (16 bytes) is not included in the metadata area, but is physically stored at the end of the file. Its calculation range is all bytes in the BIN file from Offset=0 to "file length", excluding the MD5 itself.

[0054] Final concatenation: metadata area (32B) + A_Z_OFF_TABLE (32B) + MemForPY_label + MemPY2HZ_label + MemForWord + MD5 checksum (16B), output .bin file.

[0055] The binary dictionary file (.bin) generated by this invention is a static index structure optimized for embedded devices. It consists of six sequentially concatenated data areas, and the starting address of each area is strictly determined relative to the file's starting address (Offset=0): ① Metadata area (32 bytes, Offset0-31): contains magic number "CJCK" (0x43, 0x4A, 0x43, 0x4B), ASCII version number "00.00.01" (0x30, 0x30, 0x2E, 0x30, 0x30, 0x2E, 0x30, 0x31), file length field (4B), and offsets of the four main areas (16B); all fields adopt the encoding rule of big-endian (high-order byte first, low-order byte last) to ensure consistent parsing results on heterogeneous SoCs such as NXPS32G, TI Jacinto, Renesas RH850, and ARM Cortex-M; ② First letter statistics area (32 bytes, Offset32-63): namely A_Z_OFF_TABLE, each byte corresponds to ASCII 'a' to 'z', with a value of uint8_t, representing the number of non-empty pinyin buckets starting with that letter (0-255); ③ Pinyin Index Area (variable length, starting at Offset64): i.e., MemForPY_label, each entry is 16 bytes, and the number of entries N_total is equal to the total number of non-empty pinyin buckets; ④ Chinese character-word index area (variable length, starting from H_Z_OFF): that is, MemPY2HZ_label, each entry is 6 bytes, and the number of entries M_total is equal to the total number of unique Chinese characters in all the retained words; ⑤ Word storage area (variable length, starting from Word_OFF): also known as MemForWord, it adopts a linked list layout of Chinese character blocks. Each Chinese character block starts with an 8-byte counting field (N / M / K / L, each 2B, with the high-order byte first), followed by the 2 / 3 / 4-character word sub-area and phrase sub-area under that Chinese character; the total length of each Chinese character block (including the 8-byte counting field + all sub-areas) must be an integer multiple of 4 bytes, and any shortfall is padded with zeros at the end of the Chinese character block; ⑥MD5 checksum (16 bytes, located at the end of the file): also known as MD5Checksum, which is the standard digest of RFC1321; its starting address is equal to the value of the "file length" field in the metadata area; the calculation range is strictly limited to [0, file length], covering all valid data from ① to ⑤, but explicitly excluding MD5 itself; This design is based on: The "File Length" field has a value equal to the number of bytes in ② + ③ + ④ + ⑤ + ⑥. It simultaneously carries the dual semantics of "effective data length" and "MD5 starting address," serving as the mathematical hub for integrity assurance.

[0056] This embodiment demonstrates how to convert the original TXT file into a BIN file that conforms to the definition of this invention. The entire process can be verified byte by byte by a technician without external tools using a hexadecimal editor (such as HxD).

[0057] The content of the input file dict.txt is as follows: Chongqing chong|qing8500 Heavy industry zhong|gong6200 Gravity zhong|li4100 Important zhong|yao9300 Hairdressing mei|fa1700 Plum blossom mei|hua1900 Beautiful mei|li12500 Every day mei|tian8900 Sister mei|mei3100 Step 1: First character extraction and word pinyin anchoring "Chongqing" → First character "Chong" → Pinyin field chong|qing → Key = chong; "Heavy industry" → First character "Chong" → zhong|gong → Key = zhong; "Gravity" → zhong|li → zhong; "Important" → zhong|yao → zhong; "Hairdressing" → mei|fa → mei; "Plum blossom" → mei|hua → mei; "Beautiful" → mei|li → mei; "Every day" → mei|tian → mei; "Sister" → mei|mei → mei; → Pinyin bucket set: zhong, chong, mei (a total of 3 buckets); → A_Z_OFF_TABLE['z'-97]=1,['c'-97]=1,['m'-97]=1; → User input zhong → Load the zhong bucket → Display "Important", "Heavy industry", "Gravity"; → User input chong → Load the chong bucket → Display "Chongqing".

[0058] Step 2: Descending order of word frequencies within the buckets<{Z} zhong bucket: ("Important", 9300), ("Heavy industry", 6200), ("Gravity", 4100); chong bucket: ("Chongqing", 8500); mei bucket: ("beautiful", 12500), ("every day", 8900), ("sister", 3100), ("plum blossom", 1900), ("hairdressing", 1700) → After descending order: ("beautiful", 12500), ("every day", 8900), ("sister", 3100), ("plum blossom", 1900), ("hairdressing", 1700); Step 3: Chinese character block solidification (real structure of MemForWord) There is one Chinese character block for each of the four Chinese characters in the mei bucket; All Chinese character blocks are concatenated in ascending order according to GB2312 encoding (C3C0 beautiful < C3C3 sister < C3B7 plum blossom < C3BF every); There is no global preamble field in MemForWord, but each Chinese character block has its own N / M / K / L.

[0059] Step 4: Storage capacity constraint Assume the user specifies MAX_N = 2 (that is, each Chinese character block retains at most 2 two-word phrases), then: For the "zhong" block in the zhong bucket, retain "important" and "heavy industry"; For the "chong" block in the chong bucket, retain "Chongqing"; For the "mei" block in the mei bucket, retain "beautiful" and "hairdressing"; For the "mei" block in the mei bucket, retain "plum blossom"; For the "mei" block in the mei bucket, retain "sister"; For the "mei" block in the mei bucket, retain "every day"; N_zhong_zhong = 2, N_chong_zhong = 1, N_mei_mei = 2, N_mei_mei = 1, N_mei_mei = 1, N_mei_every = 1; The corresponding count fields in MemForWord are updated accordingly to ensure strict consistency with the actual content.

[0060] Step 5: Construct MemForPY_lable entries For the zhong bucket: The corresponding number of Chinese characters is 1; Entry content: 7A686F6E67000000000000 (zhong\...)+0001 (1, high byte first)+XXXXXXXX (offset of the Chinese character "zhong" in MemPY2HZ_lable, high byte first); For the chong bucket: Similarly, generate an independent 16-byte entry; For the mei bucket: Corresponding to the four Chinese characters "mei", "mei", "mei", "mei"; Entry content: 6D656900000000000000 (mei\0...) + 0004 (4, high byte first) + XXXXXXXX (offset of Chinese character "美" in MemPY2HZ_lable, high byte first, and the rest of the Chinese characters "妹", "梅", "每" follow closely); The three buckets are adjacent entries in MemForPY_lable with no ambiguity.

[0061] Step 6: Construct MemForWord entries Length of the Chinese character "重" (zhong) block: 8 bytes + 2×2 = 12 bytes; Length of the Chinese character "重" (chong) block: 8 bytes + 2 + 2 (padding 0 for 4-byte alignment of the Chinese character block) = 12 bytes; Length of the Chinese character "美" (mei) block: 8 bytes + 2×2 = 12 bytes; Length of the Chinese character "妹" (mei) block: 8 bytes + 2 + 2 (padding 0 for 4-byte alignment of the Chinese character block) = 12 bytes; Length of the Chinese character "梅" (mei) block: 8 bytes + 2 + 2 (padding 0 for 4-byte alignment of the Chinese character block) = 12 bytes; [[ID=1​​​​​​​​​​​​​​​​​

[0064] MemForWord data area structure description The MemForWord section uses a "Chinese character block linked list" layout: Each Chinese character block begins with an 8-byte count field, consisting of four double-byte unsigned integers, with the high-order byte first and the low-order byte last. These integers represent the number of 2-character words (N), 3-character words (M), 4-character words (K), and phrases (L) corresponding to the Chinese character, respectively. The following sections are the 2-character word sub-region (containing only the 2nd character, 2×N bytes), the 3-character word sub-region (containing only the 2nd and 3rd characters, 4×M bytes), the 4-character word sub-region (containing only the 2nd and 4th characters, 6×K bytes), and the phrase sub-region (containing the following characters and ending with `0x000x00`).

[0065] Each Chinese character block is linearly concatenated in ascending order according to its GB2312 encoding; All words within each sub-region are arranged in strict descending order according to the word frequency field in the original TXT file; The total length of each Chinese character block (including the 8-byte counting field + all sub-regions) is a multiple of 4 bytes; if the original length is L bytes, then (4-L%4)%4 bytes of 0x00 are appended to the end of the block to make its total length meet the alignment requirements. A_Z_OFF_TABLE Function Description A_Z_OFF_TABLE is a 32-byte initial letter count table, where each byte corresponds to ASCII 'a' to 'z', defined as follows: Each byte is a uint8_t unsigned integer, with a value range from 0 to 255; The byte value represents the number of non-empty pinyin bins that begin with that letter.

[0066] Example: If the input contains two buckets, zhong and chong, then A_Z_OFF_TABLE['z'-'a']=1,['c'-'a']=1; If the input contains three buckets: mei, men, and meng, then A_Z_OFF_TABLE['m'-'a']=3; If there is no pinyin bucket under a certain letter (such as 'x') → A_Z_OFF_TABLE['x'-'a']=0; The loader uses this value for a quick O(1) time determination: if the value is 0, all subsequent processing corresponding to that letter is immediately skipped; if the value is cnt, cnt consecutive 16-byte entries in MemForPY_label are scanned. This design does not perform address calculations but only provides size suggestions, ensuring lightweight, deterministic, and cross-platform compatibility.

[0067] Example of the Complete Structure of a Single Chinese Character Block in MemForWord (Taking the "mei" Block as an Example) To enable those skilled in the art to accurately understand the data organization method of MemForWord, the following takes the Chinese character block corresponding to the Chinese character "mei" in the mei bucket as an example, and gives the complete hexadecimal layout (HexDump) and address mapping description in the BIN file. Assume that the block contains: 2-character words: beautiful (12500), wonderful (8900), hairdressing (3100) → N = 3; 3-character words: beautiful girl (2200), wonderful (900) → M = 2; 4-character words: beautiful life (900) → K = 1; Phrase: California, USA (300) → L = 1.

[0068] Then the physical storage format of the "mei" block in MemForWord is as follows (all double-byte integers are written with the high byte first): MemForWord Single Chinese Character Example Table: So far, the original content of the "mei" block is 42 bytes in total (0x0000 - 0x0029); 42 % 4 = 2, so 2 bytes of 0000 are filled at 0x002A - 0x002B; The total length of this block is 44 bytes (0x0000 - 0x002B), meeting the 4-byte alignment requirement; This zero-padding operation is applied to the end of this Chinese character block, does not affect the addresses of other blocks, and does not affect the correctness of the relative addresses recorded in MemPY2HZ_lable.

[0069] This example shows that: Each Chinese character block strictly starts with an 8-byte count field (N / M / K / L), without global shared fields; No 0x000x00 is added to the sub-regions of 2-character words, 3-character words, and 4-character words, and their boundaries are uniquely determined by the prefixed count fields; The phrase sub-region must end with 0x000x00, and this terminator is included in the sub-region length; Zero-padding is added to the end of the entire Chinese character block to 4-byte alignment (such as 0000 at the end of the above table), ensuring that its starting address can be used for the ldr instruction; All GB2312 encodings are written with the high byte first (such as "mei" = C3C, "li" = C3C3), conforming to the GB / T2312 - 1980 standard.

[0070] This example is generated based on real input data. Technicians can use any hexadecimal editor (such as HxD, xxd) to open the output BIN file, locate the "beautiful" block at the Word_OFF offset, and verify byte by byte, which fully meets the legal requirements of "clear, complete, and achievable".

[0071] All data collection and extraction in this invention are carried out under compliance with laws and regulations.< / tab> < / tab> < / tab> < / tab>

Claims

1. A method for generating a four-level indexed pinyin bucket dictionary, characterized in that, include: Obtain the Chinese dictionary text, extract the first Chinese character and the first pinyin of each word in each line as the classification key, and generate triplet; The triplets are categorized into corresponding pinyin buckets according to their pinyin, and the words in the buckets are sorted in descending order of word frequency. An independent Chinese character block is generated for each unique Chinese character, and the words are solidified by partitioning according to word length using the first character reuse mechanism. A four-level static index is constructed sequentially, consisting of an initial letter statistics area, a pinyin index area, a Chinese character and word index area, and a word storage area. Each index area uses fixed-length entries and absolute offset addressing. Construct a metadata area containing a dual semantic field of file length, perform deterministic truncation of the lexicon based on a preset threshold prioritizing high frequencies, concatenate the data areas and append data integrity verification values ​​to generate an embedded binary lexicon.

2. The method for generating a four-level indexed pinyin bucket dictionary according to claim 1, characterized in that, The triplet includes words, pinyin, and word frequency. The pinyin uses lowercase ASCII characters without tone. Polyphonic characters are separated by separators to distinguish multiple pronunciations. When classifying, only the first pinyin is taken as the key. Different pronunciations of the same Chinese character are classified into the corresponding pinyin bucket.

3. A method for generating a four-level indexed pinyin bucket dictionary according to claim 1 or 2, characterized in that, The sorting of words in the bucket by word frequency in descending order specifically includes: words in the Pinyin bucket are arranged in descending order by word frequency using a stable sorting algorithm, and the input order in the original text is maintained when words have the same frequency; when fixing words by word length, words are divided into four sub-regions according to the number of Chinese characters, including two-character words, three-character words, four-character words, and phrases, and different sub-regions are stored in physical isolation.

4. The method for generating a four-level indexed pinyin bucket dictionary according to claim 3, characterized in that, The specific implementation of the first-character reuse mechanism for word length partitioning and solidification includes: for two-character words, only the GBK encoding of the second Chinese character is stored in the sub-region; for three-character words, only the encoding of the second and third Chinese characters is stored in the sub-region; and for four-character words, only the encoding of the second to fourth Chinese characters is stored in the sub-region. No additional terminator is added to any sub-region, except for the phrase sub-region which uses a double-byte illegal encoding as the termination mark.

5. The method for generating a four-level indexed pinyin bucket dictionary according to claim 4, characterized in that, The initial letter statistics area is a fixed-length continuous storage space, with its storage units corresponding one-to-one with the 26 English letters. Each storage unit records the number of non-empty pinyin bins under the corresponding initial letter.

6. The method for generating a four-level indexed pinyin bucket dictionary according to claim 5, characterized in that, The Pinyin index area consists of several consecutive entries of fixed length, with the total number of entries equal to the total number of non-empty Pinyin buckets. Each entry includes, in sequence, a fixed-length Pinyin string storage space, a field for the total number of Chinese characters corresponding to the Pinyin, and a field for the starting offset of the Chinese character entry in the Chinese character word index area. All entries are arranged in ascending order according to the Pinyin ASCII code. If the Pinyin string storage space is insufficient, it is padded with empty characters.

7. A method for generating a four-level indexed pinyin bucket dictionary according to claim 1 or 6, characterized in that, The Chinese character and word index area consists of several consecutive entries of fixed length. The total number of entries is equal to the total number of unique Chinese characters in the word database. Each entry includes a fixed-length Chinese character GB code field and a relative address field of the independent Chinese character block corresponding to the Chinese character in the word storage area. All entries are arranged in ascending order according to the Chinese character GB code.

8. The method for generating a four-level indexed pinyin bucket dictionary according to claim 7, characterized in that, The specific steps of performing high-frequency priority deterministic truncation on the lexicon according to the preset threshold include: by setting the upper limit threshold for the number of words of different word length categories, truncating the words in the corresponding sub-regions within each Chinese character block, retaining only the words with the first threshold number after the word frequency is sorted in descending order, and synchronously updating the corresponding counting field and index entries after truncation.

9. The method for generating a four-level indexed pinyin bucket dictionary according to claim 8, characterized in that, The metadata area is a fixed-length contiguous storage space, using big-endian encoding with the most significant byte first. It sequentially stores a fixed-length format magic number field, an ASCII format version number field, a file length field, and the absolute offset fields of the four index areas. The file length field simultaneously carries the dual semantics of valid data length and the starting address of the integrity check value.

10. A method for generating a four-level indexed pinyin bucket dictionary according to claim 9, characterized in that, The data integrity check value is a fixed-length binary data generated by a standard hash digest algorithm and is physically stored at the end of the binary dictionary file. The calculation scope of the data integrity check value covers all valid data in the metadata area and the four major index areas except for the check value.

Images