A general corpus construction method and system for heterogeneous information units and a storage medium

Abstract

The application discloses a kind of general corpus construction method, system and storage medium for heterogeneous information unit.It is determined that the method includes: obtaining original data, its information unit type, type at least includes natural language text type, rule-driven symbol expression type and structured data type one of them;According to type, split processing is carried out, and information unit is obtained;For the information unit that needs to be templated, replace variable data field with placeholder, generate template information unit with placeholder;According to the condition of warehousing, the information unit of warehousing is determined and corpus identifier is distributed;Heterogeneous information unit library is constructed, including template information unit and complete information unit.The application realizes the cross-type unified library of multi-source heterogeneous data, and considers the compression efficiency and the integrity of information coverage, and can provide general corpus infrastructure for various downstream applications.

Description

Technical Field

[0001] This invention relates to the field of data processing and information management technology, specifically to a method, system, and storage medium for constructing a general corpus for heterogeneous information units. The corpus constructed by this invention is a general corpus infrastructure that can provide corpus support for various downstream applications, including but not limited to: secure document storage and retrieval, satellite communication and IoT compression processing, differentiated translation of system call instructions, processor instruction fetch acceleration, log compression, and security auditing. Background Technology

[0002] Corpora, as organized collections of information, have wide applications in natural language processing, information retrieval, and data compression. However, existing technologies for corpus construction have the following limitations: I. Domain Separation of Single-Type Corpora Existing corpus technologies are typically designed for specific data types. For example, vocab libraries in the field of natural language processing (such as BERT's vocab.txt) only handle text; instruction set manuals in the field of program code only handle machine instructions; and log template libraries in the field of log analysis (such as templates extracted by the Drain algorithm) only handle system logs. The construction methods and technical goals of these corpora are domain-isolated—log template libraries cannot handle machine instructions, vocab libraries cannot handle sensor messages, and they cannot meet the unified corpus construction requirements of multi-source heterogeneous data. II. Insufficient Granularity of Cross-Type Data Management Technologies Content management systems and digital asset management systems can store and manage various types of files, but they are essentially "archiving" files rather than extracting and abstracting information units within files at a uniform granularity. They manage file objects, not independently usable information units within files, and therefore cannot support semantic-level matching and substitution. III. The Temporality and Uncertainty of Dynamic Dictionary Compression Dynamic dictionary compression algorithms such as LZ77 and LZW achieve data compression by dynamically constructing dictionaries. However, these technologies differ fundamentally from this invention: the dictionary in dictionary compression is dynamically constructed, the encoding result depends on the data context, and the same data fragment may correspond to different codes in different locations or different files; the dictionary itself is temporary and cannot form a static corpus asset that can be independently managed, sustainably reused, and applied downstream. Specifically, existing corpus construction techniques have the following drawbacks: Lack of a unified cross-type processing framework: Different types of data (text, code, structured data) require different processing tools, making it impossible to complete the unified construction from raw data to a corpus in the same pipeline. Lack of information unit-level templated abstraction capabilities: Existing technologies either store the original information in its entirety or perform simple hash mapping, lacking analysis and abstraction of the internal structure of information units, and are unable to identify and template-merge similar information units that "only differ in variable fields". The existing corpus has a single structure: it usually contains only one type of entry (such as all original entries or all templates), and lacks a hybrid structure that can accommodate both templated and original entries, thus failing to balance compression efficiency and the completeness of information coverage. The corpus is disconnected from the application: The construction of existing corpora usually does not take into account the specific needs of downstream applications (such as compressed storage, secure storage, fast retrieval, etc.), resulting in the corpus lacking application-specificity. Therefore, there is an urgent need for a method and system that can handle heterogeneous information units, process them uniformly across types, support templated abstraction, and build hybrid structure corpora, so as to provide a general corpus infrastructure for various downstream applications. Summary of the Invention

[0003] Technical problems to be solved This invention aims to provide a general corpus construction method and system for heterogeneous information units, in order to solve the problems of single type, lack of template-based abstraction ability, single corpus structure, and disconnection from downstream applications in existing corpus construction technologies, and to provide a general corpus infrastructure for applications such as secure document storage, communication compression, instruction translation, and processor acceleration. Technical solution In a first aspect, the present invention provides a method for constructing a general corpus for heterogeneous information units, comprising: Obtain raw data, wherein the raw data has a corresponding information unit type, and the information unit type includes at least one of the following: natural language text type, rule-driven symbolic expression type, and structured data type; Based on the information unit type, the original data or its internal data segments are segmented to obtain information units; the segmentation process includes extracting multiple information units from the same original data segment, and the extracted information units are allowed to have partial overlap in content; Based on preset conditions, information units that require template processing are templated. The template processing includes replacing variable data fields in the information unit with placeholders to generate template information units with placeholders. Based on preset data entry conditions, the data entry units are determined from the data entry units obtained through segmentation and / or the template data entry units generated through template processing. Assign a corpus entry identifier to the information unit that is entered into the corpus. The corpus entry identifier is used to uniquely identify the corresponding information unit or template information unit in the corpus. Construct a heterogeneous information unit library, which includes template information units with placeholders generated through template processing and complete information units that have not undergone template processing. Furthermore, the determination of the information unit type can be done by pre-marking before data input (e.g., by file type, data source, user specification, etc.), or by automatic identification by the system after data input. Both methods fall within the protection scope of this invention. In one embodiment, the system automatically determines the information unit type based on the file extension or MIME type; in another embodiment, the user specifies the type of data to be processed through a configuration file. Furthermore, the templated processing includes: clustering multiple information units within the same information unit type, and merging information units that differ only in the variable data field into a single template information unit with placeholders. Furthermore, before clustering, the process includes: counting the frequency of each information unit in the original data; identifying information units with a frequency higher than a preset threshold as participating in the clustering and template processing; and retaining information units with a frequency lower than or equal to the preset threshold in their complete form. Furthermore, the core of the inclusion criteria is to balance the compression value of information units with the cost of corpus space. In one embodiment, the system calculates an inclusion score for each candidate information unit, which comprehensively considers the following factors: the frequency of occurrence of the information unit in the original data, the original length of the information unit, whether multiple variations can be covered by templates, and the available capacity of the current corpus. When the score exceeds a preset threshold and the corpus still has available capacity, the information unit is included in the corpus. Through this mechanism, under the premise of feasible corpus space, items with high compression value are included in the corpus as much as possible, while taking into account the size of the corpus, ensuring the conciseness and efficiency of the corpus. Furthermore, the variable data field includes at least one of the following: when the information unit belongs to a rule-driven symbolic representation type and is a machine instruction, the variable data field includes, but is not limited to: an immediate numeric field, an address offset field, a register number field, and a displacement field; when the information unit is a natural language text unit or a rule-driven symbolic representation unit, the variable data field includes, but is not limited to: numerical parameters, date and time parameters, named entities, variable names, placeholders, measurement values, numbered identifiers, and replaceable text expressions. Furthermore, identifying variable data fields within an information unit includes: automatically determining the variable data fields within the information unit based on statistical feature analysis or pattern recognition of multiple information units within the same information unit type. Furthermore, the rule-driven symbolic representation type includes at least one of the following: program code, machine instructions, mathematical formulas, chemical molecular formulas, chemical reaction equations, gene sequence representations, and regular expressions; the structured data type includes at least one of the following: sensor data messages, database records, communication signaling, configuration file items, and key-value pair data. Furthermore, the corpus entry identifier includes a type field and a sequence field. The type field indicates the information unit type to which the corresponding information unit belongs, and the sequence field is uniquely numbered within the same information unit type. Furthermore, the raw data originates from at least two of the following data sources: office document library, program code library, professional literature library, communication protocol specification document, processor instruction set manual, system log, and database operation record. Secondly, the present invention provides a general corpus construction system for heterogeneous information units, comprising: The data acquisition module is used to acquire raw data; The type acquisition module is used to determine the information unit type corresponding to the original data. The information unit type includes at least one of the following: natural language text type, rule-driven symbolic expression type, and structured data type. The segmentation processing module is used to segment the original data or its internal data segments according to the information unit type to obtain information units; the segmentation processing includes the ability to extract multiple information units from the same original data segment, and the content of each extracted information unit is allowed to partially overlap. The template processing module is used to perform template processing on information units that need to be templated according to preset conditions. The template processing includes replacing the variable data fields in the information unit with placeholders to generate template information units with placeholders; and clustering multiple information units within the same information unit type, merging information units that differ only in the variable data fields into one template information unit with placeholders. The inbound screening module is used to determine the information units to be inbound from the information units obtained by segmentation and / or the template information units generated by template processing, based on preset inbound conditions. The identifier allocation module is used to assign a corpus entry identifier to the information unit that is entered into the corpus. The corpus entry identifier is used to uniquely identify the corresponding information unit or template information unit in the corpus. The library building module is used to build a heterogeneous information unit library, which includes template information units with placeholders generated by template processing and complete information units that have not been templated.

[0004] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0005] Beneficial effects Compared with the prior art, the present invention has the following beneficial effects: Unified cross-type processing: By acquiring the information unit type of the original data and segmenting it according to the type, a unified database construction process for various heterogeneous data such as natural language text, rule-driven symbolic expression, and structured data is realized, breaking the domain isolation of existing technologies. Efficient templated abstraction: By comparing multiple information units within the same type, information units that differ only in variable data fields are clustered and merged into templates with placeholders. One template can cover countless original instances, greatly improving the compression efficiency and information density of the corpus. Flexible hybrid corpus structure: The constructed corpus contains both template information units and complete information units, which ensures both the compression efficiency of high-frequency patterns and the complete coverage of low-frequency or unique information, achieving a balance between compression rate and coverage. It provides general infrastructure for downstream applications, and provides unified corpus support for various downstream applications such as document compression and storage, secure storage, communication compression, instruction translation, and processor acceleration.

[0006] Flexible inclusion filtering: By using inclusion conditions based on the balance between compression value and corpus capacity, the system ensures that the corpus includes as many high-compression-value entries as possible under capacity constraints, thereby achieving efficient utilization of corpus capacity.

[0007] Terminology Definition In this specification, unless otherwise stated, the following terms have the following meanings: Information unit: refers to an information unit that can be independently processed and separated from raw data. The form and granularity of an information unit depend on its type and application requirements. For example, for natural language text, it can be a character, word, phrase, sentence, or paragraph; for program code, it can be an instruction, a function, or a code block; and for structured data, it can be a data record or a data field. Information Unit Type: A classification identifier used to distinguish different data sources and structures. The type system of this invention includes at least the following three basic types, but in a specific implementation, only one or more of these may be involved: natural language text types (such as article paragraphs, dialogue statements), rule-driven symbolic expression types (such as program code, machine instructions, mathematical formulas, chemical formulas, gene sequences), and structured data types (such as sensor data messages, database records, communication signaling, key-value pair data). This type system is scalable. Segmentation processing refers to the process of extracting information units from raw data based on the type of information unit and employing corresponding segmentation or extraction strategies. Segmentation processing includes, but is not limited to, segmentation based on predetermined rules, segmentation based on semantic analysis, and segmentation based on statistical features. In one embodiment, segmentation processing is full-granularity extraction—that is, for the same raw data segment, multiple possible information units are extracted with different window lengths and / or different starting positions. The extracted information units are not required to be mutually exclusive or non-overlapping. For example, for the natural language text "I love Beijing Tiananmen", multiple information units of different granularities such as "I love", "I love Beijing", "Beijing", "Tiananmen", "Beijing Tiananmen", and "I love Beijing Tiananmen" can be extracted; for machine code sequences, instruction segments can be extracted starting from each byte offset with different window lengths. Variable data fields refer to specific fields or fragments within an information unit that change depending on the specific instance. Examples include the immediate numeric field and address offset field in machine instructions, timestamps and IP addresses in log text, and any textual expression in a document sentence that can be replaced without altering the sentence structure, such as names, places, dates, numerical values, modifiers, and object names. The specific types of variable data fields depend on the type of information unit and the application scenario; the examples above are merely illustrative and do not constitute a limitation on the scope of variable data fields. Placeholder: A special identifier used in template information units to mark the location of variable data fields. A placeholder itself does not contain a specific data value; it only indicates the existence and location of the variable field. In downstream applications, the placeholder position can be replaced by the actual variable data field, thus restoring the complete original information. Template processing: This refers to the process of identifying variable data fields in an information unit and replacing them with placeholders to generate a template information unit with placeholders. Template processing may optionally include a clustering and merging step, that is, merging multiple information units that differ only in variable data fields into a single template information unit. Clustering and merging: This refers to comparing multiple information units within the same information unit type and merging information units that differ only in variable data fields into a template information unit with placeholders. Template information unit: refers to an information unit containing placeholders obtained after template processing. One template information unit can correspond to multiple instances of the original information unit that differ only in the variable data fields. Complete information unit: refers to an information unit that has not undergone template processing and retains its original, complete form. This includes, but is not limited to: information units that occur less frequently than a preset threshold and are therefore not included in template processing; information units that cannot be clustered or merged with other information units of the same type; information units that do not contain variable data fields; or information units that retain their original form according to a preset strategy. The above list does not constitute a restriction on the source of complete information units. Heterogeneous information unit corpus: refers to a hybrid corpus composed of templated information units and complete information units. Its "heterogeneity" is reflected in two dimensions: first, the heterogeneity of item types—it contains both templated items and complete items; second, the heterogeneity of information unit types—it can contain information units from different data types. Inclusion criteria: These are the selection rules used to determine whether an information unit should be included in the corpus. The core of inclusion criteria is to assess the balance between the compression value of the information unit and the cost of corpus space. Specifically, factors such as the frequency of occurrence of the information unit, its original length (the longer the length, the higher the subsequent compression benefit), whether it can be templated to cover multiple variations, and the current available capacity of the corpus can be considered. Corpus entry identifier: A corpus entry identifier is a symbol that uniquely identifies a corpus entry (information unit or template information unit) within a corpus. Corpus entry identifiers can be in any form, including but not limited to: numerical codes, strings, hash values, etc. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the overall process of corpus construction in this embodiment of the invention. Figure 2 This is a schematic diagram of the template processing procedure in an embodiment of the present invention. Figure 3 This is a schematic diagram of the heterogeneous information unit library in an embodiment of the present invention. Figure 4 This is a block diagram of the corpus construction system in an embodiment of the present invention. Figure 5 This is a schematic diagram of the encoding structure of the corpus entry identifier in an embodiment of the present invention.

[0009] Figure 6 This is a schematic diagram of the mixed data input and processing process in an embodiment of the present invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that these embodiments are only for explaining the invention and do not constitute a limitation thereof. Example 1: Overall Method Flow This embodiment demonstrates the complete method flow of the present invention. Step S101: Obtain raw data. Raw data can originate from various data sources, including but not limited to office document libraries, program code libraries, professional literature libraries, communication protocol specification documents, processor instruction set manuals, system logs, and database operation records. Raw data has corresponding information unit types, which can be determined through pre-marking before data input (e.g., by file extension, data source, user specification, etc.) or automatically identified and determined by the system after data input. In one implementation, the system automatically determines the type of information unit based on the file extension or MIME type—for example, .txt or .docx files correspond to natural language text, .c or .cpp files correspond to rule-driven symbolic representation (program code), and .json or .xml files correspond to structured data types. In another implementation, the user explicitly specifies the type of data to be processed through a configuration file or input parameters. Step S102: Perform segmentation processing based on the information unit type. Based on the information unit type of the original data, the system employs a corresponding segmentation strategy to divide the data into information units. The segmentation method can be flexibly selected according to data type and application requirements. For example: For natural language text, it can be segmented according to rules based on sentence boundaries (periods, question marks, exclamation marks, line breaks, etc.) to obtain information units based on sentences; or it can be semantically segmented using natural language processing models or AI models to treat semantically complete expression segments as information units. For rule-driven symbolic representation types (such as machine instructions), they can be segmented according to instruction boundaries (opcode length, instruction format) to obtain single instruction information units; For structured data types (such as JSON sensor data), they can be segmented by record delimiters or object boundaries to obtain individual data record information units. In a preferred embodiment, the system utilizes a pre-trained language model to perform semantic segmentation on natural language text. Semantic segmentation can identify the semantically complete expression boundaries in the text, avoiding the semantic fragmentation that may result from mechanically segmenting by punctuation marks, and improving the effectiveness of subsequent template processing. For example, for complex long sentences containing multiple clauses, semantic segmentation can preserve them as a complete information unit, rather than mechanically breaking them down by commas or semicolons. Step S103: Statistical frequency (optional). In a preferred embodiment, the system statistically analyzes the frequency of occurrence of each segmented information unit in the original data. This frequency analysis provides a basis for subsequent template processing and database filtering. Step S104: Template processing. Based on preset conditions, the system determines which information units need to be templated. Template processing includes: comparing multiple information units within the same information unit type, identifying variable data fields, replacing variable data fields with placeholders, and generating template information units with placeholders. The triggering conditions for templated processing can be based on one or a combination of the following rules: Information units that appear more frequently than a preset threshold are used for template processing; Information units whose length exceeds a preset threshold are subject to template processing; Information units belonging to a specific information unit type are subject to template processing by default. For information units that appear infrequently, do not contain variable data fields, or cannot be merged with similar information units, the system retains their original complete form and does not perform template processing. Step S105: Determine the information unit to be put into storage based on the storage conditions. The core of the inclusion criteria is balancing the compressed value of an information unit with the storage capacity cost. In one embodiment, the system calculates an inclusion score for each candidate information unit, taking into account the following factors: The frequency of this information unit's occurrence (the higher the frequency, the greater the cumulative compression value); The original length of the information unit (the longer the length, the more space is saved after encoding and replacement, and the higher the compression benefit). Can multiple variants be covered by templates (after templates are used, one entry can replace multiple original instances, which has an amplification effect)? The available capacity of the current corpus (the closer it is to the capacity limit, the more stringent the screening). When the score exceeds a preset threshold and the corpus still has available capacity, the information unit is included in the corpus. Through this mechanism, the system includes items with high compression value as much as possible, while taking into account the size of the corpus, ensuring the corpus is concise and efficient. Step S106: Assign corpus entry identifiers. Each piece of information (whether a template piece or a complete piece) is assigned a unique corpus entry identifier. This identifier is used to uniquely identify the corresponding entry within the corpus. In one embodiment, the corpus entry identifiers employ a structured encoding method, comprising a type field and a sequence field—the type field indicating the type to which the information unit belongs, and the sequence field providing a unique identifier within the same type. This encoding method facilitates efficient corpus management and retrieval. Step S107: Construct a heterogeneous information unit library. The resulting heterogeneous information unit library contains both templated information units with placeholders generated through template processing and complete information units that have not undergone template processing. This hybrid structure allows the corpus to maximize compression efficiency while ensuring complete coverage of rare or unique information. AI-based unified library construction implementation method In one implementation, some or all of the above method steps are performed by an AI model. Specifically, the raw data is input into a trained neural network model, which implicitly performs information unit type determination, segmentation, variable data field identification, and template processing. The model outputs structured corpus entries, containing template information units, complete information units, and their corresponding corpus entry identifiers. It is understood that regardless of whether the above steps are executed by an explicit rule engine or implicitly by an AI model in an end-to-end manner, as long as the processing objectively includes the steps defined in the claims of the present invention, they fall within the protection scope of the present invention. Example 2: Unified Processing of Multiple Data Types This embodiment demonstrates how the system processes mixed raw data containing multiple information unit types, showcasing the cross-type unified processing capability of the present invention. Scenario Description: A company's knowledge base contains a technical document that includes text paragraphs in natural language, sample program code snippets, and test data records presented in JSON format. Processing flow: First, the system obtains the document and its corresponding information unit type information. In this embodiment, different fragments of the document have different types—the main text paragraphs are natural language text, the code fragments are rule-driven symbolic representations, and JSON data is a structured data type. Type information can be obtained through document annotations, formatting tags (such as Markdown code block tags), or automatic recognition. Then, the system iterates through each segment of the document and performs segmentation based on the information unit type of each segment: Main text paragraphs: Segmented according to sentence boundaries (periods, question marks, exclamation marks, line breaks, etc.), or semantically segmented using natural language processing models to obtain text information units; Code snippets are divided according to instruction lines or function boundaries to obtain code line or function information units; JSON data is split according to the boundaries of the record object to obtain a single data record information unit. The segmented information units enter the template processing stage. The system performs comparison and clustering within each type: In the text sentences, sentences such as "The system will be maintained on X year X month X day" and "The system will be maintained on Y year Y month Y day" were found to differ only in the date parameter. These sentences were clustered and merged into the template "The system will be maintained on {{date}}". In the code snippet, instructions such as "MOV EAX, 0x1" and "MOV EAX, 0x2" that differ only in their immediate values ​​were found and clustered together into the template "MOV EAX, {{imm32}}". In the JSON data, it was found that multiple reports from the same sensor differed only in the numerical fields, and were merged into a JSON template with placeholders. Finally, after template processing and database filtering, the three types of information units each generated their own template information units and retained complete information units, which were then uniformly incorporated into the heterogeneous information unit database. Each entry in the database was assigned a corpus entry identifier, and the type field in its encoding indicated the type of information unit it belonged to. This embodiment demonstrates that the method of the present invention does not require the establishment of separate processing pipelines for different types of data, but completes the entire process from data segmentation to data storage within a unified framework. Example 3: Specific Implementation of Template Processing This embodiment demonstrates in detail the templated processing procedure, particularly the specific implementation of cluster merging. For example... Figure 2 As shown. Step S301: For multiple information units of the same information unit type, the system performs pairwise comparisons. For information units of the same length, the content is compared field by field. Step S302: For fields that differ, the system determines whether these differing fields belong to variable data fields. The criteria for determining variable data fields include: (1) the field presents different values ​​in multiple different instances; (2) the field conforms to specific pattern characteristics, such as timestamp format, IP address format, numerical format, etc.; (3) the position of the field in the information unit is not a key position that plays a decisive role in the semantic structure. Step S303: When two or more information units differ only in the identified variable data fields and are otherwise identical, the system clusters them into a group. Step S304: Merge the group of information units to generate a template information unit with placeholders. In the template, variable data fields are replaced with their corresponding placeholders, while the rest remains unchanged. Example A – Machine Instructions: Input commands: B8 01 00 00 00 (MOV EAX, 0x1) and B8 02 00 00 00 (MOV EAX, 0x2). Comparison results: The two instructions differ only in the second to fifth bytes. According to the x86-64 instruction set manual, these bytes correspond to a 32-bit immediate numeric field, which is a variable data field. The merged template is: B8 ?? ?? ?? ??, where ?? is a placeholder representing a 32-bit immediate value. This template corresponds to the opcode MOV EAX, imm32. Example B - System Log: Input logs: 2024-01-15 10:23:45 ERROR Connection to 10.0.0.1 failed and 2024-01-15 10:24:01 ERROR Connection to 10.0.0.2 failed. Comparison results: There are differences between the timestamp and IP address portions. Statistical analysis shows that the values ​​at these two locations vary across a large number of log instances, consistent with the characteristics of variable data fields. The merged template is: {{timestamp}} ERROR Connection to {{ip_address}} failed, where {{timestamp}} and {{ip_address}} are placeholders. Example C – Office Document Sentence Structure: Enter the following sentences: The meeting will be held on January 15, 2024; The meeting will be held on March 20, 2024; The meeting will be held on June 10, 2024. Comparison results: Only the date portion is different. The merged template reads: "The meeting will be held on {{date}}, where {{date}} is a placeholder." Example 4: Application of Fixed-Length and Variable-Length Placeholders This example demonstrates the application of fixed-length placeholders and variable-length placeholders in different scenarios. Fixed-length placeholder scenario – machine instructions: The x86-64 instruction MOV EAX, imm32's machine code is B8 ?? ?? ?? ??, where four ?? form a fixed-length placeholder, occupying a fixed 4 bytes. Regardless of the actual value of the immediate value, the position and length of this placeholder remain unchanged. During template restoration, the 4-byte parameter value is precisely filled into the placeholder positions. Variable-length placeholder scenarios – Natural language text: In the sentence template “{{subject}} especially loves to eat {{object}}”, {{subject}} and {{object}} are variable-length placeholders. In specific examples, {{subject}} can be replaced with “I” (1 character), “Xiaoming” (2 characters), and “Teacher Zhang” (3 characters), and {{object}} can be replaced with “apple” (2 characters), “passion fruit” (3 characters), and “roasted chestnuts” (4 characters). Variable-length placeholders do not have a fixed preset length; they dynamically adapt to the length of the actual parameter value during restoration. In actual corpora, the type of placeholders (fixed-length or variable-length) is stored as metadata for template information units. For fixed-length placeholders, the metadata also records their fixed-length value; for variable-length placeholders, the metadata marks them as variable-length types. Example 5: Frequency Filtering and Adaptive Database Ingestion This example demonstrates the role of frequency statistics in template processing and database filtering. Frequency screening process: After segmentation, the system counts the frequency of each information unit in the original data. Information units with high frequency (exceeding a preset threshold, such as 8 times) are prioritized for clustering and template processing. Information units with low frequency (e.g., appearing only 1-2 times) are deemed to have limited benefit from template processing and are retained in their complete form. This frequency-based screening mechanism has the following technical advantages: It covers as much information as possible with a limited storage capacity; This avoids meaningless template processing of rare information units and saves computing resources; This ensured that only templates with sufficient compression value were retained in the corpus, thus controlling the corpus size; Rare information units are preserved in their complete form, ensuring full coverage of information. Adaptive data insertion mechanism: During the corpus selection process, the system comprehensively considers compression value and corpus capacity constraints. When the corpus has sufficient capacity, the entry threshold is low, and most information units with a certain compression value can be included. When the corpus approaches its capacity limit, the system automatically raises the entry threshold, only including information units with the highest compression value score, while discarding entries with low scores that have not been used by downstream applications for a long time. Through this adaptive mechanism, the corpus can always maintain optimal coverage of the current application scenario under limited capacity constraints, reflecting the principle of "including as many items with compression value as possible in the corpus, provided that the corpus capacity is feasible". Example 6: Automatic Recognition of Variable Data Fields This embodiment demonstrates how the system can automatically identify variable data fields in information units without manual annotation. Methods based on statistical feature analysis: For multiple information unit instances of the same information unit type, the system aligns them and performs position-by-position statistics. For each position, the system counts the number and distribution of different values ​​at that position. When a position exhibits high variability across multiple instances (i.e., many value types with uniform distribution), while other positions remain unchanged, that position is automatically marked as a variable data field. For example, when processing system logs, the system traverses hundreds of logs of the same type and finds that the first set of characters (timestamp) is different in every log, the second set of characters (log level) only varies between ERROR / WARN / INFO, and the third set and subsequent characters (message body) are highly stable. Based on this, the system automatically identifies the timestamp portion as a variable data field, and the log level portion can be selected as a variable field or an independent template variant according to the strategy. Pattern recognition-based methods: The system has a built-in pattern library for common variable data fields, including but not limited to: date and time patterns (such as YYYY-MM-DD, HH:MM:SS), IP address patterns (such as XXX.XXX.XXX.XXX), numeric patterns (integers, floating-point numbers), and hash value patterns (fixed-length hexadecimal strings). Fields in the segmented information units are matched against these patterns; fields that match are automatically identified as variable data fields. The two methods described above can be used individually or in combination to achieve accurate identification of variable data fields. Example 7: Encoding Structure of Corpus Entry Identifiers This embodiment demonstrates a coding structure design for corpus entry identifiers. For example... Figure 5 As shown. The corpus entry identifiers use fixed-length binary encoding, with a total length of 16 bits (2 bytes), divided into two fields: Type field (high 2 bits): Indicates the type of information unit corresponding to this entry. Encoding schemes, for example: 00 represents natural language text type, 01 represents rule-driven symbolic expression type, 10 represents structured data type, and 11 is reserved for extended types. Sequence field (lower 14 bits): A unique identifier within this type. A 14-bit sequence field can support up to 16,384 entries per type. In another embodiment, the corpus entry identifiers use variable-length encoding, with the total bit width dynamically determined based on the size of the corpus. For large-scale corpora, this can be expanded to 24 or 32 bits. This structured coding design has the following advantages: (1) The identifier itself contains type information, which facilitates the distributed storage and retrieval of the corpus; (2) The separation of the type field and the sequence field makes the management of different type entries independent of each other; (3) The reserved extended coding reserves space for adding new types in the future. Example 8: Providing a corpus for secure document storage applications This embodiment demonstrates how the corpus constructed by the method of the present invention supports secure document storage and retrieval applications. Scenario Description: Enterprises need to securely store and efficiently manage their massive amounts of internal office documents. The documents cover various professional fields, including but not limited to: textual descriptions in technical documents, mathematical formulas, chemical equations, program code examples, project management forms, etc. Database creation process: The system collects enterprise documents as raw data. These documents contain various types of information units: Natural Language Text (Paragraphs, Headings, Comments) Mathematical formulas (such as quadratic equations and integral expressions) Chemical molecular formulas and reaction equations (such as structural formulas of organic compounds and balanced equations). Program code snippets (such as example functions or configuration items) Tabular data (such as project schedules, financial statements) The system segments and templates different fragment types. For example, the quadratic polynomials "x² + 2x + 1 = 0" and "y² + 3y + 2 = 0" in mathematical formulas differ in variable names and coefficients but have the same structure. They are clustered and merged into the template "{{var}}² + {{coef_a}}{{var}} + {{coef_b}} = 0". The system employs different processing strategies for different types of document content to maximize compression efficiency. For frequently occurring molecular formulas (such as H2O, O2, H2, CO2, etc.), since they do not contain variable fields, the system directly incorporates them into the corpus as complete information units and assigns them corpus entry identifiers. For frequently occurring reaction segments (such as the reactant combination 2H₂ + O₂, which appears repeatedly in various oxidation reactions), the system also stores them as complete information units in the database. Compared to template-based processing of the entire equation, treating stable reaction segments as independent entries achieves a higher compression ratio. For a complete chemical equation (such as 2H₂ + O₂ → 2H₂O), the system evaluation found that template information units generated by completely templateting the equation (with coefficients and molecular formulas as variable fields) require backfilling a large number of parameters during restoration, resulting in limited compression benefits. Based on the compression value assessment of the database entry conditions, the system chooses to either retain the entire equation as the original text or split it into combinations of the aforementioned molecular formulas and reaction fragments. The specific choice depends on the frequency of the equation's occurrence in the corpus and its degree of decomposability. It is understood that the above data types are merely illustrative. In practical applications, the types of raw data can be further extended to any rule-driven or structured expression that can be divided into information units, such as gene sequence representations, engineering drawing symbols, circuit diagram netlists, and medical image report templates. All such extensions fall within the protection scope of this invention. The resulting heterogeneous information unit library contains templated entries and complete entries for each type of information unit. Each entry in the library has a unique corpus entry identifier. Example 9: Providing a Corpus for Satellite Communication Compression Applications This embodiment demonstrates the process of constructing a corpus for satellite communication and Internet of Things compression processing applications using the method of the present invention. Scenario Description: In a satellite IoT system, numerous terminal devices report sensor data and status information to the satellite. Data formats include JSON structured data, binary sensor messages, and signaling control messages. Database creation process: The system collects communication protocol specification documents and historical communication data as raw data. It then segments the data based on data type: JSON sensor data is segmented according to the boundaries of the recorded objects; Signaling messages are segmented according to the message boundaries defined by the protocol; Binary messages are segmented according to their field structure. During the template processing phase, the system compares JSON data of the same type and finds that the data reported by different devices differ only in fields such as device ID, timestamp, and sensor readings, with identical structures. The system clusters and merges this data into a JSON template, where fixed field names are preserved in their original form, and variable numeric fields are replaced with placeholders. For signaling messages (such as ATTACH_REQ attachment request, HANDOVER_CMD switching command, etc.), which are fixed-format control messages and do not contain variable fields, the system stores them as complete information units in the database.

[0011] The completed corpus includes sensor data templates, signaling templates, and binary message templates.

[0012] Example 10: Providing a corpus for system call instruction translation applications This embodiment demonstrates the process of constructing a corpus by calling a differentiated translation application using the method of the present invention. In particular, this embodiment demonstrates a corpus construction scenario with a single type of data as input. Scenario Description: The security system needs to translate system call instructions in the operating system to prevent malware from performing illegal operations through system calls. The corpus required by this application only involves machine instructions as the information unit type. Database creation process: The system uses executable files and library files (.dll, .so, etc.) of the target operating system as raw data. These files belong to a single information unit type—rule-driven symbolic representation type (machine instructions). The type information of the raw data is determined by the file format (such as ELF, PE header information) or by user specification. The system disassembles the executable file, segments it according to instruction boundaries, and obtains individual machine instructions as information units. During the template processing phase, the system performs clustering and comparison on instructions with the same opcode. For example: MOV EAX, 0x0 (System call number setting for sys_read) MOV EAX, 0x1 (System call number setting for sys_write) MOV EAX, 0x2 (sets the system call number for sys_open) These instructions differ only in the immediate numeric field (the specific value of the system call number); their opcode and register encoding portions are identical. The system clusters and merges them into the template MOV EAX, {{syscall_number}}. For instructions that do not contain variable fields (such as the SYSCALL instruction itself, the RET instruction, etc.), the system stores them as complete information units in the database. The completed corpus contains template information units as well as complete information units that have not undergone template processing. This embodiment specifically states that even if the original data contains only a single type of information unit, the method of this invention is still fully applicable. Each step in the method—type acquisition, segmentation processing, template processing, database filtering, identifier allocation, and heterogeneous database construction—can be executed completely for a single type of data. Although the constructed corpus contains only one type of information unit, its construction method fully follows the technical solution of this invention. Example 11: Providing a Corpus for Processor Instruction Fetch Acceleration Applications This embodiment demonstrates the process of constructing an instruction corpus mapping library for processor instruction fetching acceleration applications using the method of the present invention. Scenario Description: Modern processors face a bottleneck in instruction fetch bandwidth. By replacing high-frequency instruction patterns with shorter codes, more valid instructions can be stored with the same cache capacity, effectively increasing instruction fetch bandwidth. Database creation process: The system collects a large number of binary programs as raw corpora, including core library files of mainstream operating systems, binary files of popular application software, and specialized software in the target domain. The data volume can reach several terabytes of code segments. The system disassembles and segments binary programs according to the processor instruction set architecture (such as x86-64, ARM). It scans code segments with a 1-byte step and variable window length, using a disassembler to identify instruction boundaries and obtain valid instruction sequence information units. During the template processing phase, the system identifies variable fields of the same opcode instruction. For x86-64 architecture: B8 01 00 00 00 (MOV EAX, 0x1) B8 02 00 00 00 (MOV EAX, 0x2) B8 FF FF FF FF (MOV EAX, 0xFFFFFFFF) These instructions differ only in the immediate numeric field and are clustered and merged into template B8 ?? ?? ?? ?? (MOV EAX, imm32). Similarly, instructions for various opcodes are templated according to this rule. For instructions that do not contain immediate values ​​and address offsets (such as RET and NOP), the system stores them as complete information units. The frequency of each instruction and template is statistically analyzed and sorted from highest to lowest frequency. Based on the capacity budget of the target processor's instruction corpus mapping unit, the highest-ranking entries are selected for inclusion in the database. Each entry is assigned a corpus entry identifier. Multi-instruction combination template In a further implementation, the information unit is not limited to a single instruction, but can also be a sequence of instructions composed of multiple instructions. During the segmentation and template phases, the system identifies frequently occurring instruction sequence patterns. For example, a function preamble in the x86-64 architecture consists of three instructions: PUSH RBP (Machine Code: 55) MOV RBP, RSP (Machine code: 48 89 E5) SUB RSP, imm32 (Machine code: 48 81 EC ?? ?? ?? ??) This sequence repeats at the beginning of the vast majority of functions, with only the immediate numeric segment (stack space size) of the third instruction varying from function to function: Example 1: 55 48 89 E5 48 81 EC 20 00 00 00 (Allocate 32 bytes of stack space) Example 2: 55 48 89 E5 48 81 EC 40 00 00 00 (allocate 64 bytes of stack space) The system identifies the combination of these three instructions as a single information unit, compares them, generates a template 55 48 89 E5 4881 EC ?? ?? ?? ??, and assigns it a corpus entry identifier. Compared to assigning a separate information unit to each instruction, the multi-instruction combination template further improves the compression ratio. Combined template containing multiple variable fields In more complex instruction sequence patterns, a single unit of information may contain multiple variable data fields. For example, a pattern consisting of two consecutive immediate load instructions: Example 1: B8 01 00 00 00 BB 64 00 00 00 (MOV EAX, 0x1; MOV EBX, 0x64) Example 2: B8 02 00 00 00 BB C8 00 00 00 (MOV EAX, 0x2; MOV EBX, 0xC8) The system identified two variable data fields in the sequence—the 32-bit immediate values ​​of the first and second MOV instructions. A template B8 ?? ?? ?? ?? BB ?? ?? ?? ?? was generated, containing two fixed-length placeholders. This template is assigned a corpus entry identifier. It is understandable that there is no limit to the number and type (fixed-length or variable-length) of variable data fields contained in the information unit. The template may contain one or more placeholders, and the position, length and type information of each placeholder are stored in the corpus as metadata of the template information unit. The completed corpus can be deployed in different forms according to application requirements. In one embodiment, the corpus is stored in the processor's read-only memory. In another embodiment, the corpus is stored as firmware and loaded into memory upon system startup. In yet another embodiment, the corpus is deployed on a cloud server, providing corpus query services to multiple terminal devices via a network interface. Example 12: Log Compression and Security Audit Application This embodiment demonstrates the application of the method of the present invention in the scenario of operation and maintenance log compression and security auditing. Scenario Description: Enterprise IT systems generate massive amounts of operational and audit logs daily. These logs need to be retained long-term to meet compliance requirements, but storage costs are high. Meanwhile, audit logs contain sensitive information and require secure storage. Database creation and compression process: The system uses historically accumulated system logs as raw data. Log records are segmented by newline characters. Through statistical feature analysis and pattern recognition, the system automatically extracts log templates. For example: 2024-01-15 10:23:45 ERROR Connection to 10.0.0.1 failed 2024-01-15 10:24:01 ERROR Connection to 10.0.0.2 failed 2024-01-15 10:25:12 WARN Memory usage at 85% 2024-01-15 10:26:05 INFO User admin logged in The system automatically identified three log templates: {{timestamp}} ERROR Connection to {{ip}} failed {{timestamp}} WARN Memory usage at {{percent}} {{timestamp}} INFO User {{username}} logged in Each template is assigned a unique corpus entry identifier. After screening, high-frequency log templates and rare complete log entries are jointly incorporated into a heterogeneous information unit library. This corpus provides the corpus foundation for compressed storage and security auditing of logs. Layered templates to further improve compression ratio In a further implementation, the system performs hierarchical processing on the variable data fields, separating the subfields with low change frequency into independent information units and incorporating them into the corpus, while retaining only the subfields with the highest change frequency as placeholders. Taking timestamps as an example, the original log template was `{{timestamp}} ERROR Connection to {{ip}} failed`. System analysis revealed that the year, month, day, and hour portions of the timestamp (e.g., 2024-01-15 10:) remained almost unchanged within the same hour on the same day, with a much lower frequency of change than the minute and second portions. The system incorporated the year, month, day, and hour combinations (e.g., 2024-01-15 10:, 2024-01-15 11:, etc.) into the corpus as complete information units, assigning each a corpus entry identifier. The log template was updated to `{{date_hour_id}}:{{min}}:{{sec}} ERROR Connection to {{ip}} failed`, where `{{date_hour_id}}` was replaced with the corpus entry identifier for the corresponding time period, while `{{min}}` and `{{sec}}` were retained as fixed-length placeholders. Similarly, for IP addresses, the system incorporates frequently occurring IP addresses as complete information units into the corpus. The variable field of IP address in the log template is replaced with the corpus entry identifier of the corresponding IP entry, and only a very small number of IP addresses not in the corpus are stored in the original text. By employing this layered strategy of "trading database capacity for compression ratio," although the number of entries in the corpus may increase moderately, the size of a single compressed log record is significantly reduced. For systems that generate massive amounts of logs every hour and need to retain them for a long time, the storage cost of expanding the database is a one-time investment, while the amount of compressed log data continuously benefits, resulting in extremely significant overall gains. Example 13: Corpus Updates and Maintenance This example demonstrates the dynamic update mechanism of the corpus. As new data is continuously input, the system continuously monitors the frequency of occurrence of new information units. When the frequency of occurrence of a certain information unit exceeds the entry threshold, and it has sufficient compression value (long length, high frequency), the system automatically includes it in the corpus. The entry process may trigger re-clustering—the new information unit is compared with existing entries to determine whether it can be merged with existing entries or whether a new template needs to be created. Meanwhile, the system periodically evaluates the usage of existing entries. For entries that have not been used by downstream applications for a long time, the system automatically lowers their scores. When corpus capacity is limited, low-scoring entries are eliminated, freeing up space for high-value new entries. This dynamic update mechanism enables the corpus to continuously adapt to the current application scenario, demonstrating its adaptive learning capability. Example 14: Corpus Construction System like Figure 4 As shown, this embodiment provides a general corpus construction system for heterogeneous information units, including: Data acquisition module 401: Used to acquire raw data. Raw data can come from various data sources such as the local file system, network data streams, and databases. Type acquisition module 402: Used to determine the type of information unit corresponding to the raw data. Type determination methods include, but are not limited to: reading pre-labeled type information, automatic determination based on file extension or MIME type, and determination based on user input. Information unit types include at least natural language text types, rule-driven symbolic representation types, and structured data types. Segmentation module 403: This module is used to segment the original data or its internal data segments according to the information unit type to obtain information units. It incorporates multiple segmentation strategies, each corresponding to a different information unit type. Template processing module 404: Used to perform template processing, including identifying variable data fields in information units, replacing variable data fields with placeholders, and clustering and merging multiple information units of the same type that differ only in variable data fields into a template information unit with placeholders. The inbound screening module 405 is used to determine the information units to be inbound from the segmented information units and the template information units generated by template based on preset inbound conditions. The inbound conditions are based on the principle of balancing compressed value and storage capacity, and comprehensively consider factors such as frequency, length, and template coverage. Identifier allocation module 406: Used to assign corpus entry identifiers to the information units entering the corpus. The identifier uniquely identifies the corresponding entry in the corpus. Library creation module 407: Used to build a heterogeneous information unit library, which contains template information units and complete information units, and is stored in the specified storage medium. Industrial applicability This invention can be widely applied to various information processing systems that require the construction of general-purpose corpora, particularly providing corpus infrastructure for applications such as secure document storage and retrieval, satellite communication and IoT compression processing, differentiated translation of system call instructions, processor instruction fetch acceleration, log compression, and security auditing. The methods and systems of this invention can be implemented in software and deployed on servers, terminal devices, or in the cloud, demonstrating good industrial applicability. The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a general corpus for heterogeneous information units, characterized in that, include: Obtain raw data, wherein the raw data has a corresponding information unit type, and the information unit type includes at least one of the following: natural language text type, rule-driven symbolic expression type, and structured data type; Based on the information unit type, the original data or its internal data segments are segmented to obtain information units; the segmentation process includes extracting multiple information units from the same original data segment, and the extracted information units are allowed to have partial overlap in content; Based on preset conditions, information units that require template processing are templated. The template processing includes replacing variable data fields in the information unit with placeholders to generate template information units with placeholders. Based on preset data entry conditions, the data entry units are determined from the data entry units obtained through segmentation and / or the template data entry units generated through template processing. Assign a corpus entry identifier to the information unit that is entered into the corpus. The corpus entry identifier is used to uniquely identify the corresponding information unit or template information unit in the corpus. Construct a heterogeneous information unit library, which includes template information units with placeholders generated through template processing and complete information units that have not undergone template processing.

2. The method according to claim 1, characterized in that, The templated processing includes: Clustering multiple information units within the same information unit type, and merging information units that differ only in the variable data field into a single template information unit with placeholders.

3. The method according to claim 2, characterized in that, Prior to the clustering, the following is also included: The frequency of occurrence of each information unit in the original data was statistically analyzed. Information units that appear more frequently than a preset threshold are identified as information units that participate in the clustering and template processing. Information units that appear at a frequency lower than or equal to the preset threshold will be retained in their complete form.

4. The method according to claim 2, characterized in that, The variable data field includes at least one of the following: When the information unit belongs to a rule-driven symbolic representation type and is a machine instruction, the variable data fields include, but are not limited to: immediate number field, address offset field, register number field, and displacement field; When the information unit is a natural language text unit or a rule-driven symbolic expression unit, the variable data fields include, but are not limited to: numerical parameters, date and time parameters, named entities, variable names, placeholders, measurement values, numbered identifiers, and replaceable text expressions.

5. The method according to claim 1, characterized in that, In the templated processing, the variable data fields within the identified information unit include: Based on statistical feature analysis or pattern recognition of multiple information units within the same information unit type, the variable data fields in the information unit are automatically determined.

6. The method according to claim 1, characterized in that: The rule-driven symbolic representation types include at least one of the following: program code, machine instructions, mathematical formulas, chemical formulas, chemical reaction equations, gene sequence representations, and regular expressions; The structured data types include at least one of the following: sensor data messages, database records, communication signaling, configuration file items, and key-value pair data.

7. The method according to claim 1, characterized in that, The corpus entry identifier includes a type field and a sequence field. The type field indicates the type of information unit to which the corresponding information unit belongs, and the sequence field is a unique number within the same information unit type.

8. The method according to claim 1, characterized in that, The raw data comes from at least two of the following data sources: office document library, program code library, professional literature library, communication protocol specification document, processor instruction set manual, system log, and database operation record.

9. A general corpus construction system for heterogeneous information units, characterized in that, include: The data acquisition module is used to acquire raw data; The type acquisition module is used to determine the information unit type corresponding to the original data. The information unit type includes at least one of the following: natural language text type, rule-driven symbolic expression type, and structured data type. The segmentation processing module is used to segment the original data or its internal data segments according to the identified type to obtain information units; the segmentation processing includes the ability to extract multiple information units from the same original data segment, and the content of each extracted information unit is allowed to partially overlap. The template processing module is used to perform template processing on information units that need to be templated according to preset conditions. The template processing includes replacing variable data fields in the information unit with placeholders to generate template information units with placeholders. The inbound screening module is used to determine the information units to be inbound from the information units obtained by segmentation and / or the template information units generated by template processing, based on preset inbound conditions. The identifier allocation module is used to assign a corpus entry identifier to the information unit that is entered into the corpus. The corpus entry identifier is used to uniquely identify the corresponding information unit or template information unit in the corpus. The library building module is used to build a heterogeneous information unit library, which includes template information units with placeholders generated by template processing and complete information units that have not been templated.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 8.

Images