Feature value extraction method and apparatus for heterogeneous profile data

Abstract

The application provides a feature value extraction method and device for heterogeneous archive data, relates to the technical field of data management, and comprises the following steps: loading an association rule detail in response to an association task, and processing in parallel according to rule types; for a standardization sample pair containing a class rule, a large language model is called to generate a regular expression suitable for an adaptive operating environment; after regular pre-compilation and compliance verification, the regular expression is cached; heterogeneous archive feature values are extracted based on the cached regular expression, a feature value data set with a rule type identifier is generated, zero-code configuration of a containing rule is realized, the association accuracy is improved, the delivery cycle is shortened, and safety, auditability and high concurrency performance are taken into account.

Description

Technical Field

[0001] This invention relates to the technical field of data governance, and in particular to a method and apparatus for extracting feature values ​​from heterogeneous archival data. Background Technology

[0002] With the deepening of enterprise digital transformation, the demand for cross-source data association in business scenarios such as master data governance, tax invoice association, and industrial equipment history management is becoming increasingly prominent. In actual business operations, data association usually needs to be implemented based on two core logics: equality or inclusion. Among them, the "inclusion" type association rule is a key technical link to achieve accurate matching of unstructured text and semi-structured data, and its processing effect directly affects the accuracy of data association and business processing efficiency.

[0003] Current technical solutions primarily employ two implementation paths for handling association rules involving classes: one relies on manually written Java regular expressions for text feature extraction and matching, and the other uses deep learning entity extraction models to complete text feature recognition. In practice, conventional association engine systems typically use a uniform processing flow to handle different types of association rules, lacking a differentiated processing mechanism design for specific rule types.

[0004] Manually writing regular expressions is difficult and has poor maintainability; while deep learning methods are highly automated, they have problems such as being uninterpretable, relying on a large amount of labeled data, the risk of sensitive information leakage, and compilation performance bottlenecks under high concurrency, which restrict their large-scale implementation in key scenarios such as government affairs and finance. Summary of the Invention

[0005] The purpose of this invention is to provide a feature value extraction method and apparatus for heterogeneous archive data, so as to alleviate the above-mentioned technical problems in existing heterogeneous data association scenarios that include class rules.

[0006] In a first aspect, the present invention provides a method for extracting feature values ​​from heterogeneous archival data, comprising: In response to a task involving the association of heterogeneous archive data initiated under preset conditions, the full set of association rule details is loaded, and the corresponding branch is selected for parallel processing based on the type of each rule; wherein, the full set of association rule details includes inclusion class rules; For the standardized sample pairs extracted from the target data directory of the heterogeneous archive data, which contain rule parameters and processing branches including class rules, a preset large language model is invoked to output a regular expression adapted to the preset running environment; wherein, the standardized sample pairs are used to characterize the current data format and extraction requirements; The regular expression is pre-compiled and validated for compliance, and the validated regular expressions are cached. Based on cached regular expressions, feature values ​​of the heterogeneous archive data are extracted to generate a feature value dataset with rule type identifiers.

[0007] In an optional implementation, the steps of loading the full set of association rules and selecting the appropriate branch for parallel processing based on the type of each rule in response to an association task initiated for preset conditions include: In response to user operation commands, system timed scheduling, or upstream data update events, trigger the initiation of associated tasks for heterogeneous archive data; Load and simultaneously traverse the full list of associated rules in parallel, and identify the rule type field of each rule to determine its type; For equality rules, the message digest branch is used directly to generate feature values. At the same time, for inclusion rules, a context object is generated to store rule parameters, triggering AI-assisted branch parallel processing.

[0008] In an optional implementation, the step of calling a preset large language model to output a regular expression adapted to a preset operating environment for standardized sample pairs extracted from the target data directory of the heterogeneous archive data, which includes rule parameters and processing branches containing class rules, includes: For the rule parameters and processing branches containing class rules, locate the target data directory of the heterogeneous archive data and extract standardized sample pairs from the target data directory; The standardized sample pairs are analyzed for text structure complexity, feature dispersion, and extraction constraint strength to quantify the strength score of the current feature extraction requirement. Based on the extraction demand strength score, a target large language model is dynamically selected from multiple preset large language models adapted to different complexity scenarios, or an adaptive inference weight corresponding to the extraction demand strength is assigned to each preset large language model. The preset system constraint instructions and the standardized sample pairs are encapsulated into standardized prompt words; The standardized prompt words are input into a pre-deployed target large language model, and a regular expression adapted to the preset running environment is output.

[0009] In an optional implementation, the step of locating the target data directory of the heterogeneous archive data and extracting standardized sample pairs from the target data directory, for rule parameters and processing branches containing class rules, includes: Based on the AI-assisted branch to which the inclusion class rule belongs, the rule parameters in the inclusion class rule, including the pre-configured rule direction, left and right related directories and target related fields, are read to locate the target data directory corresponding to the heterogeneous archive data; From the located target data directory, perform targeted random sampling of the real business data under the target associated field according to the preset sampling quantity, and automatically remove null values, garbled characters and invalid text data, while de-identifying and normalizing the sampled data; Each data point after normalization is split into the source text to be extracted and the expected target fragment, thus constructing a standardized sample pair with a uniform format that is used to characterize the extraction features of the current business data.

[0010] In an optional implementation, the step of inputting the standardized prompt words into a pre-deployed target large language model and outputting a regular expression adapted to a preset operating environment includes: The target large language model filters out non-compliant format content from the regular expression candidate results output based on the standardized prompt words to obtain a regular expression adapted to the preset running environment. or, The standardized prompt words are input into the target large language model composed of a multi-model fusion inference link after weight allocation, and multi-way regularization candidate results are obtained. Based on the standardized sample pairs, the multiple regular expression candidate results are verified in parallel and fused to generate a target regular expression that is adapted to the preset operating environment.

[0011] In an optional implementation, the steps of precompiling and validating the regular expression, and caching the validated regular expressions, include: The regular expression is pre-compiled to generate a reusable pre-compiled object; Based on the pre-compiled object, sample extraction operations are performed in parallel on the source text of the standardized sample pairs; The extraction coverage and matching accuracy are calculated based on the target segments in the extracted samples and the standardized sample pairs, respectively. If both the extraction coverage and the matching accuracy meet the preset thresholds, the regular expression is deemed to be valid and is persisted and cached.

[0012] In an optional implementation, it further includes: New standardized prompt words are generated using the uncovered sample set consisting of samples whose extraction coverage or matching accuracy does not meet a preset threshold, and preset system constraint instructions. The preset large language model is invoked again, and the new standardized prompt word is used as input to generate a supplementary regular expression. This supplementary regular expression is then merged with the regular expression to obtain a multi-regular expression combination. The multiple regular expression combinations that pass verification within a preset number of iterations will be persisted and cached.

[0013] In an optional implementation, the step of extracting feature values ​​from the heterogeneous archive data based on cached regular expressions and generating a feature value dataset with rule type identifiers includes: The cached regular expressions are loaded first, and feature values ​​are extracted from the preset associated fields corresponding to the heterogeneous archive data by regular expression matching. When the regular expression is invalid, the extraction result is empty, or the extraction is abnormal, it is determined that the feature value extraction has failed, and the original field value of the preset associated field is directly obtained. The feature values ​​extracted by regular expressions or the original field values ​​are normalized to generate a feature value dataset with rule type identifiers. Anomaly alarm information is recorded synchronously to ensure the continuous execution of related tasks.

[0014] Secondly, the present invention provides a feature value extraction device for heterogeneous archival data, comprising: The parallel module, in response to the association task of heterogeneous archive data initiated under preset conditions, loads the full set of association rule details and selects the corresponding branch for parallel processing according to the type of each rule; wherein, the full set of association rule details includes inclusion class rules; The calling module, for the standardized sample pairs extracted from the target data directory of the heterogeneous archive data, which contain rule parameters and processing branches including class rules, calls the preset large language model to output a regular expression adapted to the preset running environment; wherein, the standardized sample pairs are used to characterize the current data format and extraction requirements; The validation module pre-compiles and performs compliance validation on the regular expressions, and caches the validated regular expressions. The generation module extracts feature values ​​from the heterogeneous archive data based on cached regular expressions, and generates a feature value dataset with rule type identifiers.

[0015] Thirdly, the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the method as described in any of the foregoing embodiments.

[0016] This invention provides a method and apparatus for feature extraction from heterogeneous archival data. First, by using rule type identification and dual-branch parallel processing, it decouples equality rules from inclusion rules, ensuring the stability of existing mature business processes while providing an independent path for the specific optimization of inclusion rules. Second, for inclusion rules, based on standardized sample pairs extracted from real business data, a private large language model is used to automatically generate regular expressions, replacing the traditional method of manually writing regular expressions and significantly reducing the configuration threshold for inclusion rules. Third, the generated regular expressions undergo pre-compilation and compliance verification, caching only those that pass verification. This pre-compilation reduces CPU resource consumption while ensuring the extraction effect of the regular expressions through pre-verification. Finally, based on the cached regular expressions, feature values ​​of the entire dataset are extracted, generating a feature value dataset with rule type identifiers, providing a standardized data foundation for subsequent association matching. Through the aforementioned comprehensive technical means, the core technical problems of existing technologies, such as reliance on manually written regular expressions for class rules, high barriers to entry, difficulty in covering multiple formats, lack of auditability due to black boxes, high risk of data leakage, and performance bottlenecks in high-concurrency scenarios, have been comprehensively solved. This enables business personnel to configure class rules with zero code, significantly improves the accuracy of data association, shortens the project delivery cycle, and at the same time takes into account data security, auditability, high concurrency performance, and high availability. It can be directly embedded into existing association engines and is suitable for various business scenarios such as enterprise-level master data governance, tax invoice association, and industrial equipment history collection.

[0017] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart of a feature value extraction method for heterogeneous archive data provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the functional modules of a feature value extraction device for heterogeneous archive data provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Current methods using manually written regular expressions have significant technical limitations: firstly, they require developers to possess professional regular expression writing skills, resulting in a high technical threshold; secondly, when dealing with multi-format text processing, the coverage and maintenance difficulty of regular expression rules increase significantly. While deep learning-based methods improve the automation of feature extraction to some extent, they suffer from technical bottlenecks such as the uninterpretable model inference process and the need for large amounts of labeled data for training. In scenarios with strict data security requirements, such as government and finance, there is also the risk of information leakage due to the uploading of sensitive data. Furthermore, existing solutions perform poorly in online compilation of regular expressions, easily triggering system task circuit breakers in high-concurrency business scenarios. These issues all hinder the large-scale application and promotion of data association technology.

[0023] Based on this, the feature value extraction method and apparatus for heterogeneous archive data provided in this embodiment of the invention can significantly reduce the threshold for rule configuration, improve the accuracy of data association and business delivery efficiency by automatically triggering AI to generate high-precision regular expressions for rules containing classes, while meeting the data security requirements and high concurrency performance requirements of strong compliance scenarios.

[0024] To facilitate understanding of this embodiment, a feature value extraction method for heterogeneous archive data disclosed in this embodiment of the invention will first be described in detail.

[0025] Figure 1 The flowchart illustrates a feature value extraction method for heterogeneous archive data provided in this embodiment of the invention.

[0026] Reference Figure 1 The method includes the following steps: S102: In response to the association task of heterogeneous archive data initiated for preset conditions, load the full association rule details and select the corresponding branch for parallel processing according to the type of each rule; Among them, the full set of association rules details include inclusion class rules and equality class rules; This step is the starting point of the invention. First, step 1.1 can start the association task based on preset trigger conditions, complete the loading and type identification of all association rules, allocate dedicated processing branches for different types of rules and realize parallel processing. Among them, heterogeneous archive data refers to unstructured and semi-structured business archive data scattered in different business systems and different storage formats (such as PDF, Excel, text files, etc.), such as corporate tax invoices, industrial equipment records, purchase contracts, etc. The association task refers to matching and associating the scattered heterogeneous archive data according to preset business rules to realize the task of data collection and governance, such as associating and matching equipment records and purchase invoices by equipment model. The association rule details refer to the pre-configured rule information used to standardize the data association logic, including core contents such as rule type, associated data directory, associated fields, priority, etc. The inclusion class rule refers to the association rule where the data association logic is a text containing a target feature substring, such as the equipment record text containing the equipment model number, which is the core processing object of the invention.

[0027] This application achieves complete decoupling between the regular and efficient processing of equality rules and the AI-assisted processing of inclusion rules by pre-identifying rule types and using dual-branch parallel processing. This ensures the stability of mature business processes and provides an independent link for the special optimization of inclusion rules, avoiding mutual interference between the processing processes of different types of rules. At the same time, parallel processing significantly improves the processing efficiency of all rules.

[0028] In some embodiments, step S102 may be further implemented through the following refined steps, including: 2.1: In response to user operation commands, system timed scheduling, or upstream data update events, trigger the initiation of associated tasks for heterogeneous archive data; Here, associated tasks can adopt three types of legitimate triggering conditions, covering three core enterprise-level business scenarios: manual triggering, scheduled automatic triggering, and automatic triggering due to data changes. This ensures that associated tasks can adapt to different business governance needs. Through the full coverage of triggering conditions across multiple scenarios, flexible scheduling of associated tasks is achieved. This satisfies both the needs of business personnel for manual temporary governance and the need for scheduled automated data governance. At the same time, tasks are automatically triggered when upstream data is updated, ensuring the real-time performance and consistency of associated data.

[0029] 2.2: Load and simultaneously traverse the full list of related rules in parallel, and identify the rule type field of each rule for type determination; Here, while loading the full list of association rules, a parallel traversal mechanism is initiated. Based on the rule type field of each rule, type identification is performed to distinguish between equality rules and inclusion rules. Equality rules refer to association rules where the values ​​of the left and right side fields are completely equal, such as the invoice number field and the ledger number field being completely identical. By executing loading and traversal in parallel, the overall time spent on rule loading and identification is significantly reduced. Furthermore, the type judgment based on fixed fields ensures the accuracy and consistency of rule classification, laying a precise classification foundation for subsequent dual-branch parallel processing.

[0030] 2.3: For equality rules, the message digest branch is directly used to generate feature values. At the same time, for inclusion rules, a context object is generated to store rule parameters, triggering AI-assisted branch parallel processing.

[0031] Based on the rule classification results, dedicated processing branches are assigned to the two types of rules. For equality rules, a message digest branch is used to generate feature values ​​using the MD5 message digest algorithm. For inclusion rules, a dedicated context object is generated to store all parameter information for that rule, and simultaneously triggers AI-assisted branch processing. The two branches execute in complete parallel. The message digest branch refers to the processing chain that generates a unique digest value from the concatenated field values ​​using the MD5 algorithm; this is a standard and efficient processing chain for equality rules. The context object is a standardized object used to store all parameters of a single rule, enabling lossless transmission of rule parameters throughout the entire process.

[0032] This application employs fully parallel processing through two branches. The equality class rules utilize a mature and efficient message digest chain, ensuring the processing efficiency and stability of core businesses without requiring modifications to the existing mature processes. The inclusion class rules achieve parameter isolation through a dedicated context object, triggering independent AI-assisted branch processing. This achieves complete decoupling of the two types of rules, avoiding the impact of AI processing on the existing mature businesses. At the same time, parallel processing significantly improves the processing efficiency of all rules.

[0033] In practical applications, as an optional embodiment, step S102 may include the following process: First, the system monitors the three types of trigger sources for associated tasks in real time. When a manual operation command initiated by a business user on the front-end page is received, the associated task of the corresponding task dimension is immediately triggered. When the system clock reaches the preset timed scheduling point, the associated task of the corresponding period is automatically triggered. When a data update event is received from the upstream business system, the incremental associated task of the corresponding data dimension is automatically triggered to ensure that the associated data is synchronized with the upstream business data in real time. Then, after the task is triggered, the system immediately reads the full set of associated rule details corresponding to the task from the database through the configuration loading module. While loading the rule details, a multi-threaded parallel traversal mechanism is started to immediately perform type identification on the loaded rule details without waiting for the full set of rules to be loaded, which greatly shortens the overall processing time. In addition, during the traversal process, the system reads the value of the RuleType field in each rule detail to accurately determine the rule type, distinguish between equality rules and inclusion rules, and records alarm logs for illegal rule types and skips the processing.

[0034] Then, for the identified equality rules, the system directly assigns them to the message digest branch for processing without triggering the AI-assisted module, ensuring efficient processing of equality rules. For the identified inclusion rules, the system generates a dedicated thread-safe context object, storing all parameters of the rule, including rule detail ID, rule direction, left and right related directories, target related fields, and priority level, in this context object. This achieves parameter isolation between different rules and avoids parameter confusion. Furthermore, the system adds the message digest branch for equality rules and the AI-assisted branch for inclusion rules to a parallel processing queue. A thread pool enables fully parallel execution of the two branches without interference, ensuring efficient processing of equality rules while providing an independent execution path for the AI-assisted processing of inclusion rules.

[0035] S104: For the standardized sample pairs extracted from the target data directory of heterogeneous archive data, which contain rule parameters and processing branches containing class rules, call the preset large language model to output a regular expression adapted to the preset running environment; wherein, the standardized sample pairs are used to characterize the current data format and extraction requirements. This step is the core generation stage of this application, and it is only executed on the dedicated processing branch containing class rules. Based on the rule parameters, the target data source is located and standardized sample pairs are extracted. The sample pairs are used as input to call the large language model to generate regular expressions adapted to the target operating environment. Here, the standardized sample pairs refer to the business samples formed by extracting and standardizing real business data, which correspond one-to-one with <source text, target fragment>, and are used to convey the data format and extraction requirements to the large language model. The large language model refers to a pre-trained language model with natural language understanding and code generation capabilities. In this invention, it is a model deployed privately on the enterprise intranet and is used to generate regular expressions. Regular expressions are logical expressions used to match text features and can realize the function of accurately extracting target substrings from long texts.

[0036] Based on real business samples, the large language model automatically generates regular expressions, eliminating the need for business personnel to master regular expression syntax. This significantly lowers the configuration threshold for rules that include class rules. Furthermore, privately deployed models only require uploading sampled and anonymized data, without transmitting the full amount of sensitive data, thus mitigating the risk of data leakage at its source. The generated regular expressions are human-readable logical statements that fully meet the auditability requirements of government and financial scenarios.

[0037] Based on this, step S104 can also generate a regular expression through the following steps: 3.1: For rule parameters and processing branches containing class rules, locate the target data directory of heterogeneous archive data and extract standardized sample pairs from the target data directory; This step, based on rule parameters, precisely locates the target data directory and extracts standardized sample pairs, serving as the foundational data preparation for subsequent regular expression generation. Sample extraction is performed only for rules containing certain classes, avoiding invalid processing of rules with the same classes. Furthermore, the precise location of the data source based on rule parameters ensures that the extracted samples perfectly match the business requirements of the current rules, providing a high-quality, highly matched sample foundation for subsequent regular expression generation.

[0038] In some embodiments, step 3.1 for extracting standardized sample pairs can also be achieved through the following steps: 4.1: Based on the AI-assisted branch to which the inclusion class rule belongs, read the rule parameters in the inclusion class rule, including the pre-configured rule direction, left and right related directories and target related fields, and locate the target data directory corresponding to the heterogeneous archive data; This step, based on the AI-assisted branch to which the inclusion rule belongs, reads the pre-configured core parameters in the rule. The rule direction refers to the inclusion matching direction of the association rule, divided into two categories: left-side inclusion of right-side and right-side inclusion of left-side. The left and right association directories refer to the storage directories of the data on the left and right sides associated by the rule. The target association field refers to the specific business field from which feature values ​​need to be extracted. Based on these parameters, the target data directory is accurately located, limiting the data source range for sample extraction. Accurately locating the data source based on the rule parameters ensures that the extracted samples come entirely from the target data that the current rule needs to process, avoiding sample bias caused by cross-directory sampling. Simultaneously, determining the sampling side based on the rule direction ensures that the samples are completely consistent with the subsequent inclusion matching direction, significantly improving the relevance and adaptability of the samples.

[0039] 4.2: From the located target data directory, perform targeted random sampling of the real business data under the target associated fields according to the preset sampling quantity, and automatically remove null values, garbled characters and invalid text data, while de-identifying and normalizing the sampled data; This step involves performing targeted random sampling from the target data directory, selecting 50 samples from the actual business data under the target associated fields. This ensures the randomness and representativeness of the samples, while automatically filtering out null values, garbled characters, and invalid text data to prevent invalid samples from interfering with regular expression generation. The sampled data is then anonymized and normalized to remove sensitive information such as ID numbers, mobile phone numbers, and company secrets, while standardizing text encoding, line breaks, spaces, and other formatting. Targeted random sampling ensures that the samples fully represent the format characteristics of the entire dataset, avoiding poor regular expression compatibility due to a single sample. Automatic filtering of invalid data ensures that all samples entering subsequent stages are valid business data, improving the accuracy of regular expression generation. Anonymization and normalization completely avoid the risk of leakage from sensitive data being passed into the large language model, meeting compliance requirements.

[0040] 4.3: Each data point after normalization is split into the source text to be extracted and the expected target fragment, and a standardized sample pair with uniform format is constructed to represent the current business data extraction characteristics.

[0041] Each sampled data point, after being normalized, is then split into two corresponding parts: the source text, which is the complete original business text containing the target information, and the target fragment, which is the key feature substring that the business needs to extract from the source text, i.e., the standard answer used in subsequent association matching. These two parts are then encapsulated into standardized sample pairs in the format of <source text, target fragment>. By splitting and constructing standardized sample pairs, the business extraction requirements are transformed into examples that the large language model can directly understand, eliminating the need for manually writing complex requirement descriptions. This allows the large language model to directly learn data features and extraction requirements through examples, significantly improving the accuracy of regular expression generation. At the same time, the unified format ensures the standardization of prompt word encapsulation, avoiding output fluctuations.

[0042] In practical applications, step 3.1 can also be implemented through the following processes, including: First, for inclusion rules entering the AI-assisted branch, the system reads all pre-configured rule parameters from the rule's context object. These parameters include rule detail ID, rule direction, left and right related directories, target related fields, and priority level. The rule direction clearly defines the inclusion matching logic, categorized into two types: left-side data including right-side data, and right-side data including left-side data. Then, based on the rule direction and related directory parameters, the system precisely locates the target data directory for this sampling. When the rule direction is left-side including right-side, the target data directory is the right-side related directory specified by the rule; when the rule direction is right-side including left-side, the target data directory is the left-side related directory specified by the rule, ensuring that the sampled data source completely corresponds to the subsequent inclusion matching logic.

[0043] Secondly, the system performs targeted random sampling of real business data under the target-related fields from the located target data directory, according to a preset sampling quantity of 50. During the sampling process, it ensures that each data point has an equal probability of being selected, guaranteeing the randomness and representativeness of the samples and fully reflecting the format characteristics and distribution patterns of the entire dataset. Then, the system automatically filters the sampled data, removing invalid data with empty field values, containing garbled characters, or lacking valid business content. If the number of valid samples after filtering is insufficient to meet the preset sampling quantity, the system automatically supplements the sampling to ensure that the final number of valid samples meets the preset requirements. Furthermore, the system performs desensitization and normalization processing on the filtered valid samples. Using preset sensitive information identification rules, it masks sensitive information such as ID card numbers, mobile phone numbers, bank account numbers, and trade secrets in the samples, while standardizing the text encoding format, line breaks, full-width and half-width characters, spaces, and other formatting elements to ensure standardized sample format.

[0044] Then, the system performs a splitting process on each de-identified and normalized sample data. The complete business text containing the target information in the sample is used as the source text to be extracted, and the key feature substrings required by the business rules are used as the expected target fragments. This ensures a one-to-one correspondence between the source text and target fragment for each sample, without deviation or omission. Furthermore, the system encapsulates the split source text and target fragments into standardized sample pairs in the format <source text, target fragment>. All sample pairs adopt a unified format specification, ultimately forming a complete list of standardized sample pairs for subsequent requirements analysis and regular expression generation.

[0045] 3.2: Analyze the text structure complexity, feature dispersion, and extraction constraint strength of standardized sample pairs to quantify the strength score of the current feature extraction requirements; Here, standardized sample pairs are subjected to multi-dimensional quantitative analysis. Text structure complexity refers to indicators such as the length, format complexity, and proportion of special characters in the sample source text; feature dispersion refers to the degree of difference in the format, length, and character type of the target fragment in the sample; extraction constraint strength refers to the level of business requirements for extraction accuracy. Based on the analysis results of the three dimensions, a weighted algorithm is used to quantify the extraction demand strength score from 0 to 100 points, with higher scores indicating greater extraction difficulty.

[0046] By conducting multi-dimensional quantitative analysis of sample pairs, the fuzzy extraction requirements are transformed into quantifiable scoring indicators, providing a precise quantitative basis for the dynamic selection or weight allocation of subsequent large language models. This avoids the problems of resource waste in simple scenarios and insufficient accuracy in complex scenarios when a fixed model is adapted to different difficulty scenarios.

[0047] 3.3: Based on the strength score of extraction needs, dynamically select the target large language model from multiple preset large language models adapted to different complexity scenarios, or assign adaptive inference weights corresponding to the strength of extraction needs to each preset large language model. This step, based on the quantitatively obtained demand strength scores, implements two adaptation strategies: First, dynamic model selection: lightweight large language models are used for simple scenarios with low scores, while high-precision large language models are used for complex scenarios with high scores. Second, multi-model weight fusion: adaptive weights are assigned to different models according to their scores, increasing the weight of lightweight models for simple scenarios and increasing the weight of high-precision models for complex scenarios. Lightweight large language models refer to models with fewer parameters, faster inference speed, and lower resource consumption, suitable for generating regular expressions in simple formats; high-precision large language models refer to models with more parameters, stronger logical understanding capabilities, and higher generation accuracy, suitable for generating regular expressions in complex and multi-format scenarios.

[0048] Based on the dynamic adaptation of large language models to extraction difficulty, it significantly reduces inference time and resource consumption in simple scenarios through lightweight models, and ensures the accuracy of regular expression generation through high-precision models or multi-model fusion in complex scenarios. It balances the efficiency, accuracy and resource consumption of regular expression generation, and achieves optimal adaptation for scenarios with different levels of complexity.

[0049] 3.4: Encapsulate the preset system constraint commands and standardized sample pairs into standardized prompt words; This step encapsulates fixed system constraint instructions and dynamic standardized sample pairs into standardized prompts that can be recognized by the large language model, according to a preset format. The system constraint instructions are pre-defined standardized texts used to regulate the output behavior of the large language model, while the sample pairs are included in the prompts as user examples, guiding the large language model to accurately understand the extraction requirements. By encapsulating these standardized prompts, the constraint specifications of each large language model call are ensured to be consistent, avoiding fluctuations in output format and syntax. Simultaneously, the use of real sample examples allows the large language model to accurately understand the current business extraction requirements, significantly improving the accuracy and adaptability of regular expression generation.

[0050] 3.5: Input standardized prompt words into the pre-deployed target large language model and output regular expressions adapted to the preset running environment.

[0051] This step involves inputting the encapsulated standardized prompts into a pre-deployed private target large language model via a dedicated intranet channel. The model then returns regular expression results, ensuring all data interactions are completed within the intranet, eliminating the risk of sensitive data leakage. By using a privately deployed large language model, the risk of data leakage from uploading all sensitive data to a public cloud model is completely avoided, meeting compliance requirements for information security standards and GDPR. Furthermore, based on a dynamically selected target model, regular expressions adapted to the preset Java runtime environment are generated, ensuring that the generated regular expressions can be directly run in the association engine without secondary modification.

[0052] In some embodiments, step 3.5 may be implemented by the following steps, including: 5.1: Filter non-compliant format content from the regular expression candidate results output by the target large language model based on standardized prompt words to obtain a regular expression adapted to the preset running environment; Here, the candidate regular expression results output by the single-objective large language model are filtered for non-compliant content. This involves removing non-regular content such as Chinese explanations, Markdown formatting, code block markers, line breaks, and redundant comments, retaining only pure regular expression strings that conform to Java syntax specifications. This non-compliant content filtering ensures that the final output regular expression can be directly compiled and executed in the Java runtime environment without manual modification. This completely solves the problems of redundant and non-standardized output content from large language models, improving the automation level of the entire process.

[0053] Alternatively, generate a regular expression based on the methods provided in steps 5.2-5.3: 5.2: Input the standardized prompt words into the target large language model composed of a multi-model fusion inference link after weight allocation to obtain multi-way regularization candidate results; This step simultaneously inputs standardized prompt words into multiple private large language models after weight allocation, and performs inference in parallel to obtain multiple regularization candidate results. The inference weight of each model corresponds to the strength score of the extraction requirement. By using multi-model parallel inference and combining the strengths of different models, the limitations of a single model are avoided, significantly improving the accuracy and adaptability of regularization generation in complex scenarios. At the same time, parallel inference ensures the efficiency of multi-model processing without significantly increasing the overall processing time.

[0054] 5.3: Based on standardized sample pairs, perform parallel verification and optimal fusion of multiple regularization candidate results to generate a target regular expression that adapts to the preset operating environment.

[0055] Based on standardized sample pairs, pre-compilation verification and extraction effect validation are performed in parallel on multiple candidate regular expressions. The extraction coverage and matching accuracy of each candidate regular expression are calculated. Based on the validation results, the optimal regular expression is selected, or multiple high-accuracy regular expressions are fused together using an OR relationship to generate the final target regular expression. By performing parallel validation based on real samples, it is ensured that the extraction effect of each candidate regular expression has been verified in practice, avoiding invalid regular expressions caused by model illusion. At the same time, by selectively fusing multiple regular expressions, the advantages of multiple regular expressions are combined, achieving full coverage of multi-format data and significantly improving the extraction coverage and matching accuracy of regular expressions.

[0056] In practical applications, step 3.5 can be implemented through the following process: First, the system can use a single-model inference processing link to send the encapsulated standardized prompt words to a dynamically selected single-target large language model, receive the regular expression candidate results returned by the model, and then perform non-compliant format content filtering on the candidate results. The candidate results are scanned line by line to remove all non-regular content such as Chinese explanations, Markdown code block markers, line breaks, extra spaces, and code comments, and only pure regular expression strings that conform to Java regular expression syntax specifications are retained as the final output regular expression.

[0057] In addition, the system can also use a multi-model fusion inference processing link to send the encapsulated standardized prompt words to multiple private large language models after weight allocation. The inference task is executed in parallel by multiple threads, and the regularization candidate results returned by each model are received to form a set of multiple regularization candidate results. The parallel inference method ensures that the overall time consumption of multi-model processing is basically the same as that of single-model inference, and will not significantly increase the process time.

[0058] Then, the system performs parallel validation on standardized sample pairs for multiple candidate regular expressions. For each candidate regular expression, pre-compilation validation is performed first, filtering out invalid regular expressions with syntax errors or those that cannot be compiled correctly. For the correctly compileable regular expressions, extraction effectiveness is validated on standardized sample pairs, calculating the extraction coverage and matching accuracy for each regular expression, thus completing a quantitative evaluation of the effectiveness of all candidate regular expressions. Furthermore, based on the validation results, the system performs a selective fusion process, prioritizing the candidate regular expression with the highest extraction coverage and matching accuracy as the base regular expression. Simultaneously, supplementary regular expressions from other high-accuracy candidate regular expressions that can cover samples not covered by the base regular expression are merged with the base regular expression using an OR relationship, ultimately generating a target regular expression that adapts to the preset Java runtime environment and balances coverage and accuracy.

[0059] As an optional embodiment, the aforementioned step S104 mainly includes the following process: First, for the inclusion-class rules that enter the AI-assisted processing branch, the system reads the rule parameters stored in the rule's context object. Based on the left and right related directories and the target related fields in the rule, it accurately locates the target data directory corresponding to the heterogeneous archive data, limiting the data source range for sample extraction and ensuring that the extracted samples fully match the business requirements of the current rule. Then, the system performs targeted random sampling from the located target data directory according to the preset sampling quantity. The sampled data is filtered, de-identified, and normalized, splitting each data item into source text and target fragments to construct standardized sample pairs with uniform format, thus completing the sample data preparation.

[0060] Secondly, the system performs multi-dimensional quantitative analysis on the generated standardized sample pairs. First, it analyzes the text structure complexity of each sample source text, and counts indicators such as text length, format type, and proportion of special characters. Then, it analyzes the feature dispersion of the target segments in all samples, and counts indicators such as format differences, length fluctuations, and character type distribution of the target segments. At the same time, combined with the extraction constraint strength in the rule configuration, the three-dimensional indicators are quantified into a feature extraction demand strength score of 0-100 points through a preset weighted algorithm. The higher the score, the greater the difficulty of the current extraction task.

[0061] Then, based on the quantitatively obtained extraction demand strength score, the system executes the corresponding model adaptation strategy. When the score is lower than the preset simple scenario threshold, a lightweight large language model is dynamically selected from the preset model library as the target model, or a higher inference weight is assigned to the lightweight model. When the score is higher than the preset complex scenario threshold, a high-precision large language model is dynamically selected as the target model, or a higher inference weight is assigned to the high-precision model. When the score is in the middle range, a multi-model fusion inference link is activated, and adaptive weights corresponding to the scores are assigned to different models. In addition, all models in the system's preset model library have been deployed privately on the enterprise intranet, and all data interactions are completed within the closed loop of the intranet, eliminating the risk of data leakage.

[0062] Furthermore, the system encapsulates pre-defined system constraint instructions and generated standardized sample pairs into standardized prompt words according to a preset format. The system constraint instructions clearly define the core constraints such as the role positioning of the large language model, Java syntax adaptation requirements, output format specifications, and capability requirements, ensuring that the model's output meets business needs. Then, the system sends the encapsulated standardized prompt words to the selected target large language model or the multi-model fusion inference link after weight allocation through a dedicated AI Client module on the intranet, and receives the regularization candidate results returned by the model.

[0063] Finally, the system filters the regular expression candidate results returned by the model, removing non-compliant content such as Chinese explanations, Markdown format, and line breaks, and retaining only the pure regular expression string to obtain a regular expression adapted to the preset Java runtime environment, which provides a foundation for subsequent verification and extraction stages.

[0064] S106: Perform pre-compilation and compliance validation on regular expressions, and cache the validated regular expressions; This step is the method's verification and caching phase. The generated regular expression is first pre-compiled, then compliance verification is performed. Only regular expressions that pass verification are persisted and cached. Pre-compilation refers to using the Java environment's `Pattern.compile()` method to convert the regular expression string into a reusable `Pattern` object, avoiding the CPU resource consumption caused by repeated compilation. Compliance verification refers to verifying the extraction effect of the regular expression based on sampled data, ensuring that the extraction capability of the regular expression meets business requirements.

[0065] Here, pre-compilation reduces the number of compilations for regular expressions from O(n) times when matching all data to 1 time, significantly reducing CPU resource consumption in high-concurrency scenarios and avoiding task timeouts and circuit breakers. Through pre-compliance verification, it is ensured that only regular expressions that meet the business extraction requirements will be cached and used, fundamentally avoiding the problem of mismatched regular expressions leading to the misassociation of millions of data points and ensuring the accuracy of data association.

[0066] Specifically, step S106 can be achieved through the following steps, including: 6.1: Precompile regular expressions to generate reusable precompiled objects; This step performs pre-compilation processing on the generated regular expressions. Using the `compile()` method of the `java.util.regex.Pattern` class in the Java environment, the regular expression string is converted into a reusable `Pattern` pre-compiled object. This object can be directly used for subsequent matching and extraction operations without repeated compilation. This pre-compilation process moves the compilation of regular expressions from the full data extraction stage to the validation stage, reducing the number of compilations from O(n) times for full data to once. This significantly reduces CPU resource consumption in high-concurrency scenarios and completely solves the task timeout and circuit breaker problem caused by repeated online compilation of regular expressions in existing technologies. Furthermore, pre-compilation can identify syntax errors in regular expressions in advance and filter out invalid regular expressions.

[0067] 6.2: Parallel sample extraction operations are performed on the source text of standardized sample pairs based on pre-compiled objects; Here, based on the generated pre-compiled objects, feature extraction is performed simultaneously on each source text in the standardized sample pair using multi-threaded parallel computation, yielding the extraction result for each sample without needing to recompile regular expressions for each sample. By executing sample extraction operations in parallel, the originally sequential, line-by-line verification time is compressed from seconds to milliseconds, significantly improving the processing efficiency of the verification stage. Furthermore, reusing pre-compiled objects further reduces CPU resource consumption in the verification stage, enhancing the system's high-concurrency processing capabilities.

[0068] 6.3: Calculate the extraction coverage and matching accuracy based on the target segments in the extracted samples and standardized sample pairs. If both the extraction coverage and matching accuracy meet the preset thresholds, the regular expression is deemed to be valid and is persisted and cached.

[0069] Here, based on the extracted results and the target fragments in the sample pairs, two core validation metrics are calculated: extraction coverage, which is the proportion of samples with non-empty values ​​extracted out of the total number of samples, used to measure the coverage capability of the regular expression; and matching accuracy, which is the proportion of samples whose extracted results completely match the target fragments out of the total number of samples, used to measure the extraction precision of the regular expression. Only when both metrics meet a preset 80% threshold is the regular expression considered valid, and valid regular expressions are persistently stored and distributedly cached. By validating both extraction coverage and matching accuracy, it is ensured that the regular expression covers the vast majority of samples and that the extracted results fully meet business requirements. This avoids the problems of high coverage but low precision or high precision but incomplete coverage that can occur with single-metric validation, thus guaranteeing the extraction effect of the regular expression from the root. Simultaneously, persistence and caching ensure that the regular expression can be quickly invoked in subsequent full extraction stages and can be directly reused by subsequent tasks with the same rules without repeated generation.

[0070] For example, step S106 can also be implemented by the following steps, including: First, upon receiving the generated regular expression, the system immediately calls the Java environment's `Pattern.compile()` method to perform pre-compilation processing, converting the regular expression string into a thread-safe, reusable `Pattern` pre-compiled object. If a syntax exception is thrown during pre-compilation, the regular expression is deemed invalid, the exception is logged, and subsequent iterative optimization processes are triggered. Then, the system stores the pre-compiled `Pattern` object in memory, providing a reusable object for subsequent parallel verification, thus avoiding repeated compilation.

[0071] Secondly, the system initiates a multi-threaded parallel computing framework. Based on pre-compiled Pattern objects in memory, it simultaneously performs feature extraction operations on each source text in the standardized sample pair list, obtaining the extraction result for each source text. The number of parallel threads can be adaptively adjusted according to the number of CPU cores on the server, maximizing server resource utilization while ensuring thread safety. Furthermore, during the parallel extraction process, the system synchronously records the extraction status of each sample, including whether a non-null value was extracted, the specific content of the extraction result, and the extraction time, providing complete data support for subsequent metric calculations.

[0072] Then, based on the results of parallel extraction and the target fragments in the standardized sample pairs, the system calculates two core metrics: extraction coverage and matching accuracy. First, it counts the number of samples with non-empty values ​​extracted, divides this number by the total number of samples to obtain the extraction coverage. Then, it counts the number of samples whose extraction results are completely identical to the target fragment, divides this number by the total number of samples to obtain the matching accuracy. Furthermore, the system compares these two calculated metrics with a preset 80% threshold. If both metrics are greater than or equal to the preset threshold, the regular expression is considered valid; if either metric fails to reach the threshold, the validation is deemed unsuccessful, and the uncovered samples are filtered out, triggering subsequent iterative optimization processes.

[0073] Finally, for valid regular expressions, the system stores them along with their corresponding rule detail ID, rule direction, generation time, and validation metrics in the `ai_extract_rule` database table, completing the persistent storage of the regular expressions and ensuring that they can still be called normally after a system restart. At the same time, the pre-compiled `Pattern` object and the regular expression string are stored in a distributed cache, with a corresponding cache expiration time set. This facilitates the subsequent full feature extraction process and related tasks with the same rules, allowing for direct and fast loading and calling without repeated generation and validation.

[0074] Based on this, embodiments of the present invention can further process regular expressions that fail the verification in the aforementioned steps as follows: 7.1: Generate new standardized prompt words by using the uncovered sample set composed of samples whose coverage or matching accuracy does not meet the preset threshold, and the preset system constraint instructions; This step filters out samples that do not meet the threshold requirements for regular expressions that fail validation, forming an uncovered sample set. The uncovered sample set includes samples for which no values ​​were extracted, as well as samples whose extraction results are inconsistent with the target fragment. The uncovered sample set and the preset system constraint instructions are then encapsulated into a new standardized prompt word. The prompt word clearly informs the large language model that the preceding regular expressions do not cover these samples and that supplementary regular expressions need to be generated.

[0075] Here, new prompt words are generated only for uncovered samples, allowing the large language model to accurately focus on the deficiencies of the preorder regularization and generate supplementary regularizations accordingly. This avoids repeatedly optimizing covered samples, greatly improving the efficiency and accuracy of iterative optimization and ensuring that the iteration direction fully aligns with business needs.

[0076] 7.2: Recall the preset large language model, take the new standardized prompt words as input, generate supplementary regular expressions, and merge them with the regular expressions to obtain multiple regular expression combinations; This step resends the new standardized prompts to the privately deployed large language model, generating supplementary regular expressions for the uncovered samples. The supplementary regular expressions are then combined with the original regular expressions using the "|" vertical bar symbol in Java regular expressions, forming a multi-regular expression combination. Matching and extraction can be completed as long as any one of the regular expressions is satisfied.

[0077] Here, multiple regular expressions are merged using OR relations. A single combination of multiple regular expressions can cover multiple text formats under the same field, which completely solves the pain points of existing technologies, such as the difficulty of single rule covering multiple formats, the need for manual splitting of multiple regular expressions, and the linear increase in maintenance workload. It can achieve full coverage of multiple formats without manual intervention, and greatly reduces the maintenance cost of rules.

[0078] 7.3: Persist and cache multiple regular expression combinations that have passed verification within a preset number of iterations.

[0079] Here, a maximum of three iterations can be preset. For the merged multiple regular expression combinations, pre-compilation and dual-metric verification are re-executed. If the verification passes within the preset number of iterations, the multiple regular expression combination is persisted and cached. If the verification still fails after reaching the maximum number of iterations, fallback logic is triggered. By setting an iteration limit, the resource consumption of the large language model and the risk of task timeout caused by infinite iteration are avoided. This balances the regular expression coverage effect with system resource consumption. At the same time, iterative verification ensures that the final multiple regular expression combination meets the business extraction requirements, guaranteeing the extraction effect.

[0080] In practical applications, firstly, when regular expression validation fails, the system filters out samples that do not meet the threshold requirements from the standardized sample pair list, forming an uncovered sample set. This sample set includes two types of samples: one type consists of samples where the pre-compiled regular expression did not extract non-empty values, directly affecting the extraction coverage metric; the other type consists of samples where values ​​were extracted, but the extraction results are not completely consistent with the target fragment, directly affecting the matching accuracy metric. Then, the system encapsulates the uncovered sample set and preset system constraint instructions into a new standardized prompt word. The prompt word explicitly informs the large language model that the previously generated regular expression failed to cover these samples, and that it is necessary to analyze the characteristics of the samples and generate supplementary Java regular expressions that can accurately cover these samples. Furthermore, the supplementary regular expression must be combined with the previous regular expression using an OR relationship.

[0081] Secondly, the system, through the AiClient module, resends the newly encapsulated standardized prompt words to the privately deployed large language model. It receives the supplementary regular expression candidate results returned by the model, performs non-compliant content filtering on these candidate results, and obtains the pure regular expression, i.e., the supplementary regular expression. Then, the system merges the generated supplementary regular expression with the original regular expression that failed the previous validation using the "|" vertical bar symbol in Java regular expression syntax, forming a new multi-regular expression combination. This multi-regular expression combination can complete feature extraction as long as it satisfies the matching rule of any one of its sub-regular expressions, and can simultaneously cover text features of various formats.

[0082] Then, the system re-executes the pre-compilation and dual-index verification process for the merged multi-regular expression combination. First, it performs pre-compilation on the multi-regular expression combination to generate a reusable pre-compiled object. Then, based on the pre-compiled object, it performs parallel sample extraction operations on the original full-scale standardized sample pairs, recalculates the extraction coverage and matching accuracy, and determines whether it meets the preset threshold. In addition, during the iteration process, the system counts the current iteration number in real time to ensure that the number of iterations does not exceed the preset upper limit of 3, avoiding resource consumption and task timeouts caused by infinite iteration.

[0083] Finally, if the combination of multiple regular expressions meets the preset thresholds for both metrics within the preset number of iterations, the verification is deemed successful. The system stores the combination of multiple regular expressions along with the corresponding rule detail ID, iteration count, verification metrics, and other information into the ai_extract_rule table for persistence, and also stores it in the distributed cache for easy subsequent calls. If the combination of multiple regular expressions still fails to meet the thresholds for both metrics after reaching the preset limit of 3 iterations, the iteration stops, triggering fallback logic to ensure that the associated tasks are not interrupted.

[0084] S108: Extract feature values ​​from heterogeneous archive data using cached regular expressions and generate a feature value dataset with rule type identifiers.

[0085] Here, based on cached pre-compiled regular expressions, batch feature value extraction is performed on the full heterogeneous archive data. The extracted feature values ​​are then bound to rule type identifiers, ultimately forming a standardized feature value dataset. The rule type identifier (containsFlag in this invention) is a standardized Boolean identifier bound to the feature value to distinguish rule types, used for differentiated logic in subsequent matching processes. The feature value dataset is a standardized set of multiple identified feature value records, forming the core data foundation for subsequent data association matching. Performing full feature extraction based on pre-compiled cached regular expressions significantly improves the feature extraction efficiency for tens of millions of heterogeneous archive data. Simultaneously, by pre-binding rule type identifiers, a direct basis is provided for differentiated processing in subsequent matching processes, eliminating the need for repeated rule type parsing during matching, further improving the overall performance of association matching. Furthermore, the standardized feature value dataset ensures the consistency of subsequent association matching logic.

[0086] For example, step S108 can be refined through the following steps, including: 8.1: Prioritize loading cached regular expressions and extract feature values ​​from the preset associated fields corresponding to heterogeneous archive data using regular expression matching; when the regular expression is invalid, the extraction result is empty, or the extraction is abnormal, it is determined that the feature value extraction has failed, and the original field value of the preset associated field is directly obtained; This step is the full feature extraction stage. It prioritizes loading pre-compiled regular expressions from the cache and performs batch feature extraction on the target related fields of the full heterogeneous archive data. A complete exception handling mechanism is also in place. If an exception occurs, such as an invalid regular expression, an empty extraction result, an AI call error, or a regular expression matching timeout, the extraction is directly judged as a failure, and a fallback logic is automatically triggered to directly obtain the original full field values ​​of the corresponding related fields. By prioritizing the loading of cached pre-compiled regular expressions, the feature extraction efficiency of tens of millions of data points is significantly improved. At the same time, the complete exception fallback mechanism ensures that valid feature values ​​are obtained regardless of whether the AI ​​regular expression is effective, completely avoiding interruptions to the association task due to AI extraction failure. This achieves 24 / 7 uninterrupted high availability operation, fully meeting the stability requirements of enterprise-level production environments.

[0087] 8.2: Regularize the feature values ​​extracted by regular expressions or the original field values ​​obtained from fallback, generate a feature value dataset with rule type identifiers, and synchronously record abnormal alarm information to ensure the continuous execution of related tasks.

[0088] This process standardizes and normalizes the extracted feature values, unifying the format, removing interfering characters, and binding a rule type identifier (contains Flag=true) to each feature value. This results in a standardized feature value dataset with rule type identifiers. Simultaneously, for extraction failures, anomaly alerts are recorded, including the anomaly type, anomaly data ID, and rule detail ID, facilitating subsequent auditing and troubleshooting. By pre-binding a rule type identifier to each feature value, a direct basis is provided for differentiated processing in subsequent matching stages, eliminating the need for repeated rule type parsing during matching and significantly improving the efficiency of association matching. Standardization ensures the uniformity of feature value formats, avoiding matching errors caused by format differences. Furthermore, the anomaly alert recording meets the auditability requirements of government and financial scenarios.

[0089] In practical applications, S108 can also be achieved through the following steps, including: First, when performing full feature extraction, the system prioritizes loading pre-compiled regular expressions for the corresponding rules from the distributed cache. If the corresponding data is not found in the cache, it reads the persistent regular expression from the `ai_extract_rule` database table, recompiles it, and loads it for use, ensuring that valid regular expression objects can be quickly obtained during the extraction process. Then, based on the loaded pre-compiled regular expressions, the system uses a batch processing framework to perform parallel regular expression matching feature extraction operations on the preset associated fields corresponding to the full set of heterogeneous archive data. The batch size can be adaptively adjusted according to server performance and data volume to maximize extraction efficiency. Furthermore, the system monitors the extraction status in real time. If situations such as regular expression compilation failure, regular expression matching timeout, empty full data extraction results, or system abnormalities occur, it immediately determines that feature value extraction has failed and automatically triggers fallback logic, directly reading the original full field values ​​of the corresponding associated fields as feature values ​​to ensure that the extraction process is not interrupted.

[0090] Then, the system performs standardization processing on the feature values ​​extracted by regular expressions, or the original field values ​​obtained as a fallback, removing leading and trailing spaces, newlines, and special interference characters, and unifying full-width and half-width characters and encoding formats to ensure the standardization and consistency of feature value formats. Furthermore, the system binds corresponding rule detail IDs, associated field names, and unique data IDs to each processed feature value record, and marks each record with the rule type identifier `containsFlag=true`, clearly indicating that the rule type corresponding to this feature value is a containment rule, ultimately forming a complete feature value dataset with rule type identifiers.

[0091] Finally, the system synchronously records detailed anomaly alarm information for all abnormal situations that occur during the feature value extraction process, including the anomaly type, anomaly occurrence time, corresponding rule detail ID, unique ID of the abnormal data, and anomaly details, and stores this information in the system alarm log for subsequent troubleshooting, auditing, and maintenance. Furthermore, the system transmits the generated feature value dataset to the subsequent association matching stage to ensure the continuous and stable execution of the association task, preventing interruptions due to anomalies in the extraction stage.

[0092] In a preferred embodiment of practical application, the system first receives a trigger instruction for the associated task and initiates the task. This trigger instruction can come from manual user operation, a pre-set scheduled task, or a data update event from an upstream business system. After the task is initiated, the system reads the pre-set full associated configuration details through the configuration loading module to ensure the complete loading of all associated rules. The configuration details include key information such as task ID, rule detail ID, rule type, left and right associated directories, associated fields, and priority level. Then, the system starts the rule traversal engine, traversing each loaded rule detail one by one according to the task dimension, establishing a rule processing queue to avoid rule omissions or duplicate processing. At the same time, the specific type of each rule is specified through the Rule Type field in the Association Rule Detail object, completing the classification and identification of rules. In addition, the system performs multiple checks simultaneously during the rule loading and recognition process, including checking the completeness of the rule configuration to ensure that no required fields such as associated directories and associated fields are missing; checking the legality of the rule type identifier to allow only the two rule types of equality and inclusion; checking the reachability of the left and right associated directories to ensure that subsequent processes can obtain data normally, and logging alarms for rules that fail the checks and skipping the processing.

[0093] Secondly, for the identified inclusion rules, the system marks their processing path as an AI-assisted branch and generates a context object for storing rule parameters to avoid interference with the processing flow of other rules. Based on the rule parameters stored in the context object, such as rule direction, left and right related directories, and target related fields, the system accurately locates the target data directory corresponding to the heterogeneous archive data, limiting the data source range for sample extraction. Then, from the located target data directory, the system performs targeted random sampling of real business data under the target related fields according to a preset sampling quantity of 50. During the sampling process, null values, garbled characters, and invalid text data are automatically removed. At the same time, the sampled data is de-identified and normalized to filter sensitive information and ensure data security. In addition, the system splits each normalized sampled data into the source text to be extracted and the expected target fragment, constructing a standardized sample pair with a unified format. This sample pair can fully represent the format characteristics and extraction requirements of the current business data.

[0094] Then, the system encapsulates the preset system constraint instructions and the generated standardized sample pairs into standardized prompt words. The system constraint instructions explicitly require the large language model, as a regular expression expert, to analyze the characteristic patterns of the samples, generate regular expressions adapted to the Java runtime environment, and output only the expression itself without any additional explanation. Subsequently, the system sends the encapsulated prompt words to the pre-deployed private large language model through a dedicated intranet Ai Client module, receives the regular expression candidate results returned by the model, and filters the candidate results, removing non-compliant content such as Chinese characters and Markdown format, to obtain pure regular expressions adapted to the preset runtime environment. In addition, for complex scenarios, the system can first perform text structure complexity, feature dispersion, and extraction constraint strength analysis on the standardized sample pairs to quantify the strength score of the current feature extraction requirement. Based on the score, the system dynamically selects the target model from multiple preset large language models adapted to different complexity scenarios, or assigns adaptive inference weights to each model, and generates more accurate regular expressions through multi-model fusion inference.

[0095] In addition, the system performs pre-compilation processing on the generated regular expressions. Using the Pattern.compile() method in the Java environment, the regular expression string is converted into a reusable Pattern pre-compiled object. After pre-compilation, the system performs parallel sample extraction operations on the source text in the standardized sample pairs based on the pre-compiled object. Based on the extraction results and the target segments in the sample pairs, the system calculates the extraction coverage and matching accuracy, respectively. The extraction coverage is the proportion of samples with non-empty values ​​extracted out of the total number of samples, and the matching accuracy is the proportion of samples whose extraction results completely match the target segments out of the total number of samples. Then, the system determines whether both indicators meet the preset 80% threshold. If both are met, the regular expression is deemed valid and stored in the ai_extract_rule table for persistence, and also stored in the distributed cache for quick retrieval during subsequent full extraction. If the threshold is not met, the uncovered samples are selected to form an uncovered sample set, new standardized prompt words are generated, and the large language model is called again to generate supplementary regular expressions. The supplementary regular expressions are then merged with the original regular expressions using an OR relationship to form a multi-regular expression combination. The validation is re-executed within the preset 3 iterations until the validation passes or the iteration limit is reached.

[0096] Finally, the system loads the validated pre-compiled regular expressions from the cache and performs batch feature value extraction on the preset associated fields corresponding to the full set of heterogeneous archive data. If an invalid regular expression, empty extraction result, or extraction anomaly occurs, the system determines that the feature value extraction has failed and automatically triggers fallback logic to directly obtain the original field value of the corresponding associated field, ensuring that the association task is not interrupted. Then, the system normalizes the feature values ​​obtained from the regular expression extraction or the original field values ​​obtained from the fallback, binds a rule type identifier to each feature value, and generates a standardized feature value dataset with rule type identifiers. At the same time, it records anomaly alarm information for extraction anomalies to facilitate subsequent auditing and problem investigation.

[0097] In some embodiments, such as Figure 2 As shown, this embodiment of the invention also provides a feature value extraction device for heterogeneous archival data, comprising: The parallel module, in response to the association task of heterogeneous archive data initiated under preset conditions, loads the full set of association rule details and selects the corresponding branch for parallel processing according to the type of each rule; wherein, the full set of association rule details includes inclusion class rules; The calling module, for the standardized sample pairs extracted from the target data directory of the heterogeneous archive data, which contain rule parameters and processing branches including class rules, calls the preset large language model to output a regular expression adapted to the preset running environment; wherein, the standardized sample pairs are used to characterize the current data format and extraction requirements; The validation module pre-compiles and performs compliance validation on the regular expressions, and caches the validated regular expressions. The generation module extracts feature values ​​from the heterogeneous archive data based on cached regular expressions, and generates a feature value dataset with rule type identifiers.

[0098] The present invention provides an embodiment of an electronic device. In this embodiment, the electronic device may be, but is not limited to, a personal computer (PC), a laptop computer, a monitoring device, a server, or other computer device with analysis and processing capabilities.

[0099] As an exemplary embodiment, see [link to example]. Figure 3 The electronic device 110 includes a communication interface 111, a processor 112, a memory 113, and a bus 114. The processor 112, the communication interface 111, and the memory 113 are connected via the bus 114. The memory 113 is used to store a computer program that supports the processor 112 in executing the above-described method. The processor 112 is configured to execute the program stored in the memory 113.

[0100] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0101] Non-volatile media can be non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar non-volatile storage media, or combinations thereof.

[0102] It is understood that the specific operation methods of each functional module in this embodiment can be referred to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0103] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program. When the computer program code is executed, it can implement the method described in any of the above embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0105] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0106] In the description of this invention, it should be noted that the terms center, up, down, left, right, vertical, horizontal, inner, and outer, indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms first, second, and third are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0107] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A method for extracting feature values ​​from heterogeneous archival data, characterized in that, include: In response to a task involving the association of heterogeneous archive data initiated under preset conditions, the full set of association rule details is loaded, and the corresponding branch is selected for parallel processing based on the type of each rule; wherein, the full set of association rule details includes inclusion class rules; For the standardized sample pairs extracted from the target data directory of the heterogeneous archive data, which contain rule parameters and processing branches including class rules, a preset large language model is invoked to output a regular expression adapted to the preset running environment; wherein, the standardized sample pairs are used to characterize the current data format and extraction requirements; The regular expression is pre-compiled and validated for compliance, and the validated regular expressions are cached. Based on cached regular expressions, feature values ​​of the heterogeneous archive data are extracted to generate a feature value dataset with rule type identifiers.

2. The method according to claim 1, characterized in that, In response to a heterogeneous archive data association task initiated based on preset conditions, the steps include loading the full association rule details and selecting the appropriate branch for parallel processing based on the type of each rule, including: In response to user operation commands, system timed scheduling, or upstream data update events, trigger the initiation of associated tasks for heterogeneous archive data; Load and simultaneously traverse the full list of associated rules in parallel, and identify the rule type field of each rule to determine its type; For equality rules, the message digest branch is used directly to generate feature values. At the same time, for inclusion rules, a context object is generated to store rule parameters, triggering AI-assisted branch parallel processing.

3. The method according to claim 1, characterized in that, The steps of calling a preset large language model to output a regular expression adapted to a preset operating environment for standardized sample pairs extracted from the target data directory of the heterogeneous archive data, including rule parameters and processing branches containing class rules, include: For the rule parameters and processing branches containing class rules, locate the target data directory of the heterogeneous archive data and extract standardized sample pairs from the target data directory; The standardized sample pairs are analyzed for text structure complexity, feature dispersion, and extraction constraint strength to quantify the strength score of the current feature extraction requirement. Based on the extraction demand strength score, a target large language model is dynamically selected from multiple preset large language models adapted to different complexity scenarios, or an adaptive inference weight corresponding to the extraction demand strength is assigned to each preset large language model. The preset system constraint instructions and the standardized sample pairs are encapsulated into standardized prompt words; The standardized prompt words are input into a pre-deployed target large language model, and a regular expression adapted to the preset running environment is output.

4. The method according to claim 3, characterized in that, The steps of locating the target data directory of the heterogeneous archive data and extracting standardized sample pairs from the target data directory, based on rule parameters and processing branches containing class rules, include: Based on the AI-assisted branch to which the inclusion class rule belongs, the rule parameters in the inclusion class rule, including the pre-configured rule direction, left and right related directories and target related fields, are read to locate the target data directory corresponding to the heterogeneous archive data; From the located target data directory, perform targeted random sampling of the real business data under the target associated field according to the preset sampling quantity, and automatically remove null values, garbled characters and invalid text data, while de-identifying and normalizing the sampled data; Each data point after normalization is split into the source text to be extracted and the expected target fragment, thus constructing a standardized sample pair with a unified format to represent the extraction characteristics of the current business data.

5. The method according to claim 3, characterized in that, The steps of inputting the standardized prompt words into a pre-deployed target large language model and outputting a regular expression adapted to the preset operating environment include: The target large language model filters out non-compliant format content from the regular expression candidate results output based on the standardized prompt words to obtain a regular expression adapted to the preset running environment. or, The standardized prompt words are input into the target large language model composed of a multi-model fusion inference link after weight allocation, and multi-way regularization candidate results are obtained. Based on the standardized sample pairs, the multiple regular expression candidate results are verified in parallel and fused to generate a target regular expression that is adapted to the preset operating environment.

6. The method according to claim 1, characterized in that, The steps of precompiling and validating the regular expression, and caching the validated regular expressions, include: The regular expression is pre-compiled to generate a reusable pre-compiled object; Based on the pre-compiled object, sample extraction operations are performed in parallel on the source text of the standardized sample pairs; The extraction coverage and matching accuracy are calculated based on the target segments in the extracted samples and the standardized sample pairs, respectively. If both the extraction coverage and the matching accuracy meet the preset thresholds, the regular expression is deemed to be valid and is persisted and cached.

7. The method according to claim 6, characterized in that, Also includes: New standardized prompt words are generated using the uncovered sample set consisting of samples whose extraction coverage or matching accuracy does not meet a preset threshold, and preset system constraint instructions. The preset large language model is invoked again, and the new standardized prompt word is used as input to generate a supplementary regular expression. This supplementary regular expression is then merged with the regular expression to obtain a multi-regular expression combination. The multiple regular expression combinations that pass verification within a preset number of iterations will be persisted and cached.

8. The method according to claim 1, characterized in that, The steps of extracting feature values ​​from the heterogeneous archive data based on cached regular expressions and generating a feature value dataset with rule type identifiers include: The cached regular expressions are loaded first, and feature values ​​are extracted from the preset associated fields corresponding to the heterogeneous archive data by regular expression matching. When the regular expression is invalid, the extraction result is empty, or the extraction is abnormal, it is determined that the feature value extraction has failed, and the original field value of the preset associated field is directly obtained. The feature values ​​extracted by regular expressions or the original field values ​​are normalized to generate a feature value dataset with rule type identifiers. Anomaly alarm information is recorded synchronously to ensure the continuous execution of related tasks.

9. A feature value extraction device for heterogeneous archival data, characterized in that, include: The parallel module, in response to the association task of heterogeneous archive data initiated under preset conditions, loads the full set of association rule details and selects the corresponding branch for parallel processing according to the type of each rule; wherein, the full set of association rule details includes inclusion class rules; The calling module, for the standardized sample pairs extracted from the target data directory of the heterogeneous archive data, which contain rule parameters and processing branches including class rules, calls the preset large language model to output a regular expression adapted to the preset running environment; wherein, the standardized sample pairs are used to characterize the current data format and extraction requirements; The validation module pre-compiles and performs compliance validation on the regular expressions, and caches the validated regular expressions. The generation module extracts feature values ​​from the heterogeneous archive data based on cached regular expressions, and generates a feature value dataset with rule type identifiers.

10. An electronic device, characterized in that, It includes a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the method as described in any one of claims 1 to 7.

Images