Automatic Report Generation Method Based on Business Rules and Regular Expression Matching

Abstract

This invention provides an automatic report generation method based on business rules and regular expression matching, belonging to the field of data processing and report generation technology. The method includes: S1. Report requirement parsing; S2. Business rule definition; S3. Regular expression matching rule configuration; S4. Raw data acquisition; S5. Raw data preprocessing; S6. Business rule execution; S7. Regular expression matching validation and field extraction; S8. Report template generation; S9. Report data filling and generation; S10. Report validation and feedback optimization. This invention achieves accurate, automated, and flexible report generation through multi-stage collaboration and multi-algorithm integration.

Description

Technical Field

[0001] This invention relates to the field of data processing and report generation technology, specifically to a method for automatically generating reports based on business rules and regular expression matching. Background Technology

[0002] Existing automated report generation technologies generally suffer from insufficient data processing accuracy, poor scenario adaptability, and weak process continuity. Traditional methods often employ single algorithms for data classification or anomaly detection, failing to consider the combined impact of multi-dimensional features. This leads to classification results deviating from actual business needs or resulting in missed or false positives for anomalies. Business rules are often fixed configurations, unable to dynamically adjust their execution intensity based on data quality and distribution characteristics. When faced with diverse data scenarios, rule rigidity easily arises, either leading to over-filtering and loss of valid data or lenient filtering allowing anomalies to enter the report. Furthermore, there is a lack of effective collaboration between algorithms, business rules, and various data processing stages. Data transfer between preprocessing, classification, and validation steps lacks feedback mechanisms, creating processing gaps. In addition, the extraction of unstructured data and the processing of structured data are often independent, making comprehensive data integration and utilization difficult. Ultimately, this affects the accuracy and automation level of report generation, requiring significant manual intervention for adjustment and optimization. Summary of the Invention

[0003] This invention provides an automatic report generation method based on business rules and regular expression matching, which achieves more accurate, automated and flexible report generation through multi-stage collaboration and multi-algorithm integration.

[0004] Automatic report generation methods based on business rules and regular expression matching include:

[0005] S1. Report Requirements Analysis: Clarify the core objectives, data range, field requirements, output format, and validation standards for report generation, and output the report requirements specification;

[0006] S2. Business Rule Definition: Based on the report requirements specification, sort out and define data filtering rules, data calculation rules, data classification rules, and data validation rules, and output a list of business rules;

[0007] S3. Regular expression matching rule configuration: Based on the field format specifications and business rule list in the report requirements specification, configure the field extraction regular expression rules, format validation regular expression rules, and data cleaning regular expression rules, and output the regular expression matching rule configuration table;

[0008] S4. Raw Data Acquisition: Based on the data range specified in the report requirements specification, raw data is collected from three preset data sources: database, local file, and third-party API interface, and the raw data set is output.

[0009] S5. Raw Data Preprocessing: Perform data deduplication, missing value handling, data type conversion, and format standardization on the raw dataset to output a clean dataset;

[0010] S6. Business Rule Execution: Call the rule engine, take the clean dataset as input, and integrate the feature importance weighting algorithm based on random forest, the improved K-means clustering algorithm, the gradient boosting decision tree anomaly detection algorithm, and the collaborative filtering rule adaptation algorithm. Through the two-way bidirectional interaction of the four algorithms, a closed-loop processing mechanism is formed to complete data filtering, calculation, classification and preliminary verification, and output the preliminary processed data set.

[0011] S7. Regular expression matching validation and field extraction: Based on the regular expression matching rule configuration table, the field format validation, unstructured field extraction and secondary data cleaning are performed on the preliminary processed data set. Data with abnormal format is fed back to S5 for reprocessing, and the final valid data set and the list of data with abnormal format are output.

[0012] S8. Report Template Generation: Based on the output format requirements of the report requirements specification, create a standardized report template that includes field layout, style specifications, and chart configuration, and output the standardized report template;

[0013] S9. Report Data Population and Generation: Populate the final valid data set into the standardized report template according to the field correspondence, complete the filling of table data, derived fields and statistical data, and the binding of chart data, convert it into an initial report file in the specified format, and output the initial report file;

[0014] S10. Report Validation and Feedback Optimization: Perform completeness, accuracy, and format standardization checks on the initial report file. If the check passes, the final report file is output. If it fails, feedback is sent to the corresponding step for optimization, and the subsequent process is re-executed.

[0015] In this specification, the feature importance weighting algorithm based on random forest in S6 includes: constructing a feature set containing transaction amount, payment frequency, commodity category code, and payment method code; calculating the feature split gain through Gini coefficients and obtaining the feature importance weights after normalization; calculating the feature weighted sum of clean data; using this feature weighted sum as the input parameter of the improved K-means clustering algorithm; and dynamically adjusting the feature importance weights by receiving the rule fit feedback from the collaborative filtering rule fit algorithm.

[0016] In this specification, the improved K-means clustering algorithm in S6 includes: determining the number of clusters based on the classification requirements of the report requirements specification; calculating the initial cluster centers using the feature importance weights output by the feature importance weighting algorithm based on random forest; using weighted Euclidean distance to measure the similarity between the data and the cluster centers; iteratively updating the cluster centers until they stabilize and then outputting the cluster labels. These cluster labels serve as input features for the anomaly detection algorithm of the gradient boosting decision tree. At the same time, the similarity within the clusters is calculated and fed back to the feature importance weighting algorithm based on random forest to adjust the feature weights.

[0017] In this specification, the anomaly detection algorithm of gradient boosting decision tree in S6 includes: constructing an input feature set containing a feature set based on a random forest-based feature importance weighting algorithm and clustering labels output by an improved K-means clustering algorithm; learning residual correction rules through iterative training of multiple regression decision trees; obtaining anomaly scores and labeling anomaly data through sigmoid function transformation; outputting anomaly labels and anomaly scores; and feeding the anomaly scores back to the improved K-means clustering algorithm and the collaborative filtering rule adaptation algorithm for parameter adjustment.

[0018] In this specification, the rule adaptation algorithm for collaborative filtering in S6 includes: taking the weighted sum of features output by the feature importance weighting algorithm based on random forest, the intra-cluster similarity output by the improved K-means clustering algorithm, and the anomaly score output by the gradient boosting decision tree anomaly detection algorithm as input features; calculating the similarity between the data and the business rules using the Pearson correlation coefficient and predicting the rule fit; calculating the fit adjustment coefficient based on the anomaly score to dynamically adjust the business rule execution threshold; and feeding the rule fit back to the feature importance weighting algorithm based on random forest to adjust the feature weights.

[0019] In this specification, the bidirectional interaction mechanism of the four algorithms in S6 is as follows: the feature importance weighting algorithm based on random forest and the improved K-means clustering algorithm provide feedback to each other through feature weights and intra-cluster similarity; the improved K-means clustering algorithm and the gradient boosting decision tree anomaly detection algorithm provide feedback to each other through cluster labels and anomaly scores; the gradient boosting decision tree anomaly detection algorithm and the collaborative filtering rule adaptation algorithm provide feedback to each other through anomaly scores and adaptation adjustment coefficients; and the collaborative filtering rule adaptation algorithm and the feature importance weighting algorithm based on random forest provide feedback to each other through rule fit and feature weighted sum.

[0020] In this manual, the field extraction regular expression rules in S3 are used to extract target fields from unstructured text, the format validation regular expression rules are used to verify the format standardization of fields, and the data cleaning regular expression rules are used to remove invalid characters.

[0021] In this specification, the missing value handling in S5 uses either direct removal or filling with default values ​​allowed by the business for the required fields defined in the report requirements specification. When optional fields are missing, preset default values ​​are used to fill them, ensuring the integrity of the clean data set to support subsequent algorithm processing.

[0022] In this specification, S10 includes an integrity check to verify whether the report contains all required fields, derived fields, and statistical data as required by the report requirements specification; an accuracy check to verify the consistency between the report data and the final valid data set; and a format compliance check to verify whether the report file format, style, and field order conform to the requirements of the report requirements specification.

[0023] In this manual, the feedback optimization path for S10 is as follows: if the completeness is missing, feedback is sent to S6 to supplement the calculation of derived fields; if the accuracy is incorrect, feedback is sent to S7 to re-execute the regular expression matching validation; if the format is not standardized, feedback is sent to S8 to adjust the report template style. After the corresponding steps are optimized, S9 and S10 are re-executed.

[0024] The embodiments described in this specification can achieve at least the following beneficial effects:

[0025] Improve the accuracy of data processing: By integrating and interacting multiple algorithms, and making full use of the synergistic effects of feature weighting, clustering classification, anomaly detection and rule adaptation, the limitations of single algorithm processing can be effectively solved, making data classification more in line with business dimensions and anomaly data identification more accurate.

[0026] Enhance the flexibility of scenario adaptation: Dynamically adjust the execution intensity of business rules based on data characteristics and abnormal situations, break the rigid limitations of fixed rules, and adapt to diverse business scenarios with different data distributions and quality differences.

[0027] Ensure process continuity and reliability: Construct a closed-loop interactive mechanism of "feature extraction - clustering classification - anomaly detection - rule adaptation" to achieve seamless connection of each link in data processing and reduce data loss or error transmission caused by process breaks.

[0028] Capable of processing multiple types of data: Integrating regular expression matching technology and intelligent algorithms, it can accurately extract and standardize the format of unstructured data, and also complete in-depth processing of structured data, fully covering the data source requirements for report generation.

[0029] Enhance automation and intelligence: Reduce reliance on manual intervention in rule adjustment, data verification, and other processes. Automate the entire report generation process through algorithm collaboration and dynamic feedback, thereby reducing labor costs and improving the stability of report output. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of an automatic report generation method based on business rules and regular expression matching, as described in some embodiments of the present invention.

[0032] Figure 2 This is a schematic diagram of the automatic report generation process involved in some embodiments of the present invention.

[0033] Figure 3 This is a schematic diagram of the collaborative interaction process of the core algorithm involved in some embodiments of the present invention.

[0034] Figure 4 This is a schematic diagram of the abnormal data processing and feedback process involved in some embodiments of the present invention. Detailed Implementation

[0035] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0036] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0037] like Figure 1 As shown, this embodiment provides a method for automatically generating reports based on business rules and regular expression matching, including:

[0038] S1. Report Requirements Analysis: Clarify the core objectives, data range, field requirements, output format, and validation standards for report generation, and output the report requirements specification;

[0039] S2. Business Rule Definition: Based on the report requirements specification, sort out and define data filtering rules, data calculation rules, data classification rules, and data validation rules, and output a list of business rules;

[0040] S3. Regular expression matching rule configuration: Based on the field format specifications and business rule list in the report requirements specification, configure the field extraction regular expression rules, format validation regular expression rules, and data cleaning regular expression rules, and output the regular expression matching rule configuration table;

[0041] S4. Raw Data Acquisition: Based on the data range specified in the report requirements specification, raw data is collected from three preset data sources: database, local file, and third-party API interface, and the raw data set is output.

[0042] S5. Raw Data Preprocessing: Perform data deduplication, missing value handling, data type conversion, and format standardization on the raw dataset to output a clean dataset;

[0043] S6. Business Rule Execution: Call the rule engine, take the clean dataset as input, and integrate the feature importance weighting algorithm based on random forest, the improved K-means clustering algorithm, the gradient boosting decision tree anomaly detection algorithm, and the collaborative filtering rule adaptation algorithm. Through the two-way bidirectional interaction of the four algorithms, a closed-loop processing mechanism is formed to complete data filtering, calculation, classification and preliminary verification, and output the preliminary processed data set.

[0044] S7. Regular expression matching validation and field extraction: Based on the regular expression matching rule configuration table, the field format validation, unstructured field extraction and secondary data cleaning are performed on the preliminary processed data set. Data with abnormal format is fed back to S5 for reprocessing, and the final valid data set and the list of data with abnormal format are output.

[0045] S8. Report Template Generation: Based on the output format requirements of the report requirements specification, create a standardized report template that includes field layout, style specifications, and chart configuration, and output the standardized report template;

[0046] S9. Report Data Population and Generation: Populate the final valid data set into the standardized report template according to the field correspondence, complete the filling of table data, derived fields and statistical data, and the binding of chart data, convert it into an initial report file in the specified format, and output the initial report file;

[0047] S10. Report Validation and Feedback Optimization: Perform completeness, accuracy, and format standardization checks on the initial report file. If the check passes, the final report file is output. If it fails, feedback is sent to the corresponding step for optimization, and the subsequent process is re-executed.

[0048] In some embodiments, the feature importance weighting algorithm based on random forest in S6 includes: constructing a feature set containing transaction amount, payment frequency, commodity category code, and payment method code; calculating the feature split gain through Gini coefficients and obtaining the feature importance weights after normalization; calculating the feature weighted sum of clean data, which is used as the input parameter of the improved K-means clustering algorithm; and dynamically adjusting the feature importance weights by receiving the rule fit feedback from the collaborative filtering rule fit algorithm.

[0049] In some embodiments, the improved K-means clustering algorithm in S6 includes: determining the number of clusters based on the classification requirements of the report requirements specification; calculating initial cluster centers using the feature importance weights output by the feature importance weighting algorithm based on random forest; measuring the similarity between the data and the cluster centers using weighted Euclidean distance; iteratively updating the cluster centers until they stabilize and then outputting cluster labels. These cluster labels serve as input features for the anomaly detection algorithm of the gradient boosting decision tree. Simultaneously, the similarity within the clusters is calculated and fed back to the feature importance weighting algorithm based on random forest to adjust the feature weights.

[0050] In some embodiments, the anomaly detection algorithm of gradient boosting decision tree in S6 includes: constructing an input feature set containing a feature set based on a feature importance weighting algorithm of random forest and clustering labels output by an improved K-means clustering algorithm; learning residual correction rules through iterative training of multiple regression decision trees; obtaining anomaly scores and labeling abnormal data through sigmoid function transformation; outputting anomaly labels and anomaly scores; and feeding the anomaly scores back to the improved K-means clustering algorithm and the collaborative filtering rule adaptation algorithm for parameter adjustment.

[0051] In some embodiments, the rule adaptation algorithm for collaborative filtering in S6 includes: taking the feature weighted sum output by the feature importance weighting algorithm based on random forest, the cluster intra-cluster similarity output by the improved K-means clustering algorithm, and the anomaly score output by the gradient boosting decision tree anomaly detection algorithm as input features; calculating the similarity between the data and the business rules using the Pearson correlation coefficient and predicting the rule fit; calculating the fit adjustment coefficient based on the anomaly score to dynamically adjust the business rule execution threshold; and feeding the rule fit back to the feature importance weighting algorithm based on random forest to adjust the feature weights.

[0052] In some embodiments, the bidirectional interaction mechanism of the four algorithms in S6 is as follows: the feature importance weighting algorithm based on random forest and the improved K-means clustering algorithm feed back to each other through feature weights and intra-cluster similarity; the improved K-means clustering algorithm and the gradient boosting decision tree anomaly detection algorithm feed back to each other through cluster labels and anomaly scores; the gradient boosting decision tree anomaly detection algorithm and the collaborative filtering rule adaptation algorithm feed back to each other through anomaly scores and adaptation adjustment coefficients; and the collaborative filtering rule adaptation algorithm and the feature importance weighting algorithm based on random forest feed back to each other through rule fit and feature weighting sum.

[0053] In some embodiments, the field extraction regular expression rules in S3 are used to extract target fields from unstructured text, the format validation regular expression rules are used to verify the format standardization of fields, and the data cleaning regular expression rules are used to remove invalid characters.

[0054] In some embodiments, the missing value handling in S5 involves directly removing or filling with default values ​​that are allowed by the business for the required fields defined in the report requirements specification. When optional fields are missing, preset default values ​​are used to fill them, ensuring the integrity of the clean data set to support subsequent algorithm processing.

[0055] In some embodiments, S10 includes an integrity check to see if the report contains all required fields, derived fields, and statistical data as required by the report requirements specification; an accuracy check to verify the consistency between the report data and the final valid data set; and a format conformity check to see if the report file format, style, and field order conform to the requirements of the report requirements specification.

[0056] In some embodiments, the feedback optimization path of S10 is as follows: if the integrity is missing, feedback is sent to S6 to supplement the calculation of derived fields; if the accuracy is incorrect, feedback is sent to S7 to re-execute the regular expression matching verification; if the format is not standardized, feedback is sent to S8 to adjust the report template style; and S9 and S10 are re-executed after the corresponding steps are optimized.

[0057] The technical concept of this invention is as follows:

[0058] First, the report requirements are analyzed, and four types of business rules are defined: data filtering, calculation, classification, and validation. Regular expression matching rules are configured for field extraction, format validation, and data cleaning. Then, raw data is collected from multiple sources and preprocessed for deduplication, completion, and standardization. The core process integrates a feature importance weighting algorithm based on random forest, an improved K-means clustering algorithm, an anomaly detection algorithm based on gradient boosting decision trees, and a collaborative filtering rule adaptation algorithm. This achieves closed-loop data processing through pairwise interactions between the four algorithms, dynamically adjusting feature weights, cluster centers, anomaly thresholds, and the intensity of business rule execution. Next, regular expression matching completes field format validation and unstructured data extraction. Based on the requirements, a standardized report template is generated and populated with data. Finally, after completeness, accuracy, and standardization checks, the final report is output, comprehensively covering the intelligent processing requirements of the entire report generation process. The automatic report generation process is as follows: Figure 2 As shown.

[0059] S1: Report Requirements Analysis

[0060] Clearly define the core objectives, data range, field requirements, output format, and validation standards for report generation.

[0061] The core objectives include one or more of the following: data aggregation and statistics, anomaly identification, and trend analysis, such as generating monthly sales data summary reports and identifying overdue and unpaid orders.

[0062] The data scope should clearly define the time interval, such as January 1, 2025 to January 31, 2025; the business scenario should be such as online e-commerce orders and offline store sales; and the data subjects should be such as users, products, and transaction records.

[0063] Field requirements: clearly define required fields such as order number, transaction amount, and payment time; optional fields such as user remarks and tracking number; and derived fields such as order discount rate and monthly sales total. Also define the data type of each field, such as string, number, date, and format specifications, such as date format YYYY-MM-DD and amount to be retained to 2 decimal places.

[0064] Output format: Specify the report type, such as table, chart, or mixed type; file format, such as Excel, PDF, or HTML; field display order, such as sorted by order number, product name, transaction amount, or payment time; style requirements, such as bold header and right alignment of numeric fields.

[0065] Verification criteria include: data integrity (e.g., no missing required fields), data accuracy (e.g., numerical range conforms to business logic, transaction amount ≥ 0), and format conformity (e.g., field formats meet preset requirements).

[0066] This step outputs a report requirements specification, which serves as the basis for executing all subsequent steps.

[0067] S2: Business Rule Definition

[0068] Based on the S1 report requirements specification, sort out and define the business rules required for data processing, including four categories: data filtering rules, data calculation rules, data classification rules, and data validation rules. All rules must clearly define the triggering conditions, execution logic, and output results.

[0069] Data filtering rules allow you to set conditions for retaining or removing data. For example, you can retain orders with a transaction amount of ≥100 yuan and a payment status of "paid" and remove transaction records with a logistics status of "cancelled".

[0070] Data calculation rules define the calculation methods for derived fields. For example, the order discount rate is equal to the original price of the product minus the actual payment amount divided by the original price of the product multiplied by 100%, the total monthly sales amount is equal to the sum of the transaction amounts of a single order, and the average daily order volume is equal to the total number of orders in the month divided by the number of natural days in the month.

[0071] Data classification rules categorize data according to business dimensions. For example, orders can be divided into three categories by product type: electronics, clothing, and food. Orders can also be divided into three categories by payment method: WeChat Pay, Alipay, and bank card payment.

[0072] Data validation rules set standards for judging the legality of data. For example, the order number must consist of 8 digits, the payment time must not be later than the deadline for report statistics, and the transaction amount must not exceed 100,000 yuan, except for special products.

[0073] This step outputs a list of business rules, including the unique identifier of each rule, rule type, triggering conditions, execution logic, and output result description, which serves as the basis for S6 execution.

[0074] S3: Regular Expression Matching Rule Configuration

[0075] Based on the field format specifications of S1 and the business rule list of S2, configure regular expression matching rules for field extraction, format validation, and data cleaning. The rules are sourced from industry data standards, business requirements specifications, and historical data format analysis.

[0076] Field extraction regular expressions are used to extract target fields from unstructured text, such as user notes and system logs. For example, the regular expression to extract the phone number from the notes is ^1[3-9]\d{9}$, and the regular expression to extract the product code from the order description is ^SP\d{6}$. The product code starts with SP followed by 6 digits.

[0077] Format validation regular expressions are used to verify whether the field format conforms to S1 requirements. For example, the regular expression to validate the date format is ^\d{4}-\d{2}-\d{2}$, the regular expression to validate the email format is ^[a-zA-Z0-9_-]+@[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$, and the regular expression to validate the order number format is ^DD\d{10}$, where the order number starts with DD followed by 10 digits.

[0078] Data cleaning regular expressions are used to remove invalid characters or redundant information from fields. For example, the regular expression to remove comma separators from transaction amounts is [,,], and the regular expression to remove special symbols from usernames is [^a-zA-Z0-9\u4e00-\u9fa5]. Only Chinese, English, and numbers are retained.

[0079] This step outputs a regular expression matching rule configuration table, which includes rule identifier, applicable fields, regular expression, rule purpose, and matching examples, serving as the basis for S7 execution.

[0080] S4: Raw Data Acquisition

[0081] According to the data range in the S1 report requirements specification, raw data is collected from the preset data sources. The data sources need to be clearly defined in advance through system configuration or manual entry, including three categories: database, local file, and third-party API interface.

[0082] Database collection, extracting data from relational databases such as MySQL or Oracle, or non-relational databases such as MongoDB through the Structured Query Language (SQL). For example, querying all transaction records in the order table of the MySQL database in January 2025. The collected fields include order number, product name, product code, transaction amount, payment time, payment method, logistics status, and user remarks.

[0083] Local file collection, reading data from local files in Excel, CSV, or TXT format. For example, reading the file with the path D: / data / sales_data_January_2025.csv. The collected fields are the same as those in database collection, and row data in the table is extracted through a file parsing tool.

[0084] Third-party API interface collection, calling the API interface provided by a third-party system to obtain data. For example, calling the e-commerce platform API interface https: / / api.xxx.com / orders, passing in the parameters startTime=2025-01-01 and endTime=2025-01-31 to obtain order data within the specified time range, and parsing and extracting the target fields from the JSON / XML format data returned by the interface.

[0085] This step outputs the original data set, which contains all the collected and unprocessed data. The fields are the same as those required in S1 and serve as the input data for S5.

[0086] S5: Preprocessing of original data

[0087] Clean and standardize the original data set collected in S4 to ensure that the data quality meets the execution requirements of subsequent steps. The processed results must fully support the execution of the business rules in S6.

[0088] Data deduplication, removing duplicate data based on a unique identifier field such as the order number. For example, deleting records with duplicate values in the order number field and only retaining the first valid record.

[0089] Missing value handling, filling or removing the required fields defined in S1 in a business-permitted manner. For example, directly removing records with missing values in the transaction amount field and filling the missing user remarks field with "no remarks".

[0090] Data type conversion, uniformly converting the field data types to the types required in S1. For example, converting the transaction amount field from string type to numeric type and converting the payment time field from timestamp format to YYYY-MM-DD HH:MM:SS string format.

[0091] Standardize the format and perform preliminary organization of unstructured fields. For example, standardize "WeChat" and "WeChat Pay" in the payment method field to "WeChat Pay", and standardize the case of the Chinese and English names in the product name field to uppercase.

[0092] This step outputs a clean dataset, which includes deduplicated, completed, and standardized structured data, and serves as the input data for S6.

[0093] S6: Business Rule Execution

[0094] The rule engine is invoked, using the clean dataset output by S5 as the core input. Four deeply correlated, pairwise interactive intelligent algorithms are integrated to collaboratively complete data filtering, calculation, classification, and preliminary verification. These four algorithms are: a feature importance weighting algorithm based on random forest, an improved K-means clustering algorithm, an anomaly detection algorithm based on gradient boosting decision trees, and a collaborative filtering rule adaptation algorithm. The algorithms form a closed-loop interaction mechanism through real-time parameter passing and dynamic result feedback, breaking the limitations of traditional single-algorithm processing and achieving a dual improvement in data processing accuracy and efficiency. The core algorithm collaborative interaction process is as follows: Figure 3 As shown, the specific implementation details are as follows:

[0095] Algorithm 1: Feature Importance Weighted Algorithm (RF) Based on Random Forest

[0096] Model building process: The core of report data processing is the accurate identification of key information. Therefore, the first step is to clarify the core objective of the algorithm: to select features that play a decisive role in data classification and anomaly detection, and assign them appropriate weights. Four core features are selected to construct a feature set. ,in The transaction amount is directly linked to core indicators of sales data statistics; Payment frequency reflects user transaction activity; Product category coding, corresponding to data classification dimensions; The payment method is encoded to reflect the characteristics of the transaction channel. To balance algorithm accuracy and computational efficiency, a random forest of 100 decision trees is constructed. The training samples for each decision tree are randomly selected from historical clean data using bootstrap sampling to ensure sample diversity. The feature subset of each decision tree is determined by randomly selecting two features, calculated based on the square root of the total number of features, to avoid a single feature dominating the decision. Feature importance weights are defined. ,in Represents the first in the feature set The importance weights of each feature range from 0 to 1, and the sum of all feature weights is 1. These weights are used to quantify the degree of influence of different features on the data processing results.

[0097] Model training process: The quality of the training data directly determines the model performance. Therefore, 10,000 historical cleaning data points covering all business scenarios are selected as the training set. Each data point contains a complete feature set. The system uses dual labels: normal data is labeled 1, and abnormal data is labeled 0. The category labels correspond to product categories, ensuring comprehensiveness of the training scenario. When training each decision tree independently, the Gini coefficient is used as the evaluation metric for feature splitting. The Gini coefficient effectively measures the purity of node data; the more significant the increase in purity of child nodes after splitting, the stronger the discriminative power of the corresponding feature. The formula for calculating the splitting gain for each feature is as follows: ,in The Gini coefficient of the parent node. The number of child nodes after the split. For the first The number of samples in each child node. The number of samples in the parent node. For the first The Gini coefficients of each child node are calculated. The split gain of each feature in each decision tree is calculated, and the average of all decision trees is taken as the importance score of that feature. Then, normalization is performed to eliminate the influence of dimensions, resulting in the final feature weights. The normalization formula is After training, the example weights are obtained. The result aligns with business logic. Transaction amount is the core statistical indicator with the highest weight, while payment method coding has a relatively small impact on classification and anomaly detection with the lowest weight.

[0098] Model application process: Extract feature sets from each data point in the clean dataset output by S5. To eliminate the impact of differences in the numerical ranges of different features, each feature is first subjected to Min-Max standardization, converting the feature values ​​into values ​​between 0 and 1. Based on the standardized feature values ​​and the feature weights obtained during training, the weighted sum of features for each data point is calculated. The weighted sum This comprehensive feature sum reflects the strength of the core features of a single data point; a higher value indicates richer key business information contained in the data. In this scheme, the core function of this feature weighted sum is to provide the initial cluster center weights for the subsequently improved K-means clustering algorithm, making the clustering process more closely aligned with the core business features and preventing the clustering results from deviating from actual business needs.

[0099] Algorithm 2: Improved K-means clustering algorithm (K-means⁺)

[0100] Model Construction Process: Based on the S2 data classification rules, the report data needs to be grouped and statistically analyzed according to product categories. Therefore, the core objective of the algorithm is to achieve accurate clustering of the data, corresponding to three categories: electronic products, clothing, and food. The number of clusters is defined. The cluster centers are consistent with the number of product categories to ensure a direct correspondence between the clustering results and business classifications. Cluster centers are a core parameter of the K-means algorithm. To address the instability caused by random initialization of cluster centers in traditional K-means, this algorithm uses the feature weights output from Algorithm 1. Weighted initialization of cluster centers, defining initial cluster centers Each cluster center is a 4-dimensional vector, corresponding to a feature set. The four feature dimensions are: clustering distance, clustering distance, and clustering distance. To improve the business relevance of clustering, clustering distance is defined. To weight the Euclidean distance, feature importance weights are incorporated into the distance calculation, allowing key features to have a greater impact on the clustering results. Cluster labels are defined. The cluster category to which each data point belongs is determined by a value of 1, 2, or 3, corresponding to the categories of electronic products, clothing, and food, respectively, to clarify the classification of the data.

[0101] Model training process: 5000 historical clean data entries processed by Algorithm 1 were selected as training samples. These samples have undergone preliminary screening through feature weighting and contain rich business feature information. First, the initial cluster centers were calculated using the feature weights output by Algorithm 1, using the following formula: ,in , For the first The first in the historical data The average value of each feature is used to calculate the initial cluster centers, making them closer to the distribution of high-weight features and improving the accuracy of cluster initialization. The iterative update process then begins: the first step calculates the weighted Euclidean distance from each training data point to each initial cluster center. ,in For a single training data point, For the first time in this data 1 eigenvalue, For the first The first initial cluster center The first step involves assigning each data point to the nearest cluster center and labeling it with a corresponding cluster tag. The third step is to recalculate the new centers for each cluster. ,in For the first The number of samples in each cluster. For the first A set of samples for each cluster; repeat the above iterative steps until the change in cluster centers is less than a threshold. To ensure stable clustering results, the final cluster centers are obtained upon completion of training. .

[0102] Model application process: Input the current clean data processed by Algorithm 1 into the improved K-means clustering algorithm one by one, and calculate the final cluster center for each data point. Weighted Euclidean distance Data is assigned to corresponding clusters based on the principle of minimum distance, and cluster labels are output. Clearly define the product category to which each data point belongs. Simultaneously, to quantify clustering quality, assess the similarity of data within clusters and calculate the internal similarity of each cluster. The similarity The closer the value is to 0, the more concentrated the data within the cluster, and the better the clustering effect. In algorithm interaction, cluster labels... The anomaly detection algorithm, which serves as the core input to the subsequent gradient boosting decision tree, provides category context information for anomaly detection; intra-cluster similarity... This feedback is then sent to Algorithm 1 to dynamically adjust the feature importance weights, making the feature weights more closely match the current data clustering distribution.

[0103] Algorithm 3: Gradient Boosting Decision Tree Anomaly Detection Algorithm (GBDT)

[0104] Model Construction Process: The accuracy of report data is paramount; therefore, the core objective of the algorithm is to accurately identify abnormal records in the data, such as abnormal transaction amounts or payment times, to prevent abnormal data from affecting the report's statistical results. Considering that anomaly detection requires combining data features and classification information, an input feature set is defined. That is, adding the clustering labels output by Algorithm 2 to the feature set of Algorithm 1. To enhance the targeting of anomaly detection, category information is utilized, as the anomaly judgment criteria differ across product categories. A gradient boosting decision tree is constructed using 50 regression decision trees, with each tree's depth limited to 3. This depth setting ensures detection accuracy while avoiding model overfitting and reducing sensitivity to noisy data. Anomaly scores are defined. This score is used to quantify the probability of data anomalies, ranging from 0 to 1. A higher score indicates a greater probability of data anomalies. An anomaly threshold should be set based on actual business needs. ,when The time stamp data is marked as abnormal data.

[0105] Model training process: 10,000 historical clean data points are selected as the training set, including 8,000 normal data points and 2,000 abnormal data points. The abnormal data covers common anomaly types such as abnormal transaction amounts (e.g., prices far exceeding those of similar products), abnormal payment times (e.g., exceeding the time range of report statistics), and abnormal classification matching (e.g., features severely mismatched with cluster categories). This ensures the model can identify diverse abnormal scenarios. The initial model prediction value is the average anomaly probability of the training set. This provides an initial baseline for model training. For each tree... To 50: The first step is to calculate the negative gradient, i.e., the residual. ,in For the first The true label of each data item: 0 for normal and 1 for abnormal; For the front The predicted values ​​of each tree, and the residuals reflect the difference between the current model's predicted values ​​and the actual values; the second step uses the residuals... As the target variable, training the first A decision tree is used to learn and correct residual patterns, resulting in a regression tree model. The third step is to calculate the step size. ,in The logarithmic loss function measures the deviation between the predicted and actual values; the step size controls the influence of each tree on the model's predictions, preventing excessive step size from causing model oscillations; the fourth step updates the model's predicted values. After all trees have been trained, the predicted values ​​are converted into anomaly scores between 0 and 1 using the sigmoid function. This makes scoring more intuitive and facilitates setting thresholds to identify anomalies.

[0106] Model application process: The clustering labels output by Algorithm 2 are used to... The original feature set of the clean data Combined into input feature set Input each data point into the trained GBDT model and calculate the anomaly score for each data point. Based on the abnormal threshold, mark. If the data is abnormal, output an anomaly flag. (Normal is 0, abnormal is 1) and abnormal score The core contribution of this algorithm is its accurate identification of abnormal data, ensuring the quality of report data, and enabling bidirectional interaction with other algorithms: anomaly scoring. Feedback is sent to Algorithm 2 to adjust the weighting coefficients of cluster distances, making clusters with more outliers more sensitive in distance calculations; outlier labeling. As the core input, the rule adaptation algorithm is passed to the subsequent collaborative filtering, providing a basis for dynamically adjusting business rules.

[0107] Algorithm 4: Collaborative Filtering Rule Adaptation Algorithm (CF)

[0108] Model building process: Business rules are the basis for report data processing, but fixed rules are difficult to adapt to diverse data scenarios. Therefore, the core goal of the algorithm is to dynamically adjust the execution strength of the business rules defined in S2, improving the adaptability of rules and data. Define the rule set. ,in For data filtering rules, For data calculation rules, For data classification rules, The data validation rules correspond one-to-one with the business rule list in S2. Define the rule adaptability. The value ranges from 0 to 1, used to quantify the degree of compatibility between the current data and each business rule; 1 indicates a complete fit, and 0 indicates a complete mismatch. Define the compatibility adjustment coefficient. The anomaly score output by Algorithm 3 The calculated threshold is used to dynamically adjust the execution threshold of business rules, making rule execution more flexible.

[0109] Model training process: 5000 historical cleaning data points and corresponding business rule execution results are selected as the training set. Each data point contains three core input features. The weighted sum of the features output by Algorithm 1 is used. The intra-cluster similarity output by Algorithm 2 The anomaly score output by Algorithm 3 It also includes rule adaptability tags. The labels are assigned by business experts based on the rule execution effect, with 1 indicating a complete fit, 0.5 indicating a partial fit, and 0 indicating no fit, ensuring the accuracy of the training labels. A collaborative filtering model is constructed, drawing on the core idea of ​​user-item collaborative filtering. Each data point is treated as a user, and each business rule as an item. The fit is predicted by calculating the similarity between the data and the rules. The similarity calculation formula is as follows: ,in For the data Input features , , , For rules Corresponding feature requirements, such as data filtering rules regarding transaction amount requirements, The weights of the input features are obtained after training. , , Cluster similarity has the greatest impact on fit. The Pearson correlation coefficient measures the linear correlation between data features and rule requirements. The initial fit is obtained through similarity calculation. Then, the model parameters are optimized by gradient descent to minimize the mean square error between the predicted fitness and the true label, thereby improving the model's prediction accuracy. After training, a stable fitness prediction model is obtained.

[0110] Model application process: Weighted sum of features from Algorithm 1 Internal cluster similarity of Algorithm 2 Abnormal scores of Algorithm 3 Input the collaborative filtering model to predict the rule fit of the current data. Calculate the adaptation adjustment coefficient based on the abnormal scores. The design logic of this formula is that the higher the anomaly score, the further the data deviates from the normal distribution. In this case, the rule execution intensity should be appropriately reduced to avoid excessively removing abnormal data that may have special business scenarios. The lower the anomaly score, the closer the rule execution intensity is to the original setting. The execution threshold of the business rule is dynamically adjusted based on the adaptation adjustment coefficient. For example, the transaction amount ≥ 100 yuan in the data filtering rule in S2 is adjusted to transaction amount ≥ 100 yuan × If a certain data has an abnormal score Then the adjustment coefficient The filtering threshold has been adjusted to 70 yuan, which retains reasonable special data without violating the core filtering logic. Meanwhile, the rule adaptability has been improved. Feedback is sent to Algorithm 1, via formula Adjusting the feature importance weights: the higher the fit, the better the current feature weights meet the business rules, and the smaller the weight adjustment range, to ensure model stability; the lower the fit, the larger the adjustment range should be to make the feature weights better fit the rule requirements.

[0111] Summary of Algorithm Collaboration and Core Contributions

[0112] Bidirectional interaction between Algorithm 1 and Algorithm 2: Feature weights of Algorithm 1 The formula provides the weighting basis for the initial cluster centers of Algorithm 2. This allows cluster initialization to better align with core business features, addressing the instability issue of traditional K-means random initialization; Algorithm 2 improves the intra-cluster similarity. Feedback is sent to Algorithm 1, via formula Adjusting feature weights leads to better clustering results and more stable feature weights, forming a closed loop of "weight-guided clustering and clustering-optimized weights".

[0113] Bidirectional interaction between Algorithm 2 and Algorithm 3: Clustering labels of Algorithm 2 As input features for Algorithm 3, the contextual information for anomaly detection is enriched, allowing for differentiated anomaly judgment criteria for different categories of data, thereby improving detection accuracy; the anomaly score of Algorithm 3 Feedback is sent to Algorithm 2, via formula Adjusting the cluster distance gives a greater distance weight to data with higher outlier scores, making it easier to identify as an independent cluster and preventing outlier data from interfering with normal clustering results.

[0114] Two-way interaction between Algorithm 3 and Algorithm 4: Outlier scores of Algorithm 3 and exception markers To provide data quality reference for Algorithm 4, the more outlier the data, the more cautious the rule fit prediction, and the more lenient the adjustment coefficient; the fit adjustment coefficient of Algorithm 4 Feedback is sent to Algorithm 3, via formula Adjusting the anomaly threshold: the lower the fit coefficient, the worse the fit between the rule and the data. Appropriately lowering the anomaly threshold will more strictly filter out abnormal data and prevent rule loopholes from causing abnormal data to flow into the report.

[0115] Bidirectional interaction between Algorithm 4 and Algorithm 1: Rule fit of Algorithm 4 Feedback is sent to Algorithm 1 to adjust the feature weights, ensuring that the feature importance aligns with business rule requirements; Algorithm 1 performs a weighted sum of features. As input features of Algorithm 4, the core business value of the quantified data is improved, the accuracy of rule fit prediction is enhanced, and the rule adjustment is made to fit the core features of the data.

[0116] The fusion and interaction of the four algorithms achieves three core contributions: First, by combining feature weighting and clustering, the accuracy of data classification is improved, ensuring the accuracy of report classification statistics; second, by linking clustering and anomaly detection, accurate identification of abnormal data is achieved, safeguarding the quality of report data; and third, through a closed loop of anomaly detection and rule adaptation, dynamic adjustment of business rules is achieved, improving the flexibility and adaptability of data processing. This step ultimately outputs a preliminary processed data set, including filtered valid data, derived field calculation results, classification results, and anomaly markers, which serve as the input data for S7.

[0117] S7: Regular Expression Matching Validation and Field Extraction

[0118] Based on the S3 regular expression matching rule configuration table, regular expression matching operations are performed on the preliminary processed data set of S6 to complete field format validation, unstructured data extraction, and secondary data cleaning.

[0119] Field format validation uses regular expression rules to verify whether the format of each field in the initially processed data meets the S1 requirements. For example, the date regular expression rule is used to verify whether the payment time field is in the format YYYY-MM-DD HH:MM:SS, and the order number regular expression rule is used to verify whether the order number field is in the format DD\d{10}. Data with incorrect format is marked as abnormal data.

[0120] Unstructured field extraction: Use field extraction regular expressions to extract target information from unstructured fields, such as user notes. For example, use a mobile phone number regular expression to extract the mobile phone number from the user notes as a new field called "Contact Phone Number", and use a product code regular expression to extract the product code from the order description to supplement the product code field.

[0121] Secondary data cleaning involves using regular expressions to remove invalid characters from fields. For example, redundant characters such as commas and spaces may be removed from the transaction amount field, and special symbols in the username are removed, leaving only Chinese, English, and numbers.

[0122] Data feedback processing involves sending data with abnormal formats back to S5, re-executing the original data preprocessing steps in S5, such as supplementing and correcting fields with incorrect formats; and marking the valid data after secondary cleaning as the final valid data set.

[0123] This step outputs the final valid dataset, including complete data that has passed format validation, unstructured fields have been extracted, and the data has been cleaned a second time. This dataset serves as the input data for S9. Simultaneously, a list of data with formatting errors is output and fed back to S5 for data correction. The process for handling and responding to errors is as follows: Figure 4 As shown.

[0124] S8: Report Template Generation

[0125] Based on the output format requirements in the S1 report requirements specification, create a standardized report template. The template must clearly define the field layout, style specifications, and chart configuration to ensure that the fields correspond one-to-one with the fields of the final valid data set.

[0126] The template structure is designed to lay out the table columns according to the field display order required by S1, such as order number, product name, product code, transaction amount, discount rate, payment time, payment method, and product category. The header name, column width, and row height are set.

[0127] Configure the style specifications as required, setting the font to, for example, SimSun 11pt, the alignment to, for example, bold and centered header, right-aligned numeric fields, and left-aligned text fields, the border style to, for example, a full border, and the color to, for example, light gray background for the header.

[0128] Chart configuration: If it is a chart-type or hybrid report, select the chart type according to the analysis objectives of S1. For example, use a bar chart to display the monthly sales amount of each product category, and use a line chart to display the daily order volume trend within the month. Set the chart title, axis labels, and data series colors.

[0129] This step outputs a standardized report template, including table structure, style specifications, and chart configuration, which will serve as a fill template for S9.

[0130] S9: Report Data Population and Generation

[0131] The final valid data set from S7 is populated into the standardized report template of S8 according to the field correspondence, and the final report file that meets the requirements is automatically generated.

[0132] The table data population function populates each record in the final valid data set into the corresponding column of the template table. For example, it populates the order number field value into the template order number column and the transaction amount field value into the transaction amount column. For categorized data, it displays the data by category, such as displaying order data by product category in chapters.

[0133] Fill in the derived fields and statistical data. Fill the derived fields calculated by S6, such as discount rate, and statistical data, such as monthly sales and average daily order volume, into the specified positions in the template. For example, fill in the monthly sales, total number of orders, and average daily order volume in the summary row at the bottom of the table.

[0134] Chart data binding allows you to link categorized statistical data, such as sales revenue for each product category, to chart components within the template, automatically generating corresponding bar charts, line charts, etc. For example, you can bind monthly sales revenue data for electronic products, clothing, and food to a bar chart to generate a sales comparison chart for each product category.

[0135] File format conversion follows the output format required by S1, converting the completed template into an Excel, PDF, or HTML file. For example, convert the template to an Excel file with a filename such as "2025 January Sales Data Summary Report.xlsx"; or convert it to a PDF file with a filename such as "2025 January Sales Data Summary Report.pdf".

[0136] This step outputs an initial report file containing fully populated data, statistics, and charts, which will serve as the verification object for S10.

[0137] S10: Report Validation and Feedback Optimization

[0138] Perform a final verification on the initial report file generated by S9 to check whether it fully complies with the report requirements specification of S1. If the verification passes, output the final report; otherwise, provide feedback to the corresponding step for optimization.

[0139] Completeness check: Check whether the report contains all required fields, derived fields and statistical data required by S1. For example, whether there are any missing required fields such as order number and transaction amount, and whether it contains derived fields and statistical data such as total monthly sales and discount rate.

[0140] Accuracy verification involves checking whether the report data matches the final valid data set in S7. For example, whether the number of orders in the table matches the number of records in the final valid data set, whether the total transaction amount matches the sum of transaction amounts in the final valid data set, and whether the chart data matches the categorized statistical data.

[0141] Format compliance check: Check whether the file format, style, and field order of the report meet the S1 requirements. For example, whether it is the specified Excel / PDF format, whether the header style and alignment are consistent with the template, and whether the field display order meets the preset requirements.

[0142] Feedback and optimization: If the verification passes, output the final report file and store it in the specified path, such as server / report output / January 2025 / directory; if the verification fails, identify the problem type, such as incompleteness, inaccuracy, or non-standard format, and provide feedback to the corresponding steps: Incompleteness is reported to S6 to supplement the calculation of derived fields; inaccuracy is reported to S7 to re-execute the regular expression matching verification; non-standard format is reported to S8 to adjust the report template style. After the corresponding steps are optimized, S9 and S10 are re-executed.

[0143] This step outputs the final report file. If the verification fails, a list of report verification issues will be output. If the verification passes, the automatic report generation process will be completed.

[0144] In a specific example: A summary report of sales data for May 2025 is generated for an e-commerce platform:

[0145] Scene setting

[0146] An e-commerce platform needs to generate a summary report of sales data for May 2025. The core requirements are to classify and statistically analyze the total sales, order volume, and average daily sales of three categories of goods: electronics, clothing, and food; identify orders with abnormal transaction amounts and payment times; and output an Excel-formatted report (including a data table and a sales bar chart for each product category). The fields should be displayed in the following order: order number, product name, product code, transaction amount, discount rate, payment time, payment method, product category, and whether it is abnormal.

[0147] S1: Report Requirements Analysis

[0148] The core objectives are to classify and statistically analyze the sales data of three types of goods and identify abnormal orders. The data range is online e-commerce orders from May 1, 2025 to May 31, 2025, and the data subject is order records.

[0149] Required fields include order number, product name, product code, transaction amount, payment time, payment method, and product category. Optional fields include user remarks. Derivative fields include discount rate, total monthly sales, number of orders per category, and average daily sales.

[0150] The field format specifications are as follows: the order number starts with DD followed by 10 digits; the transaction amount is a numeric type with 2 decimal places; the payment time format is YYYY-MM-DD HH:MM:SS; and the product category is limited to electronic products, clothing, and food.

[0151] The output format is an Excel file with bold headers, right-aligned numerical fields, and a bar chart (horizontal axis for product category, vertical axis for total sales).

[0152] The verification criteria are: no missing required fields, transaction amount ≥ 0, payment time between May 1, 2025 and May 31, 2025, and product category within the specified range.

[0153] This step outputs a report requirements specification, which will serve as the basis for subsequent execution.

[0154] S2: Business Rule Definition

[0155] The data filtering rule is to retain orders with a payment status of "paid" and a logistics status of "not cancelled," and to remove records with a transaction amount of less than 0 yuan.

[0156] The data calculation rules are as follows: Discount rate = (Original price of the product - Actual payment amount) / Original price of the product × 100% (rounded to 1 decimal place), Monthly total sales = ∑ Transaction amount of a single order, Category order volume = Number of valid orders for a certain category of products, Average daily sales = Category order volume / 31 days (rounded to 0 decimal places).

[0157] The data classification rules are as follows: based on product category, it is divided into three categories: electronic products, clothing, and food; based on payment method, it is divided into three categories: WeChat Pay, Alipay, and bank card payment.

[0158] The data verification rules are as follows: the order number format must be DD + 10 digits, the transaction amount must be ≤ 100,000 yuan (except for special products), the discount rate must be between 0% and 100%, and the payment time must not exceed the statistical range.

[0159] This step outputs a list of business rules, which will serve as the basis for S6 execution.

[0160] S3: Regular Expression Matching Rule Configuration

[0161] The regular expression rules for field extraction include ^1[3-9]\d{9}$ for extracting the phone number from the user's remarks and ^SP\d{6}$ for extracting the product code from the order description.

[0162] The format validation regular expression rules include ^DD\d{10}$ for order number, ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$ for payment time, and ^2025-05-(0[1-9]|

[12] [0-9]|3

[01] )$ for date range validation.

[0163] Data cleaning regular expressions include removing commas from transaction amounts ([,,]) and removing special characters from user names ([^a-zA-Z0-9\u4e00-\u9fa5]).

[0164] This step outputs a regular expression matching rule configuration table, which serves as the basis for S7's execution.

[0165] S4: Raw Data Acquisition

[0166] The database collected data by querying the MySQL database order table for orders in May 2025. The collected fields were order number, product name, product code, original price of the product, actual payment amount, payment time, payment method, logistics status, and user remarks. A total of 12,000 records were collected.

[0167] The local file was read from the file D: / e-commerce data / May 2025 offline supplementary orders.csv, and 800 order records were collected. The fields were consistent with those collected from the database.

[0168] The third-party API interface calls the logistics platform's API interface https: / / api.logistics.com / orders, passing in the parameters startTime=2025-05-01 and endTime=2025-05-31, to obtain supplementary logistics status data for 300 orders.

[0169] This step outputs a raw dataset containing 13,100 records, which serves as the input for S5.

[0170] S5: Raw Data Preprocessing

[0171] Data deduplication was performed based on the order number field, and two duplicate records, DD20250500123 and DD20250500456, were found. After removing them, 13098 records were retained.

[0172] In handling missing values, 3 records with missing transaction amounts were directly removed, and 200 records with missing user remarks were filled with "no remarks".

[0173] The data type conversion converts the transaction amount from a string type to a numeric type and the payment time from a timestamp 1740969600 to the format 2025-05-01 00:00:00.

[0174] The format standardization will unify "WeChat" in the payment method field to "WeChat Pay" and "iphone15" in the product name field to "IPHONE15".

[0175] This step outputs a clean data set containing 13,095 records, which serves as the input for S6.

[0176] S6: Business Rule Execution

[0177] The S5 clean data set is input into the rule engine and processed collaboratively by four algorithms. The specific process is as follows:

[0178] Algorithm 1: Feature Importance Weighted Algorithm (RF) Based on Random Forest

[0179] Model building: feature set ,in This refers to the transaction amount (the actual amount paid). Payment frequency (number of orders placed by the user in May). Code the product categories (electronic products = 1, clothing = 2, food = 3). Encode the payment methods (WeChat Pay = 1, Alipay = 2, Bank Card = 3). Construct 100 decision trees, with each tree randomly selecting 2 features.

[0180] Model training: Train the model using 10,000 historical data points and calculate the splitting gain for each feature. For example... The average splitting gain is 0.8. It is 0.4. It is 0.6. The weight is 0.2. (Normalized weight) .

[0181] Model application: Standardizing features of clean data, such as a specific order. The standardization is 0.7. Standardized to 0.5, Standardized to 0.3, Standardize to 0.3. Calculate the feature-weighted sum. This value is passed to Algorithm 2.

[0182] Algorithm 2: Improved K-means clustering algorithm (K-means⁺)

[0183] Model building: Number of clusters Initial cluster centers Depend on Weighted calculation, cluster distance is weighted Euclidean distance, cluster labels This corresponds to three categories of goods.

[0184] Model training: Train using 5000 historical data points to calculate initial cluster centers. (Electronic products) (clothing), (Food). Final cluster centers after iterative updates. , , .

[0185] Model application: Calculating the feature value of a specific order. Distance to the final cluster center ,distance Recently, tags (Electronic products). Calculate the intra-cluster similarity. Feedback is sent to Algorithm 1 to adjust the weights. .

[0186] Algorithm 3: Gradient Boosting Decision Tree Anomaly Detection Algorithm (GBDT)

[0187] Model building: Input feature set 50 decision trees (depth 3), anomaly threshold .

[0188] Model training: The model was trained using 10,000 historical data points, including 2,000 outlier data points (such as an electronic product order with a transaction amount of 99,999 yuan). After training, the model's predictions for the outlier data were transformed using a sigmoid function. .

[0189] Model Application: Input features of a certain order are Calculate the abnormal score (Normal), marked Another order had a transaction amount of 100,000 yuan. (After entering the amount...) ,mark (Abnormal). Abnormal score Feedback is sent to Algorithm 2 to adjust the cluster distance. .

[0190] Algorithm 4: Collaborative Filtering Rule Adaptation Algorithm (CF)

[0191] Model building: rule set Adaptability to the four major business rules Values ​​range from 0 to 1, adjustment coefficient .

[0192] Model training: Train using 5000 historical data points, inputting feature weights. After training, the model can accurately predict the fitness level.

[0193] Model Application: Input of a Certain Order Predicting fit Calculate the adjustment factor. The data filtering rule "transaction amount ≥ 100 yuan" has been adjusted to "transaction amount ≥ 100 × 0.94 = 94 yuan". (Adaptability) Feedback is sent to Algorithm 1 to adjust the feature weights. .

[0194] Algorithm Collaboration Results

[0195] After processing by four algorithms, 12,800 valid orders were selected, including 4,200 electronic product orders, 5,300 clothing orders, and 3,300 food orders. 295 abnormal orders were marked (200 with abnormal transaction amounts and 95 with abnormal payment times). Derivative fields such as discount rates and order volume by category were calculated, and a preliminary processed data set was output.

[0196] S7: Regular Expression Matching Validation and Field Extraction

[0197] The field format was validated using regular expressions. Three records with order numbers DD2025050078 (only 8 digits) were found and marked as data with abnormal format. The data was then sent to S5 for reprocessing.

[0198] Unstructured field extraction: Extract the mobile phone number 138XXXXXXXX from the user's remarks "Contact number 138XXXXXXXX" and add it to the data as a contact number field.

[0199] The data is cleaned a second time to remove commas from the transaction amount field, such as "5,999" being corrected to 5999, and to remove the "@" symbol from the username, such as "Zhang San@" being corrected to Zhang San.

[0200] This step outputs a final valid dataset containing 12,797 records, which serves as the input for S9.

[0201] S8: Report Template Generation

[0202] The template structure arranges the table columns in the following order: order number, product name, product code, transaction amount, discount rate, payment time, payment method, product category, and whether it is abnormal. The table header name is consistent with the field name, and the column width is set to 12 characters.

[0203] The style specification sets the font to SimSun, size 11, the table header to bold and centered, the numerical fields such as transaction amount and discount rate to right alignment, the product category field to center alignment, the border to full border, and the table header background color to light gray.

[0204] The chart configuration includes a bar chart with the horizontal axis representing product categories (electronics, apparel, food) and the vertical axis representing total sales (unit: 10,000 yuan). The chart title is "Comparison of Total Sales by Product Category in May 2025," and the data series colors are blue, red, and green.

[0205] This step outputs a standardized report template, which will be used as a fill template for S9.

[0206] S9: Report Data Population and Generation

[0207] The table data population function fills each record in the final valid data set into the template one by one. For example, the record for order number DD20250500123 is populated as DD20250500123, IPHONE15, SP202501, 5999.00, 10.0%, 2025-05-05 14:30:22, WeChat Pay, Electronic Products, No.

[0208] The derived fields and statistical data were used to calculate the total sales of electronic products as 25.1958 million yuan, with 4,200 orders and an average daily sales volume of 135; the total sales of clothing as 25.97 million yuan, with 5,300 orders and an average daily sales volume of 171; and the total sales of food as 3.1381 million yuan, with 3,300 orders and an average daily sales volume of 106. These figures were then filled into the summary row at the bottom of the table.

[0209] The chart data binding function links the total sales data of the three product categories to a bar chart, automatically generating a bar chart comparing the total sales of product categories in May 2025.

[0210] The file format conversion will convert the completed template into an Excel file named "May 2025 E-commerce Platform Sales Data Summary Report.xlsx".

[0211] This step outputs an initial report file, which serves as the verification object for S10.

[0212] S10: Report Validation and Feedback Optimization

[0213] The integrity check report contains all required fields, derived fields, and statistical data, with no missing items.

[0214] The accuracy verification confirmed that the number of orders in the table (12,797) matched the number of records in the final valid data set, the total transaction amount (54.3039 million yuan) matched the sum of the transaction amounts of each order, and the chart data matched the categorized statistical data.

[0215] The format conformity check confirmed that the report is in Excel format, the header style and alignment are consistent with the template, and the field display order meets the preset requirements.

[0216] After successful verification, the final report file will be stored in the server / report output / May 2025 / directory, completing the report generation.

[0217] In summary, several specific embodiments of the present invention have been disclosed. Without contradiction, the various embodiments can be freely combined to form new embodiments. That is, embodiments that are alternative solutions can be freely substituted for each other, but cannot be combined with each other; embodiments that are not alternative solutions can be combined with each other. These new embodiments are also part of the substantive content of the present invention.

[0218] The above embodiments describe several specific implementations of the present invention. However, those skilled in the art should understand that various changes or modifications can be made to these implementations without departing from the principles and essence of the present invention, but all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A method for automatically generating reports based on business rules and regular expression matching, characterized in that, include: S1. Report Requirements Analysis: Clarify the core objectives, data range, field requirements, output format, and validation standards for report generation, and output the report requirements specification; S2. Business Rule Definition: Based on the report requirements specification, sort out and define data filtering rules, data calculation rules, data classification rules, and data validation rules, and output a list of business rules; S3. Regular expression matching rule configuration: Based on the field format specifications and business rule list in the report requirements specification, configure the field extraction regular expression rules, format validation regular expression rules, and data cleaning regular expression rules, and output the regular expression matching rule configuration table; S4. Raw Data Acquisition: Based on the data range specified in the report requirements specification, raw data is collected from three preset data sources: database, local file, and third-party API interface, and the raw data set is output. S5. Raw Data Preprocessing: Perform data deduplication, missing value handling, data type conversion, and format standardization on the raw dataset to output a clean dataset; S6. Business Rule Execution: The rule engine is invoked, taking the clean data set as input. A feature importance weighting algorithm based on random forest, an improved K-means clustering algorithm, an anomaly detection algorithm based on gradient boosting decision trees, and a collaborative filtering rule adaptation algorithm are integrated. The feature importance weighting algorithm based on random forest includes: constructing a feature set containing transaction amount, payment frequency, product category code, and payment method code; calculating feature splitting gain using Gini coefficients and normalizing it to obtain feature importance weights; and calculating the weighted sum of the clean data features. This weighted sum is used as the input parameter for the improved K-means clustering algorithm. Simultaneously, it receives feedback from the rule-fitting algorithm of collaborative filtering to dynamically adjust the feature importance weights. The improved K-means clustering algorithm includes: determining the number of clusters based on the classification requirements of the report specification; calculating initial cluster centers using the feature importance weights output by the feature importance weighting algorithm based on random forest; measuring the similarity between data and cluster centers using weighted Euclidean distance; iteratively updating the cluster centers until they stabilize; and outputting cluster labels, which serve as input features for the anomaly detection algorithm of gradient boosting decision trees. Simultaneously, it calculates the intra-cluster similarity and feeds it back to the feature importance weighting algorithm based on random forest to adjust the feature weights. The gradient boosting decision tree anomaly detection algorithm includes: constructing an input feature set containing a feature set based on a random forest-based feature importance weighting algorithm and clustering labels output by an improved K-means clustering algorithm; learning residual correction rules through iterative training with multiple regression decision trees; obtaining anomaly scores and labeling anomaly data through sigmoid function transformation; outputting anomaly labels and anomaly scores; and feeding the anomaly scores back to the improved K-means clustering algorithm and the collaborative filtering rule adaptation algorithm for parameter adjustment. The collaborative filtering rule adaptation algorithm includes: weighting the features output by the random forest-based feature importance weighting algorithm... The improved K-means clustering algorithm outputs intra-cluster similarity, and the gradient boosting decision tree anomaly detection algorithm outputs anomaly scores as input features. The similarity between the data and business rules is calculated using the Pearson correlation coefficient to predict rule fit. Based on the anomaly scores, a fit adjustment coefficient is calculated to dynamically adjust the business rule execution threshold. The rule fit is fed back to a random forest-based feature importance weighting algorithm to adjust feature weights. A closed-loop processing mechanism is formed through the bidirectional interaction of the four algorithms, completing data filtering, calculation, classification, and preliminary verification, outputting a pre-processed data set. This closed-loop processing mechanism includes: The two-way interaction between Algorithm 1 and Algorithm 2: the feature weights of Algorithm 1 provide the weighting basis for the initial cluster centers of Algorithm 2, and the intra-cluster similarity of Algorithm 2 is fed back to Algorithm 1 through the formula. Adjust the feature weights, where, This represents the importance weight of the i-th feature in the feature set. Similarity within a cluster; The two-way interaction between Algorithm 2 and Algorithm 3: The clustering labels of Algorithm 2 serve as input features for Algorithm 3, and the outlier scores of Algorithm 3 are fed back to Algorithm 2 through the formula. Adjusting cluster distance, where cluster distance is defined. For weighted Euclidean distance, For a single training data point, As the final cluster center, This is an abnormal score; The two-way interaction between Algorithm 3 and Algorithm 4: the outlier scores and outlier markers of Algorithm 3 provide data quality references for Algorithm 4, and the adaptation adjustment coefficients of Algorithm 4 are fed back to Algorithm 3 through the formula. Adjust the abnormal threshold, where, This is the abnormal threshold. To adapt and adjust the coefficients; Two-way interaction between Algorithm 4 and Algorithm 1: The rule fit of Algorithm 4 is fed back to Algorithm 1 to adjust the feature weights, and the weighted sum of features of Algorithm 1 is used as the input features of Algorithm 4. S7. Regular expression matching validation and field extraction: Based on the regular expression matching rule configuration table, the field format is validated, unstructured fields are extracted and the data is cleaned again in the preliminary processed data set. Data with abnormal format is fed back to S5 for reprocessing and the final valid data set is output. S8. Report Template Generation: Based on the output format requirements of the report requirements specification, create a standardized report template that includes field layout, style specifications, and chart configuration, and output the standardized report template; S9. Report Data Population and Generation: Populate the final valid data set into the standardized report template according to the field correspondence, complete the filling of table data, derived fields and statistical data, and the binding of chart data, convert it into an initial report file in the specified format, and output the initial report file; S10. Report Validation and Feedback Optimization: Perform completeness, accuracy, and format standardization checks on the initial report file. If the check passes, the final report file is output. If it fails, feedback is sent to the corresponding step for optimization, and the subsequent process is re-executed.

2. The automatic report generation method based on business rules and regular expression matching according to claim 1, characterized in that, The bidirectional interaction mechanism of the four algorithms in S6 is as follows: the feature importance weighting algorithm based on random forest and the improved K-means clustering algorithm feed back to each other through feature weights and intra-cluster similarity; the improved K-means clustering algorithm and the gradient boosting decision tree anomaly detection algorithm feed back to each other through cluster labels and anomaly scores; the gradient boosting decision tree anomaly detection algorithm and the collaborative filtering rule adaptation algorithm feed back to each other through anomaly scores and adaptation adjustment coefficients; and the collaborative filtering rule adaptation algorithm and the feature importance weighting algorithm based on random forest feed back to each other through rule fit and feature weighted sum.

3. The automatic report generation method based on business rules and regular expression matching according to claim 1, characterized in that, In S3, field extraction regular expressions are used to extract target fields from unstructured text, format validation regular expressions are used to verify the format of fields, and data cleaning regular expressions are used to remove invalid characters.

4. The automatic report generation method based on business rules and regular expression matching according to claim 1, characterized in that, In S5, missing value handling involves directly removing or filling with default values ​​that are allowed by the business for required fields defined in the report requirements specification. When optional fields are missing, preset default values ​​are used to fill them, ensuring the integrity of the clean data set to support subsequent algorithm processing.

5. The automatic report generation method based on business rules and regular expression matching according to claim 1, characterized in that, In S10, the integrity check verifies whether the report contains all required fields, derived fields, and statistical data as required by the report requirements specification; the accuracy check verifies the consistency between the report data and the final valid data set; and the format compliance check verifies whether the report file format, style, and field order meet the requirements of the report requirements specification.

6. The automatic report generation method based on business rules and regular expression matching according to claim 1, characterized in that, The specific feedback optimization path for S10 is as follows: if the completeness is missing, feedback is sent to S6 to supplement the calculation of derived fields; if the accuracy is incorrect, feedback is sent to S7 to re-execute the regular expression matching validation; if the format is not standardized, feedback is sent to S8 to adjust the report template style. After the corresponding steps are optimized, S9 and S10 are re-executed.

Images