Data packet optimization processing method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business scenes such as financial science and technology and medical health, and discloses a data grouping optimization processing method, device and equipment and a medium. The method comprises the steps that input data are collected, cleaning and format unification are completed, and cleaned data are generated; generating a grouping mark from a condition constraint library based on the data attribute field; dividing the data into a plurality of subsets according to a segmentation strategy and distributing the subsets to a plurality of processing nodes; generating a grouping key in each processing node according to the grouping mark and the data attribute field; grouping identifiers are distributed for the records based on the grouping keys, and a local grouping result table is formed; and finally, collecting all local results, verifying the consistency, and generating a global grouping result table. According to the method, sorting and key generation are performed in combination with the data attribute fields and the grouping marks, high-precision grouping identification is realized, and the accuracy of a global grouping result is guaranteed through an inter-node consistency verification mechanism.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data grouping optimization processing method, device, equipment and storage medium. Background Art

[0002] As data grouping and processing technologies continue to advance, industries such as insurance, finance, and healthcare are increasingly demanding accuracy, processing efficiency, and system adaptability in data grouping. In the fintech sector in particular, the implementation of IFRS 17, the international financial reporting standard, has made the accounting process for insurance contracts increasingly complex. IFRS 17 imposes stricter rules and constraints on contract grouping, requiring, for example, detailed grouping based on contract characteristics, risk profiles, and profitability, and the use of various measurement methods, including a general model, premium allocation method, and sliding-fee method. This makes traditional policy grouping methods based on manual experience or static rule engines difficult to cover all applicable scenarios. This not only results in highly coupled grouping logic and difficulty in maintenance, but also easily leads to errors when contract classification boundaries are unclear, impacting the accuracy and compliance of the final financial statements.

[0003] In the healthcare business, the grouping and processing of patient data also faces similar challenges. For example, in scenarios such as chronic disease management, medical insurance settlement, or clinical pathway analysis, the system needs to group and classify medical records based on fields such as patient attributes, diagnosis and treatment behavior, and cost patterns to support subsequent analysis modeling and capital flow processing. However, traditional data grouping processes often use static label configuration or fixed rule mapping, lack support for dynamic combination of multiple attributes, and are difficult to adapt to complex clinical data structures and heterogeneous data sources. This leads to low grouping accuracy, which in turn affects the effectiveness of health management strategies or the accuracy of medical insurance compliance accounting.

[0004] Existing grouping methods typically rely on a serial processing architecture, making it difficult to cope with the demands of high-concurrency processing of large-scale data. During the processing process, the system lacks an efficient data segmentation mechanism and distributed scheduling capabilities, resulting in limited data processing throughput. At the same time, the data cleansing, field standardization, condition matching, and grouping logic required for different business rules are often hard-coded into the processing flow, resulting in poor system flexibility. Once the rules are adjusted or the data source structure changes, the system needs to be redeployed or the processing flow needs to be manually revised, resulting in high operation and maintenance costs. In addition, due to ambiguity in rule interpretation and the lack of a consistency verification mechanism, the grouping results generated by different nodes may conflict, further affecting data consistency and the quality of business decisions. Summary of the Invention

[0005] The main purpose of the present invention is to provide a data grouping optimization processing method, device, equipment and storage medium, aiming to solve the technical problems of the existing technology in processing complex multi-dimensional project groupings that meet specific criteria, which result in complex processing rules, low efficiency, insufficient flexibility and difficulty in ensuring data consistency due to reliance on manual experience and simple tools.

[0006] To achieve the above object, the present invention provides a data group optimization processing method, comprising:

[0007] Collecting input data to generate an initial data set, and performing cleaning and formatting processing on the initial data set to generate cleaned data;

[0008] Calling a condition constraint library and generating a corresponding grouping tag for each record in the cleaned data based on the data attribute field;

[0009] Dividing the cleaned data into multiple data subsets according to a preset segmentation strategy and distributing them to multiple processing nodes;

[0010] On each processing node, sorting the data subset based on the grouping tag and the data attribute field and generating a grouping key;

[0011] Based on the grouping key, assigning group identifiers to the records in the data subset and forming a local grouping result table;

[0012] The local grouping result tables of all processing nodes are aggregated and consistency checked to generate a global grouping result table.

[0013] Furthermore, to achieve the above-mentioned object, the present invention provides a data group optimization processing device, comprising:

[0014] The data acquisition and preprocessing module is used to collect input data to generate an initial data set, and perform cleaning and format unification processing on the initial data set to generate cleaned data;

[0015] A grouping tag generation module, configured to call a conditional constraint library and generate a corresponding grouping tag for each record in the cleaned data based on a data attribute field;

[0016] A data segmentation scheduling module is used to divide the cleaned data into multiple data subsets and distribute them to multiple processing nodes according to a preset segmentation strategy;

[0017] A grouping key generation module is used to perform sorting processing on the data subset based on the grouping mark and the data attribute field and generate a grouping key on each processing node;

[0018] A local grouping construction module, configured to assign group identifiers to records in the data subset based on the grouping key and form a local grouping result table;

[0019] The global grouping aggregation module is used to aggregate the local grouping result tables of all processing nodes and perform consistency checks to generate a global grouping result table.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data group optimization processing program stored in the memory and runnable on the processor, and when the data group optimization processing program is executed by the processor, the steps of the data group optimization processing method as described above are implemented.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data grouping optimization processing program is stored. When the data grouping optimization processing program is executed by a processor, the steps of the data grouping optimization processing method described above are implemented.

[0022] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A data grouping optimization processing method, device, equipment and medium are disclosed, including: collecting input data to generate an initial data set, performing cleaning and format unification processing on the initial data set to generate cleaned data; calling a condition constraint library and generating group tags for records in the cleaned data based on data attribute fields; dividing the cleaned data into multiple data subsets and assigning them to multiple processing nodes according to a preset segmentation strategy; on each processing node, sorting the data subsets based on the group tags and data attribute fields and generating group keys; assigning group identifiers to records in the data subsets based on the group keys and forming a local grouping result table; aggregating the local grouping result tables of all processing nodes and performing consistency verification to generate a global grouping result table. The present invention realizes high-precision group identification by combining data attribute fields with group tags for sorting and key generation, and ensures the accuracy of global grouping results through a consistency verification mechanism between nodes; adopts distributed segmentation and parallel processing to improve processing efficiency, and at the same time uses the condition constraint library to dynamically adapt complex grouping rules, thereby enhancing the processing stability and scalability of the system in a high concurrency and multi-rule environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0024] Figure 1 A schematic diagram of an application environment of a data group optimization processing method according to an embodiment of the present invention;

[0025] Figure 2 This is a flow chart of an embodiment of a data group optimization processing method of the present invention;

[0026] Figure 3 Schematic diagram of functional modules of a preferred embodiment of the data packet optimization processing device of the present invention;

[0027] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0028] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] The data grouping optimization processing method provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can collect input data from the user end to generate an initial data set, perform cleaning and format unification processing on the initial data set to generate cleaned data; call the condition constraint library and generate grouping tags for the records in the cleaned data based on the data attribute field; divide the cleaned data into multiple data subsets according to the preset segmentation strategy and distribute them to multiple processing nodes; on each processing node, perform sorting processing on the data subset based on the grouping tag and the data attribute field and generate a grouping key; assign a grouping identifier to the records in the data subset based on the grouping key and form a local grouping result table; aggregate the local grouping result tables of all processing nodes and perform consistency verification to generate a global grouping result table. The present invention realizes high-precision group identification by combining data attribute fields with grouping tags for sorting and key generation, and ensures the accuracy of the global grouping results through the consistency verification mechanism between nodes; adopts distributed segmentation and parallel processing to improve processing efficiency, and at the same time uses the condition constraint library to dynamically adapt complex grouping rules, thereby enhancing the processing stability and scalability of the system in a high concurrency and multi-rule environment. The user end may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server end may be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below using specific embodiments.

[0031] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the data group optimization processing method provided by the present invention. It should be noted that although the flow chart shows a logical order, in some cases, the steps shown or described may be performed in a different order than here.

[0032] like Figure 2 As shown, the data grouping optimization processing method proposed by the present invention includes the following steps:

[0033] S10, collecting input data to generate an initial data set, and performing cleaning and formatting processing on the initial data set to generate cleaned data;

[0034] In this embodiment, collecting input data involves establishing connection channels with multiple data sources. These data sources can include structured database systems such as MySQL and PostgreSQL, distributed file systems such as HDFS, and object storage systems such as MinIO. Establishing these connection channels relies on configuring the corresponding communication protocols and connection interfaces, such as JDBC connection configuration, API key verification, and network access control policies. After authentication, access to the source data is obtained. This process not only requires interface accessibility but also ensures permission control and data security to ensure the stability of subsequent data reading.

[0035] Reading raw input data involves extracting a data set within a specified range or format from the connected data source. This can include extracting the most recent day's data based on timestamp fields, importing the entire data from a specified business table, or incrementally reading newly added data records. Input data can be structured (such as table records), semi-structured (such as JSON logs), or unstructured (such as nested text), requiring standardized structure to adapt to subsequent operation modules.

[0036] Loading the read raw data into a memory buffer aims to construct an initial dataset for computation and processing. The buffer's data structure should include key-value mapping or a two-dimensional table structure to facilitate field indexing and batch iteration. This step requires pre-identification of different field types and format mapping, preparing for the cleansing phase.

[0037] Cleaning primarily involves removing missing fields, outliers, and redundant records. Missing fields are identified through field integrity rules. For example, the "Policy Number" field must be non-null in all records. Missing fields can be handled by deleting the entire record or filling in the field with the default value. For obvious outliers, such as time fields that are prior to the system's earliest transaction date or negative amounts, appropriate thresholds are set to filter out or correct them.

[0038] Format unification includes standardizing the data formats of key fields. Date fields can be converted to the standardized ISO 8601 format (e.g., YYYY-MM-DD). Numeric fields are standardized to floating-point numbers and their units are normalized, for example, converting RMB and USD to a standard currency unit. Text classification fields, such as status codes and insurance classifications, are normalized to enumerated values ​​using mapping tables to facilitate subsequent modeling and judgment.

[0039] The final processing result should form a data set with consistent format, complete fields, and cleared anomalies, that is, cleaned data, for use in subsequent grouping and parallel processing stages.

[0040] During implementation, ETL scheduling engines such as Apache NiFi and Airflow can be used in conjunction with Spark or Flink jobs to complete the aforementioned data reading, format conversion, and standardized cleansing. When establishing a connection, you can configure the connector module to centrally manage data source parameters and set a retry mechanism for failures. For data reading, you can set a time window to control the granularity of the read. The buffer structure uses columnar storage to improve field access efficiency.

[0041] For field cleaning, you can use a pre-set rule library to filter or replace invalid data using regular expressions or threshold matching strategies. Format standardization can be performed during the initial load phase, or by utilizing mapping tables for batch field transformations during the cleaning process. These operations are performed through configuration-driven or parameter-templated methods, enabling rapid rule updates without requiring program modifications.

[0042] For data from different sources, you can use an abstract field model to establish unified field naming conventions and data type mapping rules to avoid subsequent processing failures due to inconsistent field naming. In scenarios with multiple data sources, you can also introduce a data validation module to perform row count verification and field match rate statistics on received datasets to ensure data quality consistency.

[0043] Example: In the fintech sector, when processing policy, claims, and customer interaction data from multiple channels, the data may contain different formats, missing fields, and duplicate records. The above processing steps can consolidate policy data output from different systems into a dataset with consistent fields and accurate content, facilitating the subsequent implementation of contract grouping and measurement rules in accordance with IFRS 17.

[0044] In the healthcare sector, data uploaded by hospital information systems, medical insurance platforms, and third-party service providers often suffer from formatting discrepancies and field conflicts. By establishing a structured input buffer and standardized field mapping rules, medical records, expense items, and patient information from various sources can be cleansed and standardized, improving the accuracy and compliance of subsequent grouping statistics and medical insurance reimbursement reviews.

[0045] This embodiment collects, cleans, and formats input data to construct a standardized, comprehensive dataset, laying an accurate foundation for subsequent data grouping and parallel processing. The unified data format improves the accuracy of rule judgments, eliminates abnormal data, and reduces the risk of incorrect grouping. The standardized data structure significantly improves data consistency and scheduling efficiency across multiple processing nodes.

[0046] S20, calling a condition constraint library and generating a corresponding grouping tag for each record in the cleaned data based on the data attribute field;

[0047] In this embodiment, the call condition constraint library refers to a pre-built information set in the system that contains grouping judgment rules, constraint expressions, and logical condition combination relationships. The library can exist in the form of a rule tree, a JSON structure, or a condition expression file, and supports multiple expressions such as logical conditions (such as greater than, less than, equal to), enumeration matching, and regular judgment. The core function of the condition constraint library is to provide a grouping basis for different types of data records. Different rule sets are suitable for grouping requirements in different business contexts.

[0048] Data attribute fields are fields in the cleaned data that can be used for grouping decisions. These typically include category fields (such as product type, customer level, and insurance type), time fields (such as underwriting time and application date), and status fields (such as contract status and claim status). These fields can be used individually or in combination to perform logical calculations that match conditional constraints. The system extracts field values ​​and identifies their types to provide the foundation for subsequent generation of grouping tags.

[0049] Generating group tags means assigning a unique identifier to records that meet a set of conditional constraints. This identifier can be a static label (such as "Group A" or "Group B") or a dynamically calculated result (such as a hash value, a composite key, or a conditional path code). The generation method can be based on the matching path, performing classification coding, or forming a group sequence number based on the rule priority.

[0050] In practice, this process requires traversing every record in the cleaned data and comparing the values ​​of one or more data attribute fields in each record against the matching rules in the conditional constraint library to determine which group the record should be assigned to. The system uses a conditional engine component to parse and execute conditional expressions, and then performs logical judgment based on field values. Records that meet the same set of constraints are assigned the same grouping tag. Records that do not meet the same set of constraints can be assigned to the "default group" or assigned a new tag.

[0051] This process can be implemented using a rule-driven conditional engine such as Drools or an expression-based function library (such as a custom DSL interpreter). The conditional constraint library can be stored in a configuration file and loaded into memory for initialization. The field mapping module automatically identifies the field type of data attribute fields and performs preprocessing operations such as whitespace removal, unit standardization, and case unification.

[0052] For large-scale data scenarios, a parallel stream approach can be used to perform conditional matching operations on data records. Conditional judgments can be executed concurrently on multiple records through a thread pool or distributed computing framework, and grouping tags can be quickly written to records that match the rules. For rule sets containing complex logic, a rule priority mechanism can be introduced to ensure that more stringent constraints are matched first, and unmatched rules will automatically fall back to the default policy.

[0053] If the condition library contains nested judgment relationships, you can use the rule tree model or decision table structure to build a path optimization index in advance to accelerate condition matching efficiency. In scenarios where rules are frequently updated, the condition library hot loading mechanism can support rule updates without interrupting service.

[0054] Example: In the fintech business, cleaned policy data contains fields such as policy type, policy start date, and payment method. Based on these fields, the system can match the "annuity-type policy with first payment date before 2023" rule in the constraint library and assign a "traditional annuity group" tag to eligible policies for subsequent IFRS 17 measurement implementation.

[0055] In the healthcare business, after cleaning patient data, fields such as diagnosis code, admission department, and medical insurance category are retained. The system can set a rule in the conditional constraint library: "Chronic disease diagnosis code and medical insurance category is resident medical insurance." Records that meet the conditions are marked as "chronic disease group" to support subsequent chronic disease tracking modeling and cost classification.

[0056] This embodiment uses a conditional constraint library and combines data attribute fields to generate grouping tags, enabling fast, flexible, and controllable grouping of data records, improving the accuracy and consistency of the grouping process. This condition-driven approach replaces hard-coded logic, facilitating subsequent rule expansion and maintenance, significantly reducing subjective errors in manual grouping, and enhancing system automation.

[0057] S30, dividing the cleaned data into multiple data subsets according to a preset segmentation strategy and distributing them to multiple processing nodes;

[0058] In this embodiment, the preset segmentation strategy refers to a set of splitting rules formulated before system initialization or operation based on the business data distribution characteristics, field characteristics, or system processing goals. The strategy usually includes segmentation fields, segmentation granularity, segmentation logic methods (such as hash, range, mapping table), and expected data distribution indicators (such as the upper limit of the amount of data processed by each node, field value distribution balance indicators, etc.). The segmentation fields are generally selected from the cleaned data and have good distribution characteristics and stability, such as contract number, customer ID, region code, insurance classification, etc.

[0059] After cleaning, data is divided into multiple data subsets. This means logically grouping or sharding records by designated fields within the cleaned dataset based on a pre-set segmentation strategy. Each group or segment is called a data subset. Data subsets should maintain clear boundaries, non-overlapping data, and cover the entire original data set.

[0060] Distributing data across multiple processing nodes means that after data is partitioned, the system sends each data subset to a specific compute node based on the state of processing resources or scheduling policies. Processing nodes can be local threads, container services, compute processes, or concurrent task units within a distributed computing framework. Each node independently processes its own data subset. This distribution is based on information such as node health, task queue load, and CPU and memory utilization to ensure balanced data processing and avoid overloaded or idle nodes.

[0061] The partitioning and allocation operations are strongly logically coupled. The partitioning logic specified in the partitioning strategy must be compatible with the processing node scheduling mechanism. Matching the partitioning rules for field values ​​with the target number of nodes is essential for ensuring a reasonable data distribution.

[0062] In actual implementation, a segmentation strategy file can be pre-configured, including field names, segmentation algorithms, and allocation mapping tables. After reading the cleaned data, the system first extracts the field values ​​based on the segmentation fields and uses a hash function or segmentation judgment logic to calculate the corresponding subset affiliation.

[0063] When hashing is used for sharding, the field value can be hashed and modulo the total number of processing nodes to determine the target subset to which the record belongs. When range sharding is used, multiple field value range intervals can be configured to divide the records into subsets with corresponding range labels. When mapping table is used, the shard label can be obtained based on the field value lookup table, supporting irregular distribution processing.

[0064] Processing nodes can be dynamically created by the system resource monitoring module based on metrics such as current CPU load, memory usage, and task execution time, or by registering computing tasks based on existing distributed task frameworks (such as Spark, Flink, and MapReduce). During the allocation process, each data subset is encapsulated as a task unit and dispatched to the target node through an intermediate task scheduler or distributed task queue, ensuring a one-to-one correspondence between data subsets and nodes and orderly scheduling.

[0065] If the number of processing nodes or resource status changes, a new partitioning map can be generated by re-evaluating the splitting strategy, and the subset allocation relationship can be dynamically adjusted to support elastic computing resource management.

[0066] Example: In the fintech business field, cleaned policy data is hashed by the policy number field, and all policy records are evenly divided into 16 subsets. These subsets are then assigned to 16 parallel computing containers. Each container independently performs subsequent contract grouping calculations and risk classification tasks, achieving parallel acceleration of contract liability processing.

[0067] In the medical and health business field, the cleaned inpatient records are range-split according to the department code field, and records with different coding ranges such as ICU, cardiology, orthopedics, and pediatrics are divided into multiple subsets and mapped to different professional analysis nodes. Each node uses specific rules to perform patient label modeling and disease clustering analysis, thereby improving the structuring efficiency of medical records and the accuracy of subsequent analysis.

[0068] This embodiment uses a pre-set partitioning strategy to partition data and allocate nodes. This ensures accurate data subset boundaries while improving resource utilization and task scheduling flexibility for multi-node concurrent processing, meeting the requirements of distributed processing of large-scale data. By decoupling partitioning logic from allocation and scheduling, the system achieves greater adaptability and maintainability for diverse data sizes, field characteristics, and computing resources.

[0069] S40, on each processing node, performing sorting processing on the data subset based on the grouping mark and the data attribute field and generating a grouping key;

[0070] In this embodiment, the grouping tag is an identifier generated when each record in the cleaned data is classified according to the conditional constraint rules. It is used to identify the attribution of records in the same category during subsequent processing. The grouping tag can be a string, integer value, composite label, or hash signature, and is derived from the grouping information generated by calling the conditional constraint library in the previous step. This tag not only has the semantic function of grouping attribution but also participates in data sorting and grouping key construction, ensuring the stability and identifiability of records during subsequent processing.

[0071] Data attribute fields refer to the set of structured fields in a record, excluding grouping tags, used for sorting. These fields must possess highly distinguishable and meaningful features for sorting, such as timestamps, amount fields, code fields, event sequence numbers, and version numbers. Attribute fields are selected based on the business scenario, such as contract start time, patient admission time, or transaction sequence number, ensuring comparability and stability.

[0072] Sorting involves rearranging the received data subset at each processing node based on a combined sorting logic based on the grouping tag and one or more data attribute fields, ensuring that records with the same grouping tag have a deterministic order within the local data subset. Sorting can be performed using lexicographic, chronological, or other business-related rules. The sorting algorithm can use quick sort, merge sort, or distributed sorting logic, depending on the processing node's computing power and data structure.

[0073] The grouping key is an identifying field generated after sorting, used to uniquely identify each record's relative sort position or structural characteristics within its group. The grouping key can be a composite field, consisting of a grouping tag and the primary sorting field; it can also be a calculated, compressed representation, such as an MD5 digest or hash value, to ensure unique identification and fast indexing. The generation of the grouping key must be consistent and traceable, ensuring that the same logical group can be uniformly identified across different processing nodes.

[0074] After receiving the corresponding data subset, each processing node first extracts the grouping tags and the preset data attribute field set from the record. The system then calls the sorting function library to jointly sort the data subsets. The sorting function supports multi-field cascading sorting logic, ensuring that the sorting results are interpretable with business semantics.

[0075] After sorting, a grouping key is generated for each record based on its position in the sort sequence and its grouping tag. The grouping tag and sort field can be concatenated into a unique composite key, such as "grouping tag - timestamp," or the concatenated fields can be converted into a fixed-length key using a standard hash function such as SHA-256, reducing storage space and improving processing efficiency.

[0076] To improve system performance and scalability, local memory sorting combined with external caching can be used to process large amounts of data. For situations where data attribute fields may be missing or abnormal, the system configures default sort values ​​or sort priority adjustment strategies to ensure stable sorting.

[0077] In a multi-core computing node, each core can be responsible for a specific range in the subset and merge the sorting results through an intermediate distributed coordination mechanism to ensure the sorting consistency of the overall data subset.

[0078] Example: In the fintech business field, insurance contract data records contain a grouping tag field and a policy effective time field. At each processing node, the system performs sorting based on the combination of "grouping tag + effective time" and generates a grouping key in the "groupId-timestamp" format. This ensures the time order of records in the same insurance group, laying the foundation for subsequent risk assessment modeling and contract liability interval division.

[0079] In the medical and health business field, hospitalized patient data includes patient classification tags and admission time fields. The system sorts by "patient tag + admission time" in each node and generates a unique grouping key for subsequent disease clustering analysis and case trajectory tracking, thereby improving the granularity of data analysis and clustering accuracy.

[0080] This embodiment sorts data subsets based on grouping tags and data attribute fields on processing nodes and generates grouping keys, enabling local data ordering and facilitating efficient group identifier assignment and consistency verification. This operation improves data grouping discrimination and structure recognition accuracy, while reducing the risk of conflicts when merging data across nodes, providing fundamental support for building a distributed, mergeable grouping structure.

[0081] S50, assigning group identifiers to records in the data subset based on the group key and forming a local grouping result table;

[0082] In this embodiment, the grouping key is a structural identifier assigned to records after the pre-order sorting process. It contains at least one key field used to indicate group membership. It is often generated by combining a grouping tag and data attribute fields, and possesses local uniqueness or global identification capabilities. The grouping key's function extends beyond sorting; it serves as the direct basis for subsequent grouping identifier assignment, ensuring the determinism and structural consistency of data grouping.

[0083] The process of assigning group identifiers to records involves determining their logical affiliation within the current processing node based on the grouping key carried by the sorted records. Records with the same grouping key prefix or judgment criteria are grouped together and a unified group identifier is assigned to all records within that group. A group identifier can be a scalar tag, structure, mapping ID, or hash result, and is used to establish group boundaries, group context, or group index within the node.

[0084] The allocation of group identifiers must be completed independently within each processing node, independent of the state information of other nodes, to ensure parallel processing and distributed scalability. This process typically includes four steps: group key resolution, group key classification, group number generation, and identifier binding. A group boundary mapping table is constructed to support the output of the local group result table.

[0085] The local grouping result table is a data structure formed by each processing node based on its own allocation logic. Each record in this structure contains the original data fields, the grouping key generated after sorting, and the newly added grouping identifier field. The local grouping result table serves as an intermediate result and will be used in subsequent global merges and consistency checks.

[0086] The system traverses the sorted data subset within the processing node and reads the grouping key for each record. A temporary group mapping table is maintained, assigning records with the same grouping key to the same group ID. If the grouping key is a composite field, prefix matching or hashing is used to determine group assignment. For string tags, the key value can be used directly as the group identifier.

[0087] To improve processing efficiency, the system uses the hash value of the group key as the initial group identifier, combined with a conflict detection mechanism to ensure that different groups can still be distinguished even in the event of a hash collision. The system can also pre-set an identifier prefix for each processing node to avoid duplication of group identifiers between nodes.

[0088] After the identifiers are assigned, the system updates the group identifier field for each record and encapsulates it into a record structure containing the original fields, sorting information, and identifier information. All records are written to the local group result table in the order they were processed. This table can be stored in a columnar format or a structured file format to support subsequent querying and verification.

[0089] If the processing node has multi-threading capabilities, each thread processes different segmented data and shares a thread-safe group identifier generator to avoid duplicate or skipped identifiers. Optionally, a structure check field or timestamp field can be added to the local group result table for subsequent consistency comparison.

[0090] Example: In the fintech business field, for insurance contract records that have been sorted by contract signing time and risk level, the system assigns identifiers such as "G1001" and "G1002" to each type of record based on the grouping key (for example, "risk group-timestamp") within the node, and adds the identifier to each record, ultimately forming a local grouping result table, which provides a basic grouping unit for subsequent underwriting liability interval comparison.

[0091] In the healthcare business, for sorted inpatient data records, the system uses "diagnosis category-admission time" as the grouping key, and assigns a unified grouping identifier to patient data with the same diagnostic path and treatment cycle, such as "DIA001-A" and "DIA001-B", to form a structured local grouping result table for subsequent disease trajectory analysis and case clustering modeling.

[0092] This embodiment assigns group identifiers to records based on the grouping keys generated by the sorting results at each processing node, and forms a local grouping result table. This clarifies the local grouping of data, reduces the cost of data rearrangement during the global processing phase, and enhances the consistency and comparability of results when merging across multiple nodes. This operation explicitly embeds logical grouping information into record fields, allowing subsequent merges and consistency checks to be directly based on the group identifiers, improving the structural stability and computational efficiency of the overall processing chain.

[0093] S60: Gather the local grouping result tables of all processing nodes and perform consistency check to generate a global grouping result table.

[0094] In this embodiment, the local grouping result table is a structured data set generated independently by a processing node based on a data subset. Records in the table contain the original field information, sorting results, grouping keys, and assigned group identification fields. Because data is distributed across multiple processing nodes, a convergence operation is required to centrally process these distributed local grouping result tables to form a unified data set.

[0095] Aggregation involves collecting the local grouped result tables output by each processing node and consolidating them into a logically merged dataset. This process typically relies on a data dispatching service or result aggregation service for asynchronous pull or active push. Data transmission maintains a consistent format, supporting structure mapping and field alignment.

[0096] Consistency checking involves performing field consistency analysis on a collection of records from different nodes that share the same grouping identifier after aggregation. The primary goal is to verify the consistency of the grouping keys generated by sorting and to determine whether there are data conflicts or group identification discrepancies. Consistency checking primarily focuses on the consistency of grouping key field values ​​and is achieved through hash comparison, range comparison, or data version tag verification.

[0097] If consistency checks pass, the system structurally merges the record sets under the same grouping identifier. The merged results are consistent in field structure and content, and are then consolidated into the global grouping result table. The global grouping result table is a dataset that integrates the results of all processing nodes and passes consistency verification. It is complete, accurate, and comparable across nodes, and serves as the basis for the final grouping structure output.

[0098] The system collects local grouping result tables output by each processing node through a distributed coordinator. Data can be obtained using RPC batch pull or asynchronous push mechanisms based on message queues. After receiving each local grouping result table, it is loaded into a unified intermediate data structure according to field mapping rules, ensuring that the group identification field and group key field maintain consistent structure.

[0099] The system categorizes all loaded records by group identifier field and constructs a mapping table between group identifiers and record sets. For each group identifier, the system extracts the set of group key field values ​​for all records under it. If the group key field values ​​for all records in the set are consistent, the group is considered to have passed the consistency check. If an inconsistency in the group key value is detected, it is marked as a verification failure, and a verification log is optionally generated or the data traceability process is triggered.

[0100] For records that pass the consistency check, the system consolidates them into a standardized set of records and writes them as a logical group to the global grouping result table. The global grouping result table can be constructed as a partitioned table or a dataset with an index structure to support subsequent query, analysis, or export.

[0101] When data volumes are large, a sharding verification mechanism can be introduced. This divides the group identifier into multiple verification fragments based on the hash value, and the verification logic is executed in parallel to improve processing efficiency. The system can also set verification thresholds, such as considering a group as passed if a certain proportion of records are consistent, thereby enhancing fault tolerance.

[0102] Example: In the fintech business area, after insurance contract data is initially grouped by each processing node, the system verifies the contract group marked as "High Risk 2023" after aggregation to ensure that the risk level field and the signing year field of all contract records are consistent. Finally, the grouped contracts are merged into a unified record set and written into the global grouping result table for generating liability reserve model input.

[0103] In the medical and health business area, after the inpatient medical records from different departments are initially grouped, the system summarizes the record set grouped as "Chronic Cardiovascular Disease" and verifies the diagnosis code and treatment plan fields. After confirming the consistency, it constructs a unified structured medical record and generates a global grouping result table for subsequent disease progression path cluster analysis.

[0104] By aggregating the local grouping result tables of all processing nodes and performing consistency checks, this embodiment ensures the global consistency and logical correctness of the final output grouping structure in a distributed processing environment, avoiding grouping conflicts or overlaps caused by rule deviations between nodes or data fragmentation. The resulting global grouping result table is complete and verifiable and can be directly used for downstream modeling, analysis, or audit processing, improving the robustness and credibility of the entire data processing chain.

[0105] The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A data grouping optimization processing method, device, equipment and medium are disclosed, including: collecting input data to generate an initial data set, performing cleaning and format unification processing on the initial data set to generate cleaned data; calling a condition constraint library and generating group tags for records in the cleaned data based on data attribute fields; dividing the cleaned data into multiple data subsets according to a preset segmentation strategy and assigning them to multiple processing nodes; on each processing node, sorting the data subsets based on the group tags and data attribute fields and generating group keys; assigning group identifiers to records in the data subsets based on the group keys and forming a local group result table; aggregating the local group result tables of all processing nodes and performing consistency verification to generate a global group result table. The present invention realizes high-precision group identification by combining data attribute fields with group tags for sorting and key generation, and ensures the accuracy of the global group result through an inter-node consistency verification mechanism; adopts distributed segmentation and parallel processing to improve processing efficiency, and uses the condition constraint library to dynamically adapt complex grouping rules, thereby enhancing the processing stability and scalability of the system in a high-concurrency and multi-rule environment.

[0106] In one embodiment, the above step S10 includes:

[0107] S101, establishing a connection channel with the database system and the file storage system;

[0108] S102, reading original input data through the connection channel;

[0109] S103, loading the original input data into a memory buffer to construct an initial data set;

[0110] S104, identifying and deleting records in the initial data set that are missing key fields;

[0111] S105, converting the date fields in the initial data set into a standard date format;

[0112] S106, converting the numerical fields in the initial data set into numerical values ​​of a unified measurement unit;

[0113] S107, mapping the text classification fields in the initial data set to a preset coding table;

[0114] S108 , the initial data set that has completed deletion of key field missing records, date field conversion, numerical field conversion, and text classification field mapping is used as the cleaned data.

[0115] In this embodiment, establishing a connection channel with a database system and file storage system involves creating a stable data reading channel by configuring access interfaces and authentication parameters. This channel supports access to heterogeneous data sources. Database systems can include relational databases such as MySQL and Oracle, or distributed databases such as HBase and TiDB. File storage systems typically refer to centralized file servers or cloud-based object storage platforms such as HDFS, S3, and OSS. This connection channel supports mechanisms such as access control, timeout settings, and connection pool management to ensure data channel stability in high-concurrency environments.

[0116] Reading raw input data through a connection channel involves the system fetching raw data from connected data sources according to a configured task schedule. This data can be time-segmented tabular data, batch log files, or structured message records. This reading method can use paged scanning, full pull, or incremental recognition strategies, supporting field-level selection and filtering.

[0117] Loading raw input data into a memory buffer to construct an initial dataset involves organizing and temporarily storing the raw data using in-memory structured containers (such as DataFrame and ArrayBuffer). This initial dataset construction process supports automatic data structure identification and unified field name mapping. The buffer supports batch writing and multi-threaded data concatenation to ensure processing efficiency and stability.

[0118] Identifying and deleting records missing key fields from the initial dataset involves applying field integrity rules to determine whether each record contains the minimum set of available fields. If key fields, such as unique identifiers, primary business fields, or timestamps, are missing, the system will logically exclude the associated records. This deletion is typically implemented using filters or automated batch culling through sparse matrix analysis, retaining records with a high degree of field integrity.

[0119] Converting the date fields in the initial dataset to a standard date format involves using a pre-defined time parsing function to identify various date formats (such as "YYYY / MM / DD," "MM-DD-YYYY," and "20240101") and standardizing them to a unified representation, such as ISO 8601. This conversion automatically handles localized time formats and time zone information based on regional settings.

[0120] Converting numeric fields in the initial dataset to uniform units involves identifying multiple unit systems (e.g., "kilogram" versus "gram," "10,000 yuan" versus "yuan") in different fields and applying unit conversion rules to achieve uniform processing. This conversion process involves unit mapping tables, unit factor extraction, a combination of text and numeric parsing, and conversion anomaly detection strategies.

[0121] Mapping text classification fields in the initial dataset to a pre-set coding table involves converting free-text content such as labels and category names into structured enumeration codes. This process is based on pre-built coding dictionaries or standard industry classification tables and supports exact matching, fuzzy matching, and rule replacement modes. It accommodates text differences such as Chinese and English, synonyms, and common typos.

[0122] After all of the above processing is complete, the system marks the data as cleaned, after performing field missing handling, date format conversion, numerical unit conversion, and text classification mapping. This cleaned data possesses structural consistency, semantic standardization, and robust processing, laying the foundation for subsequent operations.

[0123] This embodiment effectively improves data integrity and consistency by building a connection channel to obtain raw input data and performing field missing processing, format conversion, and standard code mapping operations on the initial dataset. The identification and elimination of records with missing fields reduces potential anomalies in the subsequent grouping process, format standardization improves the accuracy of multi-source data alignment, and unit conversion and code mapping enhance the comparability and computability between data fields. This process significantly improves the usability and processing efficiency of cleaned data, providing a semantically consistent data foundation for grouping tag generation and parallel computing in subsequent steps.

[0124] In one embodiment, the above step S20 includes:

[0125] S201, extracting the data attribute field value of each record in the cleaned data;

[0126] S202, querying a condition constraint library according to the data attribute field value to obtain a matching strategy corresponding to the data attribute field value;

[0127] S203, determining the priority scores of the matching strategies, and selecting the matching strategy with the highest priority score as the target strategy;

[0128] S204, detecting whether the data attribute field value meets the matching condition of the target policy;

[0129] S205, when the matching condition is met, generating a grouping tag for the corresponding record according to the target policy;

[0130] S206, when no matching conditions are met, generating a preset default grouping tag for the corresponding record;

[0131] S207: Associating the generated grouping mark or the preset default grouping mark with the corresponding record in the cleaned data.

[0132] In this embodiment, extracting the data attribute field values ​​for each record in the cleansed data involves locating the key attribute fields contained in each structured record retained in the cleansed data and using these field values ​​as the basis for matching. These field values ​​may include contract type, risk rating, issuance date, premium amount range, business line, etc. This operation is typically accomplished through data structure traversal and field parsing mapping functions, supporting batch field extraction and vectorized calculations within a high-performance processing framework.

[0133] Querying the constraint library based on data attribute field values ​​involves using the aforementioned field values ​​as query criteria and accessing the constraint library to obtain the corresponding policy information. The constraint library can be a set of preset rules or a dynamic matching model expressed by a rule engine, containing field value ranges, field combination logic, and corresponding policy numbers. This query process supports fuzzy matching, range search, and combination logic matching, ensuring high adaptability for complex field combinations.

[0134] Determining the priority scores of matching strategies and selecting the matching strategy with the highest priority score as the target strategy involves prioritizing multiple potentially applicable matching strategies and selecting the most suitable target strategy through a scoring mechanism. Priority scores are typically determined based on the strategy source (manual, machine learning, historical experience), the number of matching fields, the strategy's scope of application, and the actual hit frequency. Scoring rules may include weighted scoring functions or rule tree evaluation mechanisms.

[0135] Checking whether data attribute field values ​​meet the target policy's matching conditions involves performing rule validation on the record's data attribute fields based on the rules defined in the target policy to verify whether they fully meet the policy requirements. This process relies on field-level conditional judgments, logical relationship analysis (such as AND and OR nesting), and boundary value restrictions defined in the policy structure. Verification methods such as regular expression matching, numerical comparison, and category mapping can also be introduced.

[0136] When a field value meets the matching criteria, a group tag is generated for the corresponding record according to the target policy. This refers to the group tag template, number, or label specified in the target policy as the group tag for the record. The group tag can be in the form of a structured ID (such as "GRP_A_001"), a label enumeration (such as "High Risk - Floating Return Group"), or a policy mapping value (such as a rule ID conversion value). The group tag generation process can combine strategies such as field value splicing, rule template replacement, and hierarchical tag nesting.

[0137] When no matching criteria are met, a default grouping tag is generated for the corresponding record. This is a general fallback strategy used by the system to cover records not matched by the current rule system, preventing subsequent process interruptions caused by omissions. This default tag is typically "UNASSIGNED," "DEFAULT_GROUP," or another recognizable identifier to facilitate subsequent rule updates or manual review.

[0138] Finally, associating the generated grouping tags or default grouping tags with the corresponding records in the cleansed data involves embedding the generated results as new fields into the original cleansed data structure, completing the structural update. This operation supports in-place writing or structural extension, and precisely binds the unique record identifier to ensure data structure consistency and accuracy of subsequent references.

[0139] This embodiment uses a conditional constraint library and automatically generates grouping tags based on the data attribute fields in the cleaned data, which can achieve efficient adaptation and dynamic strategy matching for complex business rules. This process is driven by field values ​​and combines a multi-strategy priority scoring mechanism and a rule judgment model, which not only improves the accuracy of grouping, but also reduces the cost of manual rule maintenance. At the same time, by setting the default grouping tag, it ensures that all records can fall into a clear group, avoiding omissions or process interruptions, and enhancing the stability and scalability of the system. The entire processing process realizes the automatic association and identification mapping between data records and business logic, providing a controllable and unified structural foundation for subsequent data segmentation, sorting and distribution processing.

[0140] In one embodiment, the above step S30 includes:

[0141] S301, reading the sharding dimension field in the preset sharding strategy;

[0142] S302, performing a grouping operation on the cleaned data according to the sharding dimension field to generate multiple data subsets;

[0143] S303, obtaining current processing resource status information, and creating multiple processing nodes according to the current processing resource status information;

[0144] S304, allocating the multiple data subsets to the multiple processing nodes based on a load balancing strategy;

[0145] S305: Transmit the allocated data subset to the corresponding processing node.

[0146] In this embodiment, reading the sharding dimension fields in the preset sharding strategy means that the system extracts the dimension information used to divide the cleaned data from the sharding strategy configuration previously configured or maintained by the policy generation module. The dimension field can be a single field (such as contract number, insurance type code, region code) or a combination of multiple fields (such as year + product type). These fields determine the logical demarcation basis of the data during the sharding process. The setting of the sharding dimension fields is usually based on data distribution characteristics, business processing logic or resource scheduling model, and can be obtained through configuration files, metadata tables or policy management interfaces during implementation.

[0147] Grouping cleaned data according to the sharding dimension fields to generate multiple data subsets means that the system logically categorizes data records according to the values ​​of the dimension fields, forming several independently processable data sets. This process generally uses hash sharding, range sharding, or label mapping. For example, if the region code is used as the dimension field, the system will group all records from the same region into the same data subset. The grouping process typically relies on the partitioning function or custom sharding function in the data framework, and the implementation supports multi-threaded distribution and the construction of intermediate buffer structures.

[0148] Obtaining current processing resource status information and creating multiple processing nodes based on it means the system dynamically perceives the real-time status of various computing resources (such as CPU utilization, memory usage, I / O bandwidth, node availability, etc.) in the distributed processing environment and determines whether to add new computing nodes based on a resource assessment model. This resource status information is typically collected and analyzed by the resource management module through system monitoring services (such as Prometheus, YARN, and Kubernetes). Node creation can be achieved through dynamic container scheduling, virtual thread derivation, or microservice instance creation.

[0149] Allocating multiple data subsets to multiple processing nodes based on a load balancing strategy involves optimizing the scheduling and allocation of data based on parameters such as the size of the data subsets, the load status of the processing nodes, and resource headroom. Common load balancing strategies include round-robin, weighted allocation, data volume-balanced scheduling, and latency-prediction-based allocation. This process aims to maximize the utilization of existing resources, avoid processing bottlenecks and node congestion, and ensure the stable operation of subsequent parallel processing tasks.

[0150] Transmitting the allocated data subsets to the corresponding processing nodes means pushing each logically divided data subset to the assigned processing node for execution via the network or internal communication channels. The transmission process typically uses batch buffering, sharded transmission, or streaming push mode, and the transmission protocol can be RPC, gRPC, HTTP REST, or asynchronous delivery based on message queues (such as Kafka and RabbitMQ). To ensure data consistency and integrity, the system can also introduce verification mechanisms during the transmission process, such as data summaries, batch identification, and confirmation mechanisms, to ensure that each data subset can be accurately received and subsequently parsed at the target processing node.

[0151] This embodiment realizes structural decoupling before data processing by reading the preset segmentation strategy and logically dividing the cleaned data according to the sharding dimension fields, which facilitates the scheduling and execution of parallel processing. The method of dynamically creating processing nodes in combination with the processing resource status effectively avoids idle or overloaded resources, while improving the responsiveness of the system in elastic scenarios. The use of a load balancing strategy for subset allocation further optimizes resource utilization and task distribution efficiency among multiple nodes, reducing processing delays and data skew risks. Ultimately, through efficient transmission between data subsets and processing nodes, the consistency and controllability of the distributed data processing process are ensured, providing stable input support for subsequent data sorting and grouping operations.

[0152] In one embodiment, the above step S40 includes:

[0153] S401, in each processing node, read all records of the allocated data subset;

[0154] S402, extracting the grouping mark and data attribute field value from each record;

[0155] S403, determining the sorting order using the grouping tag as the primary sorting key and the data attribute field value as the secondary sorting key;

[0156] S404, scheduling multiple parallel sorting threads to perform local sorting operations on the data subset according to the sorting order to generate a sorting result;

[0157] S405 , generating a grouping key in the processing node based on the sorting result, and adding the grouping key to the grouping key field of the corresponding record in the data subset in the processing node.

[0158] In this embodiment, each processing node reads all records of the assigned data subset. This means that after initialization, the node loads the received data into a local computation context, typically managed using a data block or row-based storage structure. During the loading process, these records may be organized into row vector structures or object arrays to facilitate subsequent field operations and batch processing.

[0159] Extracting grouping tags and data attribute field values ​​from each record involves obtaining the logical grouping tag information and field content related to business or computational logic from each local record. Grouping tags typically come from grouping labels generated by condition matching in the previous stage. Data attribute field values ​​may include timestamps, numerical indicators, classification tags, etc., which are used to further distinguish the local ordering relationships between records. The extraction process is based on a field mapping structure and supports dynamic field reading or reflection mechanism calls, as well as rapid positioning through pre-defined data models.

[0160] Using the grouping tag as the primary sort key and the data attribute field value as the secondary sort key to determine the sort order is a multi-level sorting strategy. This ensures that records within the same group are arranged in a stable and orderly manner according to their business attributes. For example, records grouped with the tag "Insurance Type A" will be sorted in ascending order based on their policy start date. If the field value is a continuous numeric value, a double-key comparison function can be used to construct multiple comparators. The sort order is typically defined in a meta-rule file but can also be dynamically injected through the policy configuration interface.

[0161] Scheduling multiple parallel sorting threads to perform local sorting operations on data subsets in sort order and generate sorted results refers to employing a multi-threaded or multi-tasking concurrent execution model within the node, splitting the data subset to be sorted into multiple segments, each processed in parallel by a different sorting thread. Parallel merge sort, parallel quick sort, or a sorting algorithm optimized based on bitmap indexes can be used. A shared cache structure is used when sufficient memory is available, and out-of-block sorting is supported when memory is insufficient. Parallel thread scheduling is dynamically determined based on the number of CPU cores, current node load, or sorting scale to improve sorting efficiency and reduce resource waiting.

[0162] Generating a grouping key within the processing node based on the sorting result and adding the grouping key to the grouping key field of the corresponding record in the data subset means that the system injects a logically unique grouping key into each record based on the position information or attribution sequence of the sorted record. For example, records with the same grouping tag and consecutive sorting are assigned to a key with a consistent serial number, or a structured key is generated through encoding in the format of "grouping tag_grouping serial number". The generated grouping key is used to subsequently identify the ownership of the records and supports operations such as merging, deduplication, and parallel aggregation. The key injection method can be achieved by appending fields, writing metafields, or logical binding through mapping table pointers.

[0163] This embodiment significantly enhances the structural orderliness of the data by performing sorting operations based on grouping tags and data attribute fields on data subsets at each processing node, providing a stable foundation for subsequent group identification generation. The use of a combination of primary and secondary sorting keys ensures the control of group granularity while achieving priority sorting of data within the group, which helps to accurately determine the group boundaries. The introduction of a multi-threaded parallel sorting mechanism improves the processing efficiency of large-scale data sets while ensuring the correctness of sorting. Ultimately, by constructing and injecting grouping keys, each record has a logical affiliation identifier, providing structural support for subsequent parallel merging, aggregation, or consistency verification steps, effectively enhancing the controllability and scalability of multi-node data processing.

[0164] In one embodiment, the above step S50 includes:

[0165] S501, traverse all records in the data subset and read the grouping key of each record;

[0166] S502, assigning a unique group identifier to each unique group key value based on the read group key;

[0167] S503, adding the group identifier to the group identifier field of the corresponding record to generate an updated record;

[0168] S504: Aggregate all updated records to form a local grouping result table.

[0169] In this embodiment, traversing all records in a data subset and reading each record's grouping key involves reading each record produced in the previous sorting phase within the processing node and accessing the contents of its grouping key field. This process typically involves establishing a key-value index in the form of a hash table or ordered array. During this traversal, the system ensures that each record is fully read to avoid skipping or duplicate processing. Null value checks and data type consistency checks are also performed on the grouping key field to ensure the accuracy of subsequent logic.

[0170] Based on the read grouping key, a unique grouping identifier is assigned to each unique grouping key value. This means that a mapping mechanism is built within the current processing node to deduplicate all unique grouping key values ​​that have appeared, usually through a hash deduplication structure or radix sort. A corresponding grouping identifier is then generated for each unique key value. The grouping identifier can be generated using incremental encoding, hash digest, or rule prefix plus numbering. For example, the key value "G001" may be mapped to the identifier "Group_01", ensuring that the identifier is unique within the current processing node and can remain consistent in subsequent global merges.

[0171] Adding a group identifier to the group identifier field of the corresponding record to generate the updated record means that the system updates each record and writes the group identifier mapped from the group key to its structure field. This process may modify the record's data structure, such as appending a field column, or writing the identifier as a field overwrite. Record updates can be performed in batches, using vectorized write techniques or batch caching operations to improve update efficiency. The group identifier field is usually stored as a structured field with the original data, and can also be attached to the record metadata as a logical tag.

[0172] Aggregating all updated records to form a local grouped result table involves reassembling all records with group identifiers into a unified data structure, forming the local output results for the current node. This table typically uses columnar storage or a nested JSON structure to support structured output, subsequent transmission, and consistency verification. The local grouped result table not only contains the original fields, grouping keys, and group identifiers, but can also include sorting information or processing node tags, providing basic support for multi-node aggregation operations.

[0173] This embodiment realizes the logical definition of the local grouping structure by reading the sorted data subset records one by one and identifying their grouping keys. A one-to-one mapping relationship between the grouping key and the grouping identifier is constructed to ensure that each logical grouping has a unique identifier, thereby forming a clear grouping affiliation at the node level. After updating the record structure and injecting the grouping identifier, each record has a clear grouping location attribute, providing a structured basis for subsequent consistency verification, global aggregation and multi-node collaborative processing. Finally, a local grouping result table is formed, which establishes a standard output format for downstream data flow and cross-node logical judgment, effectively improving the grouping accuracy and data organization efficiency in parallel processing scenarios.

[0174] In one embodiment, the above step S60 includes:

[0175] S601, collecting data of the local grouping result table from each processing node;

[0176] S602, classifying the updated records containing the group identifier field in each local grouping result table according to their group identifiers, and constructing a record set corresponding to the group identifier;

[0177] S603, for a set of records with the same grouping identifier, comparing whether the grouping key field values ​​of each record are consistent;

[0178] S604: If the group key field values ​​of the records under the same group identifier are consistent, the record sets under the same group identifier are merged into a unified group structure;

[0179] S605: Aggregate all the records with the same grouping key field values ​​to generate a global grouping result table.

[0180] In this embodiment, collecting data from the local grouping result table of each processing node means that in a distributed system architecture, the scheduling center or data aggregation module centrally transmits the local grouping result table stored on each processing node through a network communication interface, data bus, or remote procedure call protocol. Each local result table typically contains fields such as record structure, group identifier, and group key, and may also include node tag information to support subsequent comparison. During the data collection process, data integrity and format uniformity must be ensured, and collection efficiency can be improved through batch transmission and verification mechanisms.

[0181] Categorizing updated records containing the group identifier field in each local grouping result table by their group identifiers and constructing a set of records corresponding to the group identifiers involves performing a unified classification operation on all collected data. The system uses the group identifier field in the records as the basis for classification, centrally storing records with the same group identifier in a logical set, such as a hash map or key-value index for rapid classification. This process ensures that when multiple nodes may repeatedly process the same logical group, the corresponding records can be identified and managed uniformly.

[0182] After the classification is complete, the system further performs a grouping key consistency check on the set of records with the same grouping identifier, comparing the grouping key field values ​​of each record to ensure consistency. This comparison can be based on a field-by-field comparison of key values, or by constructing a grouping key digest (such as a hash digest) for rapid equivalence. This check is used to identify disagreements between different nodes during the local sorting and grouping process, preventing data conflicts or logical splits. If a conflict exists, an exception flag is triggered or a fallback mechanism is incorporated, without affecting the merging process of the consistent set.

[0183] If the grouping key field values ​​of each record under the same grouping identifier remain consistent, the record sets under the same grouping identifier are merged into a unified grouping structure. This means that after the consistency check passes, the record sets contributed by each node are merged to form a unified logical grouping entry. The merge operation can use structural splicing, deduplication merging, field completion, and other methods to ensure structural integrity and semantic consistency. The generated grouping structure generally includes a list of merged records, a shared grouping key, and the original source node identifier to support subsequent query and analysis.

[0184] Aggregating all records with consistent grouping key field values ​​to generate a global grouping result table involves structurally summarizing all logical groups that have passed consistency checks, generating an output table with a global perspective. This output result table includes not only each logical group and its member records, but also global sorting information, field formatting flags, and integrity check values. The global result table can be stored on a unified data platform or used as input for subsequent processes, such as financial aggregation, indicator calculation, or business modeling.

[0185] In addition, cleaned data and historical processing data can be fed into the learning model. During this process, the system extracts representative structure from the cleaned data generated in the current round, including features such as group label distribution, field value range, and classification label frequency. It also loads historical processing data stored from multiple past processing cycles, specifically including historical input data, its corresponding group labeling results, and the model inference path. This historical data can come from log records, model caches, or external audit result databases, and contains sufficient time series and state differences to provide a wide range of sample support for model training. The learning model can choose different architectures based on the data type and processing objectives, such as decision tree models for classification rule extraction, regression neural network models for constrained threshold fitting, graph neural network models for association rule learning, or learning systems based on reinforcement learning for constrained feedback optimization. Input data must be preprocessed into structured samples, for example, by generating feature vectors or relationship graph representations based on field distribution, with label fields serving as supervisory signals. By training the learning model to dynamically adjust the constraint library, the goal is to automatically discover or adjust rule boundaries based on changes in the distribution of new and old data. The constraint library typically stores multiple matching rules for different data attribute field values, including field combination matching, enumeration value mapping, range threshold setting, and priority sorting. During the model inference phase, the learning model outputs adjustments to the matching rules based on the deviation between the input samples and the labels, such as adding new constraints, removing invalidation policies, and modifying priority parameters. After receiving the model output, the system automatically updates the constraint library structure to ensure it reflects the latest data characteristics in the next round of processing. The entire adjustment process is a closed-loop process, whereby updates to the constraint library feed back into the grouping and labeling process for the next round of cleansed data, forming a self-learning, self-optimizing, and adaptive rule evolution system. This approach significantly differs from statically configured constraint library structures and can adapt to dynamic changes in business logic and data environments. By feeding cleansed data and historical processing data into the learning model, the system can automatically discover the underlying correlations between data distribution and processing results, enabling precise and timely adjustments and optimizations to the constraint library rule system. This mechanism effectively avoids problems such as outdated rules, delayed manual maintenance, and inaccurate boundary determination, improving the system's adaptability and responsiveness to evolving data structures or business logic updates. Furthermore, this structure incorporates model-driven feedback updates, forming a steady-state evolution mechanism that makes condition matching results more stable and generalizable, significantly enhancing accuracy and system autonomy during multi-cycle processing.

[0186] Example: In the fintech sector, an insurance company needs to group policy contract data from a new cycle to meet the liability grouping requirements under IFRS 17 reporting. This data processing task is scheduled and executed by an automated system. First, the system establishes a connection between the database system and the archiving file system to read the original policy contract data, including policyholder information, policy type, payment frequency, expected claim amount, and coverage period. The system loads the original data into a memory buffer to construct an initial dataset and deletes records with missing key fields (such as policy type code or policy period). Subsequently, date fields with inconsistent formats across regions are converted to a unified ISO format, amount fields are uniformly converted to "yuan," and text-based policy name fields are mapped to a coding table (for example, accident insurance is mapped to 101 and critical illness insurance is mapped to 102). After completing these operations, the system generates cleaned data. Next, the system extracts attribute fields from each record in the cleaned data, such as payment method, policy type code, and customer risk level, and matches them against the grouping rules stored in the constraint library. For example, a rule defines that contracts with "monthly payment method, risk level A, and critical illness insurance" should be grouped together. The system selects the most appropriate strategy based on the matching priority and generates the corresponding grouping tags; if no matching strategy exists, the default tag is used. After the data is tagged, the system performs grouping and segmentation based on the two dimensions of "insurance type code" and "customer risk level," dividing the data into multiple subsets. Simultaneously, the system retrieves the node load status on the current cloud platform, initializes multiple processing nodes based on memory capacity and concurrent threads, and distributes subset tasks to each node using a weighted round-robin approach. Within each node, the system performs local sorting of the data subset based on the grouping tags and attribute fields, for example, prioritizing ascending order by grouping tag, followed by descending order by expected claim amount. After sorting, the system constructs a grouping key for each data set, which serves as the logical basis for record attribution. The system then assigns a grouping identifier to each record, ensuring that records with the same grouping key receive the same identifier. A local grouping result table is generated within each node. After multiple nodes generate results, they are aggregated into the master control service module. The system categorizes all records based on group identifiers and performs a consistency check to determine whether the group key fields under each group identifier are consistent. For groups that meet consistency, the system merges them into a unified group structure and outputs a global group result table for downstream liability reserve provisioning, profit attribution accounting, and risk allocation analysis.

[0187] In the healthcare sector, a regional hospital management platform needs to group and process chronic disease management data uploaded by different community health centers to assist in analyzing patient management burden, resource allocation efficiency, and long-term intervention effectiveness. The system first establishes a connection between the medical data platform and the local health record repository, extracting daily follow-up data and historical treatment data for patients with hypertension, diabetes, COPD, and other conditions. The raw input data is loaded into memory to form an initial dataset, removing records with missing patient identifiers or diagnosis dates. The system further standardizes time fields to UTC format, converts blood pressure values ​​to mmHg, and normalizes BMI using a unified formula. Disease labels are also mapped to standard disease codes, such as E11 for diabetes. From this cleaned, structured data, the system extracts fields such as patient age, disease type, and frequency of the last three follow-up visits, and matches them against a multidimensional health risk model in the constraint library. For example, if a patient is over 65 years old, has both diabetes and hypertension, and has had two or more follow-up visits in the past 30 days, they are assigned a "high intervention priority group" based on the policy. Otherwise, they are assigned a different label based on the suboptimal policy. The system performs segmentation based on the "disease type + risk level" strategy to generate multiple data subsets. At the same time, it monitors the operating status of the downstream resource nodes of the platform, dynamically initializes multiple distribution nodes, and uses an on-demand capacity forecasting strategy to assign subset tasks to each processing node. Each node performs local sorting based on grouping tags and fields such as "number of days since the last hospitalization". After the sorting is completed, a grouping key is generated and added to the record. The system generates a corresponding unique identifier for each different grouping key and constructs a local grouping result table. After all processing nodes complete their tasks, the system transmits the results back to the central node and classifies them uniformly. For the record set under the same grouping identifier, a consistency check of the grouping key field is performed (for example, the main chronic disease category and management cycle of patients in the same group must be consistent). The data that passes the check is integrated into a global grouping result table for subsequent applications such as the formulation of chronic disease stratified intervention plans, the generation of drug distribution strategies, and the overall evaluation of medical insurance resources.

[0188] This embodiment implements a unified data merging mechanism across nodes by aggregating the local result tables of each processing node and constructing a record set corresponding to the group identifier. Further consistency checks are performed on the group key fields in the record set, effectively eliminating grouping logic conflicts caused by node differences, thereby ensuring the structural consistency and semantic unity of each logical grouping from a global perspective. The resulting global grouping result table is complete, consistent, and traceable, significantly improving data aggregation accuracy and system reliability in large-scale distributed processing scenarios.

[0189] In one embodiment, a data group optimization processing device is provided, which corresponds to the data group optimization processing method in the above embodiment. Figure 3 , Figure 3This is a functional module diagram of a preferred embodiment of the data grouping optimization processing device of the present invention. It includes a data acquisition and preprocessing module 10, a grouping tag generation module 20, a data segmentation and scheduling module 30, a grouping key generation module 40, a local grouping construction module 50, and a global grouping aggregation module 60. Each functional module is described in detail below:

[0190] The data collection and preprocessing module 10 is used to collect input data to generate an initial data set, and perform cleaning and format unification processing on the initial data set to generate cleaned data;

[0191] A grouping tag generating module 20 is used to call a condition constraint library and generate a corresponding grouping tag for each record in the cleaned data based on a data attribute field;

[0192] A data segmentation scheduling module 30 is used to divide the cleaned data into multiple data subsets and distribute them to multiple processing nodes according to a preset segmentation strategy;

[0193] A grouping key generation module 40 is configured to perform sorting processing on the data subset based on the grouping mark and the data attribute field and generate a grouping key on each processing node;

[0194] A local grouping construction module 50 is configured to assign grouping identifiers to records in the data subset based on the grouping key and form a local grouping result table;

[0195] The global grouping aggregation module 60 is used to aggregate the local grouping result tables of all processing nodes and perform consistency check to generate a global grouping result table.

[0196] In one embodiment, the data acquisition and preprocessing module 10 is specifically configured to:

[0197] Establish connection channels with database systems and file storage systems;

[0198] Reading original input data through the connection channel;

[0199] Loading the original input data into a memory buffer to construct an initial data set;

[0200] Identifying and deleting records in the initial dataset that are missing key fields;

[0201] Convert the date fields in the initial data set into a standard date format;

[0202] Converting the numerical fields in the initial data set into numerical values ​​of a unified measurement unit;

[0203] Mapping the text classification fields in the initial data set to a preset coding table;

[0204] The initial data set after deletion of missing records of key fields, date field conversion, numerical field conversion and text classification field mapping is used as the cleaned data.

[0205] In one embodiment, the grouping tag generating module 20 is specifically configured to:

[0206] Extracting the data attribute field value of each record in the cleaned data;

[0207] Querying a condition constraint library according to the data attribute field value to obtain a matching strategy corresponding to the data attribute field value;

[0208] Determining the priority scores of the matching strategies, and selecting the matching strategy with the highest priority score as the target strategy;

[0209] Detecting whether the data attribute field value meets the matching condition of the target policy;

[0210] When the matching conditions are met, a grouping tag is generated for the corresponding record according to the target strategy;

[0211] When no matching conditions are met, a preset default grouping tag is generated for the corresponding record;

[0212] The generated grouping mark or the preset default grouping mark is associated with the corresponding record in the cleaned data.

[0213] In one embodiment, the data segmentation scheduling module 30 is specifically configured to:

[0214] Read the sharding dimension field in the preset sharding strategy;

[0215] performing a grouping operation on the cleaned data according to the sharding dimension field to generate multiple data subsets;

[0216] Acquiring current processing resource state information, and creating multiple processing nodes according to the current processing resource state information;

[0217] allocating the plurality of data subsets to the plurality of processing nodes based on a load balancing strategy;

[0218] The allocated data subsets are transferred to the corresponding processing nodes.

[0219] In one embodiment, the group key generation module 40 is specifically configured to:

[0220] In each processing node, read all records of the assigned data subset;

[0221] Extract grouping tags and data attribute field values ​​from each record;

[0222] The sorting order is determined by using the grouping mark as the primary sorting key and the data attribute field value as the secondary sorting key;

[0223] Scheduling multiple parallel sorting threads to perform local sorting operations on the data subsets according to the sorting order to generate sorting results;

[0224] A grouping key in the processing node is generated based on the sorting result, and the grouping key is added to the grouping key field of the corresponding record in the data subset in the processing node.

[0225] In one embodiment, the local grouping building module 50 is specifically configured to:

[0226] Traverse all records in the data subset and read the grouping key of each record;

[0227] Based on the read grouping key, a unique grouping identifier is assigned to each unique grouping key value;

[0228] Adding the group identifier to the group identifier field of the corresponding record to generate an updated record;

[0229] Aggregate all updated records to form a local grouped result table.

[0230] In one embodiment, the global group aggregation module 60 is specifically configured to:

[0231] Collect data from the local grouping result table from each processing node;

[0232] Classify the updated records containing the group identifier field in each local group result table according to their group identifiers, and construct a record set corresponding to the group identifier;

[0233] For a set of records with the same grouping identifier, compare whether the grouping key field values ​​of each record are consistent;

[0234] If the group key field values ​​of each record under the same group identifier are consistent, the record sets under the same group identifier are merged into a unified group structure;

[0235] Aggregate all records with consistent grouping key field values ​​to generate a global grouping result table.

[0236] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a data packet optimization processing method.

[0237] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data packet optimization processing method.

[0238] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0239] Collecting input data to generate an initial data set, and performing cleaning and formatting processing on the initial data set to generate cleaned data;

[0240] Calling a condition constraint library and generating a corresponding grouping tag for each record in the cleaned data based on the data attribute field;

[0241] Dividing the cleaned data into multiple data subsets according to a preset segmentation strategy and distributing them to multiple processing nodes;

[0242] On each processing node, sorting the data subset based on the grouping tag and the data attribute field and generating a grouping key;

[0243] Based on the grouping key, assigning group identifiers to the records in the data subset and forming a local grouping result table;

[0244] The local grouping result tables of all processing nodes are aggregated and consistency checked to generate a global grouping result table.

[0245] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0246] Collecting input data to generate an initial data set, and performing cleaning and formatting processing on the initial data set to generate cleaned data;

[0247] Calling a condition constraint library and generating a corresponding grouping tag for each record in the cleaned data based on the data attribute field;

[0248] Dividing the cleaned data into multiple data subsets according to a preset segmentation strategy and distributing them to multiple processing nodes;

[0249] On each processing node, sorting the data subset based on the grouping tag and the data attribute field and generating a grouping key;

[0250] Based on the grouping key, assigning group identifiers to the records in the data subset and forming a local grouping result table;

[0251] The local grouping result tables of all processing nodes are aggregated and consistency checked to generate a global grouping result table.

[0252] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0253] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0254] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0255] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data grouping optimization processing method, characterized in that: The following steps are involved: Collecting input data to generate an initial data set, and performing cleaning and formatting processing on the initial data set to generate cleaned data; Calling a condition constraint library and generating a corresponding grouping tag for each record in the cleaned data based on the data attribute field; Dividing the cleaned data into multiple data subsets according to a preset segmentation strategy and distributing them to multiple processing nodes; On each processing node, sorting the data subset based on the grouping tag and the data attribute field and generating a grouping key; Based on the grouping key, assigning group identifiers to the records in the data subset and forming a local grouping result table; The local grouping result tables of all processing nodes are aggregated and consistency checked to generate a global grouping result table.

2. The data grouping optimization processing method according to claim 1, wherein: Collect input data to generate an initial data set, and perform cleaning and formatting on the initial data set to generate cleaned data, including: Establish connection channels with database systems and file storage systems; Reading original input data through the connection channel; Loading the original input data into a memory buffer to construct an initial data set; Identifying and deleting records in the initial dataset that are missing key fields; Convert the date fields in the initial data set into a standard date format; Converting the numerical fields in the initial data set into numerical values ​​of a unified measurement unit; Mapping the text classification fields in the initial data set to a preset coding table; The initial data set after deletion of missing records of key fields, date field conversion, numerical field conversion and text classification field mapping is used as the cleaned data.

3. The data grouping optimization processing method according to claim 1, wherein: Call the condition constraint library and generate corresponding grouping tags for each record in the cleaned data based on the data attribute fields, including: Extracting the data attribute field value of each record in the cleaned data; Querying a condition constraint library according to the data attribute field value to obtain a matching strategy corresponding to the data attribute field value; Determining the priority scores of the matching strategies, and selecting the matching strategy with the highest priority score as the target strategy; Detecting whether the data attribute field value meets the matching condition of the target policy; When the matching condition is met, a grouping tag is generated for the corresponding record according to the target strategy; When no matching conditions are met, a preset default grouping tag is generated for the corresponding record; The generated grouping mark or the preset default grouping mark is associated with the corresponding record in the cleaned data.

4. The data grouping optimization processing method according to claim 1, wherein: The cleaned data is divided into multiple data subsets according to a preset segmentation strategy and distributed to multiple processing nodes, including: Read the sharding dimension field in the preset sharding strategy; performing a grouping operation on the cleaned data according to the sharding dimension field to generate multiple data subsets; Acquiring current processing resource state information, and creating multiple processing nodes according to the current processing resource state information; allocating the plurality of data subsets to the plurality of processing nodes based on a load balancing strategy; The allocated data subsets are transferred to the corresponding processing nodes.

5. The data grouping optimization processing method according to claim 1, wherein: On each processing node, sorting is performed on the data subset based on the grouping tag and the data attribute field and a grouping key is generated, including: In each processing node, all records of the assigned data subset are read; Extract grouping tags and data attribute field values ​​from each record; The sorting order is determined by using the grouping mark as the primary sorting key and the data attribute field value as the secondary sorting key; Scheduling multiple parallel sorting threads to perform local sorting operations on the data subsets according to the sorting order to generate sorting results; A grouping key in the processing node is generated based on the sorting result, and the grouping key is added to the grouping key field of the corresponding record in the data subset in the processing node.

6. The data grouping optimization processing method according to claim 1, wherein: Based on the grouping key, a grouping identifier is assigned to the records in the data subset and a local grouping result table is formed, including: Traverse all records in the data subset and read the grouping key of each record; Based on the read grouping key, a unique grouping identifier is assigned to each unique grouping key value; Adding the group identifier to the group identifier field of the corresponding record to generate an updated record; Aggregate all updated records to form a local grouped result table.

7. The data grouping optimization processing method according to claim 1, wherein: Aggregate the local grouping result tables of all processing nodes and perform consistency checks to generate a global grouping result table, including: Collect data from the local grouping result table from each processing node; Classify the updated records containing the group identifier field in each local group result table according to their group identifiers, and construct a record set corresponding to the group identifier; For a set of records with the same grouping identifier, compare whether the grouping key field values ​​of each record are consistent; If the group key field values ​​of each record under the same group identifier are consistent, the record sets under the same group identifier are merged into a unified group structure; Aggregate all records with consistent grouping key field values ​​to generate a global grouping result table.

8. A data group optimization processing device, characterized in that: The data group optimization processing device includes: The data acquisition and preprocessing module is used to collect input data to generate an initial data set, and perform cleaning and format unification processing on the initial data set to generate cleaned data; A grouping tag generation module, configured to call a conditional constraint library and generate a corresponding grouping tag for each record in the cleaned data based on a data attribute field; A data segmentation scheduling module is used to divide the cleaned data into multiple data subsets and distribute them to multiple processing nodes according to a preset segmentation strategy; A grouping key generation module is used to perform sorting processing on the data subset based on the grouping mark and the data attribute field and generate a grouping key on each processing node; A local grouping construction module, configured to assign group identifiers to records in the data subset based on the grouping key and form a local grouping result table; The global grouping aggregation module is used to aggregate the local grouping result tables of all processing nodes and perform consistency checks to generate a global grouping result table.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data group optimization processing program stored in the memory and capable of running on the processor. When the data group optimization processing program is executed by the processor, the steps of the data group optimization processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a data grouping optimization processing program, which, when executed by a processor, implements the steps of the data grouping optimization processing method according to any one of claims 1 to 7.

Images