A multi-source policy data processing method and system

Abstract

This invention discloses a method and system for processing multi-source insurance policy data, belonging to the field of insurance data processing technology. It acquires a master table of policy data synchronously formed from multiple data sources, as well as dimension fields and preset partition thresholds for splitting the master table. Based on the dimension fields, it calculates the data volume of each dimension combination and generates an initial splitting task. For each dimension combination, it obtains isolated peak values ​​and resource co-injection values ​​to determine the overall anomaly level and classifies it into three types: normal splitting, mild concentration, and severe concentration. For normal splitting, the initial splitting task is retained; for mild concentration, a monitoring flag is set; and for severe concentration, a dynamic auxiliary splitting field is added. After regenerating the target partition identifier and the execution range of the splitting task, partitioning processing is performed. This invention can reduce the continuous write pressure on a single target resource from abnormally concentrated dimension combinations, improving the efficiency and stability of multi-source insurance policy data partitioning processing.

Description

Technical Field

[0001] This invention relates to the field of insurance data processing technology, specifically to a method and system for processing multi-source insurance policy data. Background Technology

[0002] With the development of online insurance business and multi-channel operations, policy data typically originates from multiple data sources, including core underwriting systems, sales channel systems, institutional business systems, policy maintenance systems, payment systems, and third-party service platforms. Because different data sources differ in data structure, field meaning, update cycles, and data formats, existing technologies usually first synchronize policy data from multiple data sources to a target database to form a unified master policy data table. Then, based on preset dimension fields such as time, institution, insurance type, channel, or policy status, the master policy data table is statistically analyzed and split to facilitate subsequent monthly closing processing, report generation, business queries, or regulatory data reporting.

[0003] Existing methods for processing multi-source policy data typically calculate the data volume corresponding to each dimension combination based on fixed dimension fields and generate splitting tasks according to preset partition thresholds. For example, when the data volume under a certain dimension combination exceeds the preset threshold, the system splits the dimension combination into multiple tasks according to a fixed range of data entries and writes the corresponding data to the target partition. However, such methods usually only use the number of data entries as the basis for task generation, failing to further identify the differences in data distribution between different dimension combinations, and also failing to consider whether the splitting tasks are concentrated in the same partition, the same index, the same storage resource, or the same write channel in the target database.

[0004] Therefore, in scenarios such as centralized marketing, batch renewals, historical data supplementation, or peak sales through internet channels, the combination of dimensions corresponding to certain months, institutions, insurance types, or channels may form abnormally concentrated large-scale data. Even if existing technologies split this data into multiple tasks according to preset thresholds, a large number of tasks may still access the same source data range or write to the same target resource, causing lock waits, index hotspots, log write congestion, and disk I / O bottlenecks. This can slow down or even block the entire policy data splitting batch, affecting the efficiency of subsequent month-end closing, querying, and reporting. Therefore, there is an urgent need for a multi-source policy data processing method that can identify the degree of abnormal concentration of dimension combinations before generating splitting tasks and dynamically adjust for abnormal concentration situations. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for processing multi-source insurance policy data to address the shortcomings in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-source policy data processing method, comprising: Obtain the general policy data table generated by synchronizing multiple data sources, and obtain the dimension fields and preset partition thresholds used to split the general policy data table; The data volume corresponding to each dimension combination in the policy data table is calculated based on the dimension fields, and the initial splitting task for each dimension combination is generated based on the data volume and the preset partition threshold. For each dimension combination, obtain the dimension solitary peak value, which represents the degree of data abruptness relative to similar dimension combinations, and the resource co-infusion value, which represents the degree to which the initial split task set falls into the same target resource. The overall anomaly level of the corresponding dimension combination is determined based on the isolated peak value of the dimension and the co-injection value of resources, and the dimension combination is divided into three types according to the overall anomaly level: normal split, mild concentration and severe concentration. For normal splitting types, the initial splitting task is retained; for lightly centralized types, the initial splitting task is retained and a monitoring flag is set; for heavily centralized types, a dynamic auxiliary splitting field is added, the target partition identifier and the execution range of the splitting task are regenerated, and the policy data master table is partitioned according to the regenerated splitting task.

[0007] Preferably, a master policy data table generated by synchronizing multiple data sources is obtained, along with dimension fields and preset partition thresholds used to split the master policy data table, including: Read the current synchronization batch number, obtain the policy data summary table after synchronizing multiple data sources to the target database based on the current synchronization batch number, and compare the sum of the number of records synchronized from each data source with the corresponding number of records in the policy data summary table; if they match, read the enabled dimension fields and their dimension order from the dimension configuration table, and read the preset partition threshold that matches the current synchronization batch number from the partition threshold configuration table; associate and store the dimension fields, dimension order, and preset partition threshold with the current synchronization batch number for later use when calculating the data volume of dimension combinations.

[0008] Preferably, the data volume corresponding to each dimension combination in the policy data master table is calculated based on the dimension fields, and an initial splitting task for each dimension combination is generated based on the data volume and a preset partition threshold, including: Extract the dimension field values ​​corresponding to each policy record in dimensional order, and combine each dimension field value with a field length value and a fixed separator character to form a dimension combination identifier, so that policy records with the same dimension field value are grouped into the same dimension combination identifier; for policy records under the same dimension combination identifier, count them in a fixed order according to the policy number, and record the current policy number as the task boundary value when the count reaches an integer multiple of the preset partition threshold; form the initial split range based on the task boundary value, and write the dimension combination identifier, target partition identifier, starting task boundary value, ending task boundary value and current synchronization batch number into the corresponding initial split task.

[0009] Preferably, the determination of the fixed sequence of policy numbers includes: The policy numbers under the same dimension combination identifier are converted into a unified character code, and the order of the policy numbers is determined by comparing each character code digit by digit. When two policy numbers have the same character code in the common position, the policy number with the shorter character length will be listed first. Record task boundary values ​​in a fixed order according to the determined policy number, so that the same initial split task is still limited by the same starting task boundary value and ending task boundary value when it is re-executed.

[0010] Preferably, obtaining dimensional isolated peaks includes: For the current dimension combination, retain one dimension field value in sequence as a reference condition, and filter out the same type of dimension combination with the same reference condition from the data volume of each dimension combination; Remove the current dimension combination from the same dimension combination, and arrange them in ascending order of data volume, selecting the data volume in the middle order and the data volume in the later order as the basic reference value and the high-order reference value respectively; The data volume of the current dimension combination is compared with the basic reference volume and the high-order reference volume, and combined with the number of initial splitting tasks corresponding to the current dimension combination, the dimensional peak value of the current dimension combination is determined.

[0011] Preferably, obtaining resource irrigation values ​​includes: Read the target partition identifiers in each initial splitting task corresponding to the current dimension combination, and determine the target resource identifiers that each initial splitting task actually points to based on the target partition identifiers to obtain the target resource landing point sequence; Merge identical target resource identifiers in the target resource landing sequence and determine the target resource identifier that carries the most initial splitting tasks as the main injection target resource identifier; Based on the order of the starting task boundary value and the ending task boundary value, determine the continuous distribution of the initial splitting tasks pointing to the main infusion target resource identifier, and determine the resource co-infusion value by combining the number of initial splitting tasks corresponding to the main infusion target resource identifier and the total number of initial splitting tasks in the current dimension combination.

[0012] Preferably, the comprehensive anomaly degree of the corresponding dimension combination is determined based on the isolated peak value of the dimension and the resource co-injection value, including: Map the isolated peak value of the dimension to the data surge evaluation level, map the resource co-irrigation value to the resource co-irrigation evaluation level, and convert the data surge evaluation level and the resource co-irrigation evaluation level into the corresponding level number respectively. The higher of the two gear numbers is used as the candidate comprehensive gear number, and the lower of the two gear numbers is used as the mutual verification gear number. The comprehensive abnormal gear number is determined based on the gear difference between the candidate comprehensive gear number and the mutual verification gear number. When the difference in rating levels indicates that two rating levels are not adjacent, the candidate comprehensive rating level number is lowered and used as the comprehensive abnormal rating level number to reduce severe concentrated misjudgments caused by anomalies in a single direction.

[0013] Preferably, based on the degree of overall anomaly, the dimensional combinations are divided into three types: normal splitting, mild concentration, and severe concentration, including: When the comprehensive abnormal level number is a low level, the current dimension combination is classified as a normal split type, and its initial split task is retained. When the comprehensive anomaly level number is the median level, the current dimension combination is classified as a slightly concentrated type, and a monitoring tag associated with the dimension combination identifier, target partition identifier, and task boundary value is written in its initial split task. When the comprehensive anomaly level number is a high level, the current dimension combination is classified as a heavily concentrated type, and the dynamic auxiliary splitting field determination process is triggered.

[0014] Preferably, for heavily concentrated data types, a dynamic auxiliary splitting field is added to regenerate the target partition identifier and the execution scope of the splitting task, including: Read all policy numbers under the corresponding dimension combination of the heavily concentrated type, and extract candidate character segments with the same character position according to the fixed order of the policy numbers. Determine the dynamic auxiliary splitting field based on the number of different values ​​of the candidate character segments and the number of policy records corresponding to each value. Add the dynamic auxiliary splitting field to the original dimension combination identifier to obtain the reconstructed dimension combination identifier. Then, redetermine the target partition identifier based on the reconstructed dimension combination identifier. Redefine the task boundary values ​​according to the fixed order of the policy numbers corresponding to the reconstructed dimension combination identifier to form a reconstructed splitting task. Then, perform partitioning processing on the policy data master table based on the reconstructed splitting task.

[0015] This invention also provides a multi-source policy data processing system, comprising: The data acquisition module acquires a master table of policy data generated by synchronizing multiple data sources, and acquires the dimension fields and preset partition thresholds used to split the master table of policy data. The initial task generation module calculates the data volume corresponding to each dimension combination in the policy data master table based on the dimension fields, and generates the initial splitting task for each dimension combination based on the data volume and the preset partition threshold. The abnormal value acquisition module acquires, for each dimension combination, the dimensional isolated peak value, which represents the degree of data abrupt increase relative to similar dimension combinations, and the resource co-infusion value, which represents the degree to which the initial split task set falls into the same target resource. The anomaly type classification module determines the comprehensive anomaly degree of the corresponding dimension combination based on the isolated peak value of the dimension and the co-injection value of resources, and classifies the dimension combination into three types: normal split, mild concentration, and severe concentration based on the comprehensive anomaly degree. The partitioning task adjustment module retains the initial partitioning task for normal partitioning types; retains the initial partitioning task and sets a monitoring flag for lightly concentrated types; and adds a dynamic auxiliary partitioning field for heavily concentrated types, regenerates the target partition identifier and the execution range of the partitioning task, and partitions the policy data master table according to the regenerated partitioning task.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention differs from existing technologies that mechanically generate splitting tasks based solely on fixed dimension fields and preset partition thresholds. Its core technical approach lies in: after generating the initial splitting tasks, further acquiring isolated peak values ​​for dimensions and resource co-flow values, identifying abnormal dimension combinations from two directions: the degree of data surge and the degree of concentrated inclusion of target resources. This allows for the detection of a few dimension combinations with abnormally concentrated data volumes and tasks concentrated on the same target resource before partitioning, avoiding the problem of existing technologies still pushing a large number of tasks onto the same partition, index, or storage resource in data skew scenarios, thus reducing the risks of lock waiting, write congestion, and task blocking from the source.

[0017] This invention further categorizes dimensional combinations into three types based on the degree of overall anomaly: normal splitting, mild concentration, and severe concentration. Only for the severe concentration type, a dynamic auxiliary splitting field is added to regenerate the target partition identifier and the execution scope of the splitting task. This approach does not simply increase the number of tasks; instead, it redistributes policy records originally concentrated in a single dimensional combination and a single target resource across multiple reconstructed dimensional combinations and target partitions. Normal data maintains its original path, mildly concentrated data is marked with monitoring tags, and severely concentrated data is dynamically diverted. This ensures consistency in policy splitting criteria while improving the concurrency stability of partition processing and shortening batch processing time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is a flowchart of the system modules of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1, please refer to Figure 1 As shown in this embodiment, a multi-source policy data processing method includes: Obtain the general policy data table generated by synchronizing multiple data sources, and obtain the dimension fields and preset partition thresholds used to split the general policy data table.

[0023] Before performing policy data splitting, the server first reads the master policy data table from the target database. This master table is generated by synchronizing multiple data sources, including at least two of the following: underwriting data source, policy maintenance data source, premium payment data source, claims data source, channel sales data source, and institutional business data source. Each data source carries a data source identifier, synchronization batch number, synchronization time, business date, and data version number during synchronization. Based on the synchronization batch number, the server writes the synchronized policy records within the same batch into the same master policy data table, ensuring that subsequent splitting processing is performed based on the same data batch.

[0024] The policy data master table includes at least the following fields: policy number, institution code, insurance type code, channel code, policy status, effective date, premium amount, data source identifier, synchronization batch number, and update time. When retrieving the policy data master table, the server first reads the current batch number to be processed, and then queries the policy data master table in the target database based on the current batch number to obtain the range of data to be split. The range of data to be split is jointly defined by the synchronization batch number and the synchronization completion identifier. When the synchronization completion identifier is "completed," it means that all data sources participating in this batch have completed synchronization; when the synchronization completion identifier is "incomplete," the subsequent dimension field retrieval steps are not performed.

[0025] To avoid inconsistencies in field names across different data sources that could lead to inconsistent splitting methods, the server performs field standardization before writing data to the master policy data table. Field standardization is accomplished using a field mapping table, which includes the source field name, standard field name, field type, length limit, and conversion rules. For example, "policyNo" from the channel sales data source and "policy_id" from the underwriting data source are both mapped to the standard field "policy number"; date fields are written in "yyyyMMdd" format; and amount fields are written as integers in cents. The server transforms each data source record according to the field mapping table and writes the transformed records to the master policy data table.

[0026] After obtaining the master policy data table, the server performs a data integrity check on it. Let n be the number of data sources participating in the current batch synchronization, Si be the number of records synchronized from the i-th data source, and D be the number of records corresponding to the current batch in the master policy data table. The server calculates the record consistency value of the master table as follows: first, calculate the sum of the record counts from S1 to Sn; then compare the record count D corresponding to the current batch in the master policy data table with this sum. When D equals the sum of the record counts from S1 to Sn, the record consistency value is 1; when D does not equal the sum of the record counts from S1 to Sn, the record consistency value is 0. When the record consistency value is 1, the server confirms that the master policy data table meets the preconditions for splitting; when the record consistency value is 0, the server records the abnormal batch number, the number of differing records, and the identifier of the differing data source, and stops the splitting process for the current batch.

[0027] After confirming that the master policy data table meets the preconditions for splitting, the server retrieves the dimension fields used to split the master policy data table. Dimension fields are read from the dimension configuration table, which includes at least the dimension field identifier, standard field name, dimension type, dimension order, activation identifier, and dimension value rules. Dimension fields include at least one of the following: time dimension field, organization dimension field, insurance type dimension field, channel dimension field, and policy status dimension field. The time dimension field corresponds to the effective date or business date in the master policy data table; the organization dimension field corresponds to the organization code; the insurance type dimension field corresponds to the insurance type code; the channel dimension field corresponds to the channel code; and the policy status dimension field corresponds to the policy status.

[0028] The server filters valid records in the dimension configuration table according to the enabled identifier and determines the dimension combination order during splitting based on the dimension order. For example, if the dimension order is time dimension, institution dimension, insurance type dimension, and policy status dimension, the server will subsequently perform data volume analysis according to the combination of "time dimension value, institution code, insurance type code, and policy status". When a dimension field contains null values, the server processes them according to the dimension value retrieval rules; if the dimension value retrieval rule is to fill with default values, null values ​​are converted to the fixed value "UNKNOWN"; if the dimension value retrieval rule is to remove anomalies, policy records containing null values ​​are written to the anomaly record table and do not participate in the current batch splitting.

[0029] The server further retrieves the preset partition threshold. The preset partition threshold is read from the partition threshold configuration table, which includes at least a threshold identifier, applicable batch type, applicable dimension type, single-task record limit, single-task byte limit, and enable identifier. The single-task record limit restricts the number of policy records that a splitting task can process, and the single-task byte limit restricts the number of data bytes that a splitting task can process. If the current batch type matches the applicable batch type in the partition threshold configuration table, and the enable identifier is enabled, the server reads the corresponding single-task record limit as the preset partition threshold. The preset partition threshold is represented by a positive integer, for example, 10000, indicating that the maximum number of policy records corresponding to an initial splitting task is 10000.

[0030] When the partition threshold configuration table configures both the single-task record limit and the single-task byte limit, the server calculates the effective partition threshold. Let the single-task record limit be R, the single-task byte limit be B, and the average number of bytes per record in the current batch of the policy data table be A. The effective partition threshold is the minimum of R and B divided by A, rounded down. The average number of bytes per record A is calculated as follows: read the number of bytes E occupied by the current batch of the policy data table and the number of records D in the current batch, then divide E by D to obtain A. The server uses the effective partition threshold as the preset partition threshold for generating initial splitting tasks, and associates the dimension fields, dimension order, dimension value rules, and preset partition threshold with the current synchronization batch number for later use when calculating the data volume corresponding to each dimension combination.

[0031] The system calculates the data volume corresponding to each dimension combination in the policy data table based on the dimension fields, and generates the initial splitting task for each dimension combination based on the data volume and the preset partition threshold.

[0032] After the server obtains the master policy data table, dimension fields, and preset partition limits, it reads each policy record corresponding to the current synchronization batch number one by one, according to the dimension order of the dimension fields in the master policy data table. The dimension order is determined by the dimension configuration obtained in the previous steps. For example, if the time dimension, organization dimension, insurance type dimension, and policy status dimension are arranged in sequence, the server first reads the time dimension field value from the policy record, and then reads the organization code, insurance type code, and policy status. The server concatenates the values ​​of each dimension field in this reading order to form a dimension combination identifier. Adjacent dimension field values ​​are separated by a fixed delimiter character, and a field length value is written before each dimension field value to avoid conflicts in the dimension combination identifier caused by directly concatenating different field values. The field length value is equal to the number of characters contained in the corresponding dimension field value. For example, if the organization code is "GD", the field length value is 2; if the insurance type code is "LIFE", the field length value is 4.

[0033] After generating a dimension combination identifier for each policy record, the server establishes a correspondence between the dimension combination identifier and the policy number, and then aggregates the data according to the dimension combination identifier. Policy records with the same dimension combination identifier are grouped into the same dimension combination; policy records with different dimension combination identifiers are grouped into different dimension combinations. During the aggregation process, the server synchronously accumulates the number of policy records corresponding to each dimension combination. For each policy record aggregated, the data volume of the corresponding dimension combination increases by 1. After aggregation is complete, each dimension combination receives its corresponding data volume and a set of policy numbers for that dimension combination.

[0034] To ensure that subsequent task boundaries can be reproduced upon re-execution, the server does not use temporary line numbers as the basis for task segmentation. Instead, it uses the fixed order of policy numbers as the boundary segmentation basis. For policy records under the same dimension combination identifier, the server arranges them according to the character sorting result of the policy number from smallest to largest. When a policy number contains both numbers and letters, the server first converts the policy number to a unified character encoding, then compares each character encoding sequence digit by digit. If the character encoding value of the earlier character is less than that of the later character, the earlier policy number is ranked first. If two policy numbers have the same character encoding at a common position, the policy number with the shorter character length is ranked first. This results in a stable arrangement of policy numbers within the same dimension combination.

[0035] After sorting the policy numbers, the server sequentially counts the policy records under the same dimension combination identifier. The count starts at 0, and increments by 1 for each sorted policy record read. When the count is an integer multiple of a preset partition limit, the server records the currently read policy number as a task boundary value. If the preset partition limit is 10000, then policy numbers at counts of 10000, 20000, and 30000 are recorded as the 1st, 2nd, and 3rd task boundary values, respectively. If the last count does not reach an integer multiple of the preset partition limit, the server records the last sorted policy number within the same dimension combination as the final task boundary value. The server thus obtains the data volume corresponding to that dimension combination and the sequence of task boundary values ​​arranged in order of policy number.

[0036] The server determines the initial split range based on the data volume and the sequence of task boundary values. For the same dimension combination identifier, the starting boundary of the first initial split range is the policy number ranked first in that dimension combination, and the ending boundary is the first task boundary value; the starting boundary of the second initial split range is the policy number following the first task boundary value, and the ending boundary is the second task boundary value; and so on, until the last task boundary value. Each initial split range is limited by a starting task boundary value and an ending task boundary value, and the number of corresponding policy records does not exceed a preset partition limit. If the data volume of a certain dimension combination is less than or equal to the preset partition limit, the server forms only one initial split range, with the starting task boundary value being the policy number ranked first and the ending task boundary value being the policy number ranked last.

[0037] The server then generates initial splitting tasks based on the dimension combination identifier and the initial splitting range. Each initial splitting task includes at least the dimension combination identifier, target partition identifier, start task boundary value, end task boundary value, task sequence number, and current synchronization batch number. The target partition identifier is derived from the dimension combination identifier, preserving the order of dimension field values ​​during the conversion. Content exceeding the database object name length limit is digested using a national cryptographic hash algorithm to calculate a fixed-length digest. The digest result, along with the dimension sequence number, is then written into the target partition identifier to ensure that the same dimension combination corresponds to the same target partition identifier. The task sequence number increments from 1 according to the order of the initial splitting range within the same dimension combination. Subsequent partitioning reads policy records based on the current synchronization batch number, dimension combination identifier, start task boundary value, and end task boundary value, ensuring that the task execution range does not depend on temporary row numbers and avoiding duplicate or missed processing due to changes in data sorting during re-execution.

[0038] For each dimension combination, obtain the dimension solitary peak value, which represents the degree of data surge relative to similar dimension combinations, and the resource co-infusion value, which represents the degree to which the initial split task set falls into the same target resource.

[0039] After the server obtains the data volume and initial splitting tasks for each dimension combination, it calculates the dimensional isolated peak value and resource co-injection value for each dimension combination. The dimensional isolated peak value characterizes the degree of data surge in the current dimension combination relative to similar dimension combinations, while the resource co-injection value characterizes the degree to which the initial splitting tasks for the current dimension combination are concentrated in the same target resource. Both are represented by integers from 0 to 100, with values ​​closer to 100 indicating a higher degree of anomaly.

[0040] For the current dimension combination, the server first establishes reference conditions according to the dimension order of the dimension fields. Assuming the current dimension combination includes time dimension values, organization dimension values, insurance type dimension values, and policy status dimension values, the server sequentially retains one dimension field value, forming multiple reference conditions. For example, when retaining the time dimension value, the reference condition is that it has the same time dimension value as the current dimension combination; when retaining the organization dimension value, the reference condition is that it has the same organization dimension value as the current dimension combination; when retaining the insurance type dimension value, the reference condition is that it has the same insurance type dimension value as the current dimension combination; when retaining the policy status dimension value, the reference condition is that it has the same policy status dimension value as the current dimension combination. Based on each reference condition, the server then filters similar dimension combinations from the already statistically analyzed data volume for each dimension combination.

[0041] After each filtering step, the server removes the current dimension combination from the resulting similar dimension combinations to prevent its data volume from participating in its own reference. After deleting the current dimension combination, the server arranges the remaining similar dimension combinations in ascending order of data volume, obtaining a sequence of similar data volumes corresponding to the current reference condition. If the similar data volume sequence is empty, the server marks the current reference condition as invalid and excludes it from dimension isolated peak calculations; if the similar data volume sequence is not empty, the server continues to calculate the basic reference volume and the higher-order reference volume.

[0042] The base reference value is calculated as follows: the base order is equal to the number of data items in the same data sequence after rounding up, divided by 2; the base reference value is equal to the data item in the same data sequence that is in the base order. The high-order reference value is calculated as follows: the high-order number is equal to the number of data items in the same data sequence after rounding up, multiplied by 9 and then divided by 10; the high-order reference value is equal to the data item in the same data sequence that is in the high-order position. For example, if the same data sequence contains 23 data items, then the base order is 11.5 after rounding up, which is 12; the high-order number is 20.7 after rounding up, which is 21. The server takes the 12th and 21st data items as the base reference value and the high-order reference value, respectively.

[0043] The data volume of the current dimension combination is compared with both the baseline reference value and the higher-order reference value. The baseline multiple is equal to the data volume of the current dimension combination divided by the baseline reference value plus 1; the higher-order multiple is equal to the data volume of the current dimension combination divided by the higher-order reference value plus 1. The baseline surge level is determined by the baseline multiple: 0 for a baseline multiple less than 2; 25 for a baseline multiple greater than or equal to 2 and less than 5; 60 for a baseline multiple greater than or equal to 5 and less than 10; and 100 for a baseline surge level. The higher-order surge level is determined by the higher-order multiple: 0 for a higher-order multiple less than 1.2; 25 for a higher-order multiple greater than or equal to 1.2 and less than 2; 60 for a higher-order multiple greater than or equal to 2 and less than 5; and 100 for a higher-order multiple greater than or equal to 5.

[0044] The task expansion level is determined based on the initial number of split tasks corresponding to the current dimension combination. The task expansion level is determined according to the initial number of split tasks: 0 for 1 initial split task; 25 for a number greater than 1 and less than 10; 60 for a number greater than or equal to 10 and less than 50; and 100 for a number greater than or equal to 50. For each valid reference condition, the server calculates a reference isolated peak value, which is equal to the base surge level multiplied by 4 and divided by 10, plus the higher surge level multiplied by 4 and divided by 10, plus the task expansion level multiplied by 2 and divided by 10. The result is rounded to the nearest integer. The server selects the largest value from all valid reference conditions' corresponding reference isolated peak values ​​as the dimension isolated peak value for the current dimension combination. This process ensures that any concentrated surge in the current dimension combination along any reference direction is reflected by the dimension isolated peak value, avoiding the weakening of abnormal surge characteristics after multidimensional averaging.

[0045] The server reads each initial split task corresponding to the current dimension combination and retrieves the target partition identifier from each initial split task. The target partition identifier points to the actual write location in the target database. The server uses the target partition identifier to query the target resource mapping and obtains the target resource identifier that each initial split task actually points to. The target resource identifier can consist of the tablespace number, data file number, and index segment number of the target partition, or it can consist of the storage group number to which the target partition belongs. The server arranges the target resource identifiers according to the task sequence number of the initial split task to obtain the target resource landing point sequence.

[0046] The server merges identical target resource identifiers in the target resource landing sequence and counts the number of initial splitting tasks corresponding to each target resource identifier. After merging, the server selects the target resource identifier that carries the most initial splitting tasks as the primary injection target resource identifier. If two or more target resource identifiers carry the same number of initial splitting tasks, the server selects the target resource identifier that appears first in the target resource landing sequence as the primary injection target resource identifier. The proportion of primary injection tasks is equal to the number of initial splitting tasks corresponding to the primary injection target resource identifier divided by the total number of initial splitting tasks corresponding to the current dimension combination.

[0047] The primary infusion level is determined based on the proportion of primary infusion tasks: 0 for primary infusion tasks less than 0.3%, 25 for primary infusion tasks greater than or equal to 0.3 and less than 0.5%, 60 for primary infusion tasks greater than or equal to 0.5 and less than 0.8%, and 100 for primary infusion tasks greater than or equal to 0.8%. This level reflects whether the initial splitting tasks of the current dimension combination are concentrated on a single target resource.

[0048] The server continues to determine the continuous distribution of initial splitting tasks pointing to the main infusion target resource identifier within the fixed order of policy numbers, based on the order of the start and end task boundary values. The server first arranges all initial splitting tasks for the current dimension combination from front to back according to the start task boundary values, and then marks the initial splitting tasks pointing to the main infusion target resource identifier in the arrangement result. If there are no unmarked tasks between two adjacent marked tasks, the two marked tasks belong to the same continuous segment; if there are unmarked tasks between two adjacent marked tasks, the continuous segment ends at the previous marked task. The server counts the number of initial splitting tasks contained in each continuous segment and selects the continuous segment with the most initial splitting tasks as the longest continuous segment. The continuous infusion ratio is equal to the number of initial splitting tasks contained in the longest continuous segment divided by the number of initial splitting tasks corresponding to the main infusion target resource identifier.

[0049] The continuous inflow level is determined based on the continuous inflow percentage: 0 for a continuous inflow percentage less than 0.3; 25 for a continuous inflow percentage greater than or equal to 0.3 and less than 0.5; 60 for a continuous inflow percentage greater than or equal to 0.5 and less than 0.8; and 100 for a continuous inflow percentage greater than or equal to 0.8. The continuous inflow level distinguishes between tasks with a concentrated but interspersed execution volume and tasks with a concentrated volume and continuous writing within a fixed policy number sequence. The latter is more likely to cause the same target resource to experience write pressure over a continuous time period.

[0050] The server determines the resource co-injection value based on the primary injection level, the continuous injection level, and the total number of initial split tasks corresponding to the current dimension combination. First, the total task level is calculated: 0 for a total of 1 initial split task; 25 for a total of more than 1 but less than 10; 60 for a total of more than or equal to 10 but less than 50; and 100 for a total of more than or equal to 50. The resource co-injection value is calculated as: primary injection level multiplied by 5 and divided by 10, plus continuous injection level multiplied by 3 and divided by 10, plus total task level multiplied by 2 and divided by 10. The result is rounded to the nearest integer. This result reflects the concentration of tasks carried by a single target resource, the degree of continuous writing, and the continuous write pressure brought by the total number of tasks. After the dimensional isolated peak value and resource co-injection value are calculated, they are associated with the current dimension combination identifier, data volume, initial split task number, and current synchronization batch number for later use in determining the overall anomaly level.

[0051] The overall anomaly level of the corresponding dimension combination is determined based on the isolated peak value of the dimension and the co-injection value of resources, and the dimension combination is divided into three types according to the overall anomaly level: normal split, mild concentration, and severe concentration.

[0052] After obtaining the isolated peak value and resource co-injection value corresponding to the current dimension combination, a tier conversion with the same caliber is performed on both. Both the isolated peak value and the resource co-injection value are integers from 0 to 100. The server converts values ​​less than 40 to a low tier, values ​​greater than or equal to 40 and less than 70 to a medium tier, and values ​​greater than or equal to 70 and less than or equal to 100 to a high tier. The data surge tier obtained from the isolated peak value is used to represent the data surge state of the current dimension combination relative to similar dimension combinations; the resource co-injection tier obtained from the resource co-injection value is used to represent the state where the initial splitting tasks corresponding to the current dimension combination fall into the same target resource.

[0053] To facilitate comparison, the server assigns a low-rank evaluation file as 1, a median evaluation file as 2, and a high-rank evaluation file as 3. The data surge ranking number corresponding to the current dimension combination's data surge evaluation file is compared with the resource co-flow ranking number corresponding to the resource co-flow evaluation file. The higher ranking number is taken as the candidate comprehensive ranking number, and the lower ranking number is taken as the mutual verification ranking number. The calculation process is as follows: the candidate comprehensive ranking number equals the maximum value between the data surge ranking number and the resource co-flow ranking number; the mutual verification ranking number equals the minimum value between the data surge ranking number and the resource co-flow ranking number; the ranking difference equals the candidate comprehensive ranking number minus the mutual verification ranking number.

[0054] The server determines the overall anomaly level of the current dimension combination based on the level difference. When the level difference is 0, it indicates that the data surge direction and the resource co-flow direction are at the same evaluation level, and the server directly uses the candidate comprehensive level number as the comprehensive anomaly level number. When the level difference is 1, it indicates that the two are adjacent, and the lower evaluation level can support the higher evaluation level; the server still uses the candidate comprehensive level number as the comprehensive anomaly level number. When the level difference is 2, it indicates that one direction reaches a high evaluation level while the other direction is only at a low evaluation level; the server subtracts 1 from the candidate comprehensive level number to obtain the comprehensive anomaly level number. Through this process, a surge in one direction or a concentration of resources in one direction will not be directly identified as the highest anomaly state; only when the data surge and resource co-flow both reach adjacent or consistent evaluation states will the highest anomaly state be retained.

[0055] The processing type of the current dimension combination is determined based on the comprehensive anomaly level number. When the comprehensive anomaly level number is 1, the current dimension combination is classified as a normal split type; when the comprehensive anomaly level number is 2, the current dimension combination is classified as a mild concentration type; when the comprehensive anomaly level number is 3, the current dimension combination is classified as a severe concentration type.

[0056] After the partitioning is completed, the server will write the current dimension combination identifier, dimension isolated peak value, resource co-injection value, data surge evaluation file, resource co-injection evaluation file, comprehensive anomaly file number and processing type into the anomaly evaluation record corresponding to the current synchronization batch number, for subsequent tasks to retain, monitor flag settings or dynamically adjust processing calls.

[0057] For example, if the isolated peak value of the current dimension combination is 82 and the resource co-flow value is 76, both are converted to high-level evaluation tiers. The data surge tier number is 3, the resource co-flow tier number is 3, the candidate comprehensive tier number is 3, the mutual verification tier number is 3, the tier difference is 0, and the comprehensive anomaly tier number is 3. The current dimension combination is classified as heavily concentrated. As another example, if the isolated peak value of the current dimension combination is 88 and the resource co-flow value is 25, the data surge tier number is 3, the resource co-flow tier number is 1, the candidate comprehensive tier number is 3, the mutual verification tier number is 1, the tier difference is 2, and the comprehensive anomaly tier number is adjusted to 2. The current dimension combination is classified as lightly concentrated. Thus, the server can distinguish between two scenarios: a surge in data volume with dispersed resource distribution and a surge in data volume with concentrated resource distribution, avoiding misjudgment of task processing type due to a single evaluation direction.

[0058] For normal splitting types, the initial splitting task is retained; for lightly centralized types, the initial splitting task is retained and a monitoring flag is set; for heavily centralized types, a dynamic auxiliary splitting field is added, the target partition identifier and the execution range of the splitting task are regenerated, and the policy data master table is partitioned according to the regenerated splitting task.

[0059] After completing the processing type division of the current dimension combination, read the initial split task corresponding to the dimension combination according to the current synchronization batch number, and determine the subsequent writing method according to the processing type.

[0060] When the current dimension combination is classified as a normal splitting type, the server does not change the initial splitting task corresponding to that dimension combination. Instead, it directly writes the dimension combination identifier, target partition identifier, start task boundary value, end task boundary value, and task sequence number from the initial splitting task into the list of tasks to be executed. The records in the list of tasks to be executed maintain a one-to-one correspondence with the initial splitting task, ensuring that policy records of the normal splitting type enter subsequent partitioning processing according to the original dimension combination identifier and the original task boundary value.

[0061] When the current dimension combination is classified as a lightly centralized type, the server retains the initial splitting task corresponding to that dimension combination and writes it to the pending task list. Simultaneously, the server writes a monitoring flag to the corresponding task record. The monitoring flag is bound to the dimension combination identifier, target partition identifier, start task boundary value, and end task boundary value. The monitoring flag includes the processing type number, dimension isolated peak value, resource co-flow value, and comprehensive anomaly level number. A processing type number of 2 indicates a lightly centralized type. Subsequent partitioning processes, after reading the monitoring flag, calculate the task start time, task end time, number of written records, and number of failures according to the task boundary values, ensuring that the lightly centralized type retains traceable information without changing the splitting scope.

[0062] When the current dimension combination is classified as a heavily concentrated type, the server does not directly use the initial splitting task corresponding to that dimension combination. Instead, it first forms a dynamic auxiliary splitting field based on the fixed order of policy numbers under that dimension combination. The server reads all policy numbers under that dimension combination and arranges them in a fixed order. Then, it extracts character segments with the same character position from each policy number, with a segment length of 2. Candidate character segments slide sequentially from the first position of the policy number, moving one position forward each time, until no character segment of length 2 can be obtained. The server counts the number of different values ​​corresponding to each candidate character segment and the number of policy records under each value, and calculates the discrete evaluation value. The discrete evaluation value is equal to the number of different values ​​multiplied by the minimum number of policy records under each value, and then divided by the maximum number of policy records under each value plus 1. The server selects the candidate character segment with the highest discrete evaluation value as the source of the dynamic auxiliary splitting field; if there are more than two candidate character segments with the same discrete evaluation value, the candidate character segment with the earlier character position is selected.

[0063] After obtaining the source of the dynamic auxiliary splitting field, the corresponding character segment value is extracted for each policy record under this dimension combination, and this character segment value is used as the dynamic auxiliary splitting field value. The reconstructed dimension combination identifier consists of the original dimension combination identifier and the dynamic auxiliary splitting field value, separated by a fixed delimiter character, with the character length value written before the dynamic auxiliary splitting field value. Thus, a dimension combination of the same heavily concentrated type is split into multiple reconstructed dimension combination identifiers. The server re-aggregates policy records according to the reconstructed dimension combination identifiers, distributing the policy records originally concentrated under a single dimension combination into multiple sub-combinations based on the policy number character segment.

[0064] The target partition identifier is redefined based on each restructured dimension combination identifier. The target partition identifier is derived from the restructured dimension combination identifier, preserving the original dimension field value order and appending the dynamic auxiliary splitting field value to the end. If the target partition identifier length does not exceed the database object name length limit, the transformation result is used directly; if the target partition identifier length exceeds the database object name length limit, a summary is processed on the restructured dimension combination identifier, and the summary result, dimension order number, and dynamic auxiliary splitting field value are written together into the target partition identifier. In this way, different dynamic auxiliary splitting field values ​​under the same heavily concentrated dimension combination type can correspond to different target partition identifiers, reducing the situation where multiple tasks are concentrated and written to the same target resource.

[0065] The execution scope of the splitting task is redefined according to the fixed order of the policy numbers corresponding to the reconstructed dimension combination identifier. For any reconstructed dimension combination identifier, the server counts the corresponding policy records according to the fixed order of the policy numbers; the count starts at 0, and increases by 1 for each policy record read; when the count reaches an integer multiple of the preset partition limit, the current policy number is recorded as the reconstructed task boundary value; when the last count does not reach an integer multiple of the preset partition limit, the last policy number in the sorted sequence is recorded as the last reconstructed task boundary value. The server forms a reconstructed splitting task based on the reconstructed task boundary values, and writes the reconstructed dimension combination identifier, the redefined target partition identifier, the starting task boundary value, the ending task boundary value, the task sequence number, and the current synchronization batch number into the reconstructed splitting task.

[0066] During the partitioning phase, the server sequentially reads the task records from the list of tasks to be executed. For normal splitting and lightly concentrated types, the server reads the policy data master table and writes it to the corresponding target partition according to the dimension combination identifier, target partition identifier, start task boundary value, and end task boundary value. For heavily concentrated types, the server reads the policy data master table and writes it to the corresponding target partition according to the reconstructed dimension combination identifier, the newly determined target partition identifier, start task boundary value, and end task boundary value. Through this process, normal splitting maintains the original execution path, lightly concentrated types add traceable markers, and heavily concentrated types introduce dynamic auxiliary splitting fields based on the fixed order of policy numbers, thereby distributing the concentrated write pressure to multiple target partitions.

[0067] Example 2: To verify the effectiveness of the multi-source policy data processing method in this application in identifying abnormal concentrated dimension combinations and improving the pressure of concentrated writing to target resources, a test batch with the current synchronization batch number "20xx0430" was selected as the verification object. The policy data in this test batch was synchronously generated from underwriting data sources, policy maintenance data sources, premium payment data sources, claims data sources, channel sales data sources, and institutional business data sources. After synchronization, a general policy data table was formed in the target database. This general policy data table includes fields for policy number, institution code, insurance type code, channel code, policy status, effective date, premium amount, data source identifier, synchronization batch number, and update time. After integrity verification, the sum of the number of synchronized records from the six data sources is consistent with the number of records corresponding to the current synchronization batch number in the general policy data table, with a record consistency value of 1, and proceeds to subsequent splitting processing.

[0068] This embodiment uses time, organization, insurance type, and policy status as dimension fields, in the following order: time, organization, insurance type, and policy status. The preset partition limit is 10,000 records, meaning the number of policy records corresponding to the same initial splitting task will not exceed 10,000. The target resource identifier consists of the storage group number and index segment number of the target partition. For ease of explanation, this embodiment selects five representative dimension combinations for comparison; the relevant data is shown in Table 1 below.

[0069] Table 1. Dimensional Combination Data Volume and Initial Splitting Task Information

[0070] As shown in Table 1, the data volume of the fourth dimension combination "20xx04+ONLINE+HEALTH+VALID" is 386,000 records, which are divided into 39 initial splitting tasks according to the preset partition limit of 10,000 records. If the existing fixed splitting method is used, these 39 initial splitting tasks will still point to the same main target resource identifier "Resource 05", which can easily lead to continuous write pressure.

[0071] The server calculates the isolated peak value and resource co-supply value for each dimension combination. Taking the fourth dimension combination as an example, the server uses time dimension value, institution dimension value, insurance type dimension value, and policy status dimension value as reference conditions to filter similar dimension combinations, removing the current dimension combination to form a similar data volume sequence. When using the time dimension value "20xx04" as the reference condition, the number of data in the middle order of the similar data volume sequence is 26,000, and the number of data in the later order is 52,000. The current dimension combination has 386,000 data. The current data volume divided by the base reference value plus 1 is 386,000 ÷ 26,001, which is 14.85; the current data volume divided by the high reference value plus 1 is 386,000 ÷ 52,001, which is 7.42. According to the aforementioned level rules, the base surge level is 100, and the high surge level is 100. The initial number of split tasks for this dimension combination is 39, corresponding to a task expansion level of 60. The reference isolated peak value is equal to the base surge level multiplied by 4 and divided by 10, plus the high-level surge level multiplied by 4 and divided by 10, plus the task expansion level multiplied by 2 and divided by 10, i.e., 100×4÷10+100×4÷10+60×2÷10, which equals 92. Therefore, the reference isolated peak value under this reference condition is 92. After the server performs the same calculation on other reference conditions, it takes the maximum value as the dimension isolated peak value. In this embodiment, the dimension isolated peak value of the fourth dimension combination is 92.

[0072] For resource-based infusion values, among the 39 initial splitting tasks in the fourth dimension combination, 36 initial splitting tasks point to resource 05, and the remaining 3 initial splitting tasks point to other target resources. The proportion of primary infusion tasks is 36 ÷ 39, resulting in 0.923. According to the primary infusion level rule, the primary infusion level is 100. After arranging according to the order of the starting and ending task boundary values, there is a longest continuous segment among the tasks pointing to resource 05, which contains 34 initial splitting tasks. The proportion of continuous infusion is 34 ÷ 36, resulting in 0.944, corresponding to a continuous infusion level of 100. The total number of initial splitting tasks in this dimension combination is 39, corresponding to a total task level of 60. The resource co-infusion value equals the main infusion level multiplied by 5 and divided by 10, plus the continuous infusion level multiplied by 3 and divided by 10, plus the total task level multiplied by 2 and divided by 10, which is 100×5÷10+100×3÷10+60×2÷10, resulting in 92. Therefore, the resource co-infusion value of the fourth dimension combination is 92.

[0073] Table 2. Dimensional Isolated Peak Values, Resource Co-irrigation Values, and Processing Types

[0074] As shown in Table 2, the method of this application can identify the fourth dimension combination as a heavily concentrated type. For normal splitting types, the server retains the initial splitting task; for lightly concentrated types, the server retains the initial splitting task and writes a monitoring flag; for heavily concentrated types, the server adds a dynamic auxiliary splitting field to redetermine the target partition identifier and the execution scope of the splitting task.

[0075] In the fourth dimension combination, the server reads all policy numbers under this dimension combination and arranges them in a fixed order. Then, starting from the first digit of the policy number, it sequentially extracts candidate character segments of length 2, counts the number of different values ​​for each candidate character segment, and calculates the minimum and maximum number of policy records for each value, as well as the discrete evaluation value. The calculation results are shown in Table 3.

[0076] Table 3. Data table for selecting fields for dynamic auxiliary splitting.

[0077] As shown in Table 3, the discrete evaluation value is calculated by multiplying the number of different values ​​by the minimum number of policy records, and then dividing by the maximum number of policy records plus 1. Taking the 3rd to 4th position as an example, the discrete evaluation value is 24 × 14200 ÷ 17601, resulting in 19.36. The discrete evaluation value of the 3rd to 4th position is the highest, so the server uses the character segment of the 3rd to 4th position as the source of the dynamic auxiliary splitting field. Subsequently, the server combines the original dimension combination identifier with the value of this character segment to form multiple reconstructed dimension combination identifiers, and redetermines the target partition identifier and reconstructing splitting task for each reconstructed dimension combination identifier. The target resource writing status of the 4th dimension combination before and after dynamic adjustment is shown in Table 4.

[0078] Table 4 Comparison of resource concentration before and after dynamic adjustment

[0079] As shown in Table 4, when using the fixed splitting method, 36 out of the 39 initial splitting tasks fell into the same target resource, with the main flooding task ratio being 0.923. After adopting the dynamic adjustment method of this application, this heavily concentrated type of dimension combination was split into 24 target partitions, the number of tasks on the single target resource with the most tasks decreased to 4, and the main flooding task ratio decreased to 4÷48, i.e., 0.083. The resource co-flooding value decreased from 92 to 31, indicating that the continuous write pressure on the same target resource decreased significantly.

[0080] To further verify the beneficial effects, under the same server configuration, database configuration, policy data master table, and preset partition limits, partitioning was performed using both the fixed splitting method and the method described in this application. The execution results are shown in Table 5.

[0081] Table 5 Comparison of Partition Processing Results

[0082] As shown in Table 5, with the same number of policy records across all batches, although the number of tasks increased from 51 to 60 due to task restructuring and splitting, the longest single task time decreased from 182 seconds to 64 seconds, the total batch completion time decreased from 426 seconds to 231 seconds, the maximum number of write waits for a single target resource decreased from 2870 to 640, and the number of write failure retries decreased from 11 to 1. After the tasks were completed, the policy records corresponding to the current synchronization batch number in the policy data master table were compared with the policy records in each target partition. There were no differences, indicating that the dynamic auxiliary splitting field does not change the policy record attribution criteria and does not cause policy records to be omitted or duplicated.

[0083] As can be seen from this embodiment, the method of this application does not simply increase the number of tasks. Instead, after identifying isolated peaks in dimension combinations and resource co-occurrence values, it performs dynamic auxiliary splitting only on heavily concentrated types, thereby distributing the initial splitting tasks that fall concentrated on the same target resource to multiple target partitions. This processing can reduce the continuous write pressure on a single target resource, reduce write waiting and failure retries, and shorten the processing time of the entire batch of partitions while maintaining the consistency of the splitting criteria in the policy data master table. Therefore, the method of this application can effectively solve the problem that a few heavily concentrated dimension combinations slow down the entire splitting batch in data skew scenarios when using fixed dimension and fixed partition limit splitting methods.

[0084] Example 3, please refer to Figure 2 As shown in this embodiment, a multi-source policy data processing system includes: The data acquisition module acquires a master table of policy data generated by synchronizing multiple data sources, and acquires the dimension fields and preset partition thresholds used to split the master table of policy data. The initial task generation module calculates the data volume corresponding to each dimension combination in the policy data master table based on the dimension fields, and generates the initial splitting task for each dimension combination based on the data volume and the preset partition threshold. The abnormal value acquisition module acquires, for each dimension combination, the dimensional isolated peak value, which represents the degree of data abrupt increase relative to similar dimension combinations, and the resource co-infusion value, which represents the degree to which the initial split task set falls into the same target resource. The anomaly type classification module determines the comprehensive anomaly degree of the corresponding dimension combination based on the isolated peak value of the dimension and the co-injection value of resources, and classifies the dimension combination into three types: normal split, mild concentration, and severe concentration based on the comprehensive anomaly degree. The partitioning task adjustment module retains the initial partitioning task for normal partitioning types; retains the initial partitioning task and sets a monitoring flag for lightly concentrated types; and adds a dynamic auxiliary partitioning field for heavily concentrated types, regenerates the target partition identifier and the execution range of the partitioning task, and partitions the policy data master table according to the regenerated partitioning task.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for processing multi-source policy data, characterized in that, include: Obtain the general policy data table generated by synchronizing multiple data sources, and obtain the dimension fields and preset partition thresholds used to split the general policy data table; The data volume corresponding to each dimension combination in the policy data table is calculated based on the dimension fields, and the initial splitting task for each dimension combination is generated based on the data volume and the preset partition threshold. For each dimension combination, obtain the dimension solitary peak value, which represents the degree of data abruptness relative to similar dimension combinations, and the resource co-infusion value, which represents the degree to which the initial split task set falls into the same target resource. The overall anomaly level of the corresponding dimension combination is determined based on the isolated peak value of the dimension and the co-injection value of resources, and the dimension combination is divided into three types according to the overall anomaly level: normal split, mild concentration and severe concentration. For normal splitting types, the initial splitting task is retained; for lightly centralized types, the initial splitting task is retained and a monitoring flag is set; for heavily centralized types, a dynamic auxiliary splitting field is added, the target partition identifier and the execution range of the splitting task are regenerated, and the policy data master table is partitioned according to the regenerated splitting task.

2. The multi-source policy data processing method according to claim 1, characterized in that, Obtain the master policy data table generated by synchronizing multiple data sources, and obtain the dimension fields and preset partition thresholds used to split the master policy data table, including: Read the current synchronization batch number, obtain the policy data summary table after synchronizing multiple data sources to the target database based on the current synchronization batch number, and compare the sum of the number of records synchronized from each data source with the corresponding number of records in the policy data summary table; if they match, read the enabled dimension fields and their dimension order from the dimension configuration table, and read the preset partition threshold that matches the current synchronization batch number from the partition threshold configuration table; associate and store the dimension fields, dimension order, and preset partition threshold with the current synchronization batch number for later use when calculating the data volume of dimension combinations.

3. The multi-source policy data processing method according to claim 2, characterized in that, Based on the dimension fields, calculate the data volume corresponding to each dimension combination in the policy data master table, and generate initial splitting tasks for each dimension combination based on the data volume and preset partition thresholds, including: Extract the dimension field values ​​corresponding to each policy record in dimensional order, and combine each dimension field value with a field length value and a fixed separator character to form a dimension combination identifier, so that policy records with the same dimension field value are grouped into the same dimension combination identifier; for policy records under the same dimension combination identifier, count them in a fixed order according to the policy number, and record the current policy number as the task boundary value when the count reaches an integer multiple of the preset partition threshold; form the initial split range based on the task boundary value, and write the dimension combination identifier, target partition identifier, starting task boundary value, ending task boundary value and current synchronization batch number into the corresponding initial split task.

4. The multi-source policy data processing method according to claim 3, characterized in that, The determination of the fixed sequence of policy numbers includes: The policy numbers under the same dimension combination identifier are converted into a unified character code, and the order of the policy numbers is determined by comparing each character code digit by digit. When two policy numbers have the same character code in the common position, the policy number with the shorter character length will be listed first. Record task boundary values ​​in a fixed order according to the determined policy number, so that the same initial split task is still limited by the same starting task boundary value and ending task boundary value when it is re-executed.

5. The multi-source policy data processing method according to claim 3, characterized in that, Obtain isolated peak values ​​in a dimension, including: For the current dimension combination, retain one dimension field value in sequence as a reference condition, and filter out the same type of dimension combination with the same reference condition from the data volume of each dimension combination; Remove the current dimension combination from the same dimension combination, and arrange them in ascending order of data volume, selecting the data volume in the middle order and the data volume in the later order as the basic reference value and the high-order reference value respectively; The data volume of the current dimension combination is compared with the basic reference volume and the high-order reference volume, and combined with the number of initial splitting tasks corresponding to the current dimension combination, the dimensional peak value of the current dimension combination is determined.

6. The multi-source policy data processing method according to claim 3, characterized in that, Obtaining resource allocation values ​​includes: Read the target partition identifiers in each initial splitting task corresponding to the current dimension combination, and determine the target resource identifiers that each initial splitting task actually points to based on the target partition identifiers to obtain the target resource landing point sequence; Merge identical target resource identifiers in the target resource landing sequence and determine the target resource identifier that carries the most initial splitting tasks as the main injection target resource identifier; Based on the order of the starting task boundary value and the ending task boundary value, determine the continuous distribution of the initial splitting tasks pointing to the main infusion target resource identifier, and determine the resource co-infusion value by combining the number of initial splitting tasks corresponding to the main infusion target resource identifier and the total number of initial splitting tasks in the current dimension combination.

7. The multi-source policy data processing method according to claim 6, characterized in that, The overall anomaly level of the corresponding dimension combination is determined based on the isolated peak value of the dimension and the co-flow value of resources, including: Map the isolated peak value of the dimension to the data surge evaluation level, map the resource co-irrigation value to the resource co-irrigation evaluation level, and convert the data surge evaluation level and the resource co-irrigation evaluation level into the corresponding level number respectively. The higher of the two gear numbers is used as the candidate comprehensive gear number, and the lower of the two gear numbers is used as the mutual verification gear number. The comprehensive abnormal gear number is determined based on the gear difference between the candidate comprehensive gear number and the mutual verification gear number. When the difference in rating levels indicates that two rating levels are not adjacent, the candidate comprehensive rating level number is lowered and used as the comprehensive abnormal rating level number to reduce severe concentrated misjudgments caused by anomalies in a single direction.

8. The multi-source policy data processing method according to claim 7, characterized in that, Based on the degree of overall anomaly, dimensional combinations are categorized into three types: normal splitting, mild concentration, and severe concentration, including: When the comprehensive abnormal level number is a low level, the current dimension combination is classified as a normal split type, and its initial split task is retained. When the comprehensive anomaly level number is the median level, the current dimension combination is classified as a slightly concentrated type, and a monitoring tag associated with the dimension combination identifier, target partition identifier, and task boundary value is written in its initial split task. When the comprehensive anomaly level number is a high level, the current dimension combination is classified as a heavily concentrated type, and the dynamic auxiliary splitting field determination process is triggered.

9. A multi-source policy data processing method according to claim 8, characterized in that, For heavily concentrated data types, a dynamic auxiliary splitting field is added to regenerate the target partition identifier and the execution scope of the splitting task, including: Read all policy numbers under the corresponding dimension combination of the heavily concentrated type, and extract candidate character segments with the same character position according to the fixed order of the policy numbers. Determine the dynamic auxiliary splitting field based on the number of different values ​​of the candidate character segments and the number of policy records corresponding to each value. Add the dynamic auxiliary splitting field to the original dimension combination identifier to obtain the reconstructed dimension combination identifier. Then, redetermine the target partition identifier based on the reconstructed dimension combination identifier. Redefine the task boundary values ​​according to the fixed order of the policy numbers corresponding to the reconstructed dimension combination identifier to form a reconstructed splitting task. Then, perform partitioning processing on the policy data master table based on the reconstructed splitting task.

10. A multi-source policy data processing system, used to implement the multi-source policy data processing method according to any one of claims 1-9, characterized in that, include: The data acquisition module acquires a master table of policy data generated by synchronizing multiple data sources, and acquires the dimension fields and preset partition thresholds used to split the master table of policy data. The initial task generation module calculates the data volume corresponding to each dimension combination in the policy data master table based on the dimension fields, and generates the initial splitting task for each dimension combination based on the data volume and the preset partition threshold. The abnormal value acquisition module acquires, for each dimension combination, the dimensional isolated peak value, which represents the degree of data abrupt increase relative to similar dimension combinations, and the resource co-infusion value, which represents the degree to which the initial split task set falls into the same target resource. The anomaly type classification module determines the comprehensive anomaly degree of the corresponding dimension combination based on the isolated peak value of the dimension and the co-injection value of resources, and classifies the dimension combination into three types: normal split, mild concentration, and severe concentration based on the comprehensive anomaly degree. The partitioning task adjustment module retains the initial partitioning task for normal partitioning types; retains the initial partitioning task and sets a monitoring flag for lightly concentrated types; and adds a dynamic auxiliary partitioning field for heavily concentrated types, regenerates the target partition identifier and the execution range of the partitioning task, and partitions the policy data master table according to the regenerated partitioning task.

Images