A method for automatically discovering reconciliation constraints for high sparse heterogeneous financial data tables

Abstract

The present application relates to the technical field of intelligent analysis of financial data, and particularly relates to a method for automatically discovering reconciliation constraints for high-sparse heterogeneous financial data tables. The present application identifies a locally consistent record set by record clustering and field sub-cluster generation based on a field non-empty mode, then discovers reconciliation relationships within a locally effective sample range, and then introduces an inverse frequency weight and enhances sparse fields, so that fields that only appear in specific business scenarios but play a key role in reconciliation structure identification are not easily covered by high-frequency fields. The common non-empty constraints and splitting of sufficient field candidate sub-clusters are performed to gradually shrink the candidate field set, avoid irrelevant fields from being mixed into the same linear solution space, and improve the concentration and interpretability of numerical reconciliation relationship discovery.

Description

Technical Field

[0001] This invention relates to the field of intelligent financial data analysis technology, and in particular to an automatic discovery method for reconciliation constraints in highly sparse and heterogeneous financial data tables. Background Technology

[0002] In financial statements, detailed notes, accounting ledgers, and cross-system financial data governance scenarios, numerous reconciliation relationships manifest as additive balances between multiple fields, serving as crucial evidence for financial verification, audit review, and data quality governance. Existing financial reconciliation verification solutions largely rely on manually pre-set rules, where finance or audit personnel maintain reconciliation formulas beforehand and then verify each item in the data to be analyzed. While this approach is feasible in scenarios with relatively stable field structures and simple templates, in financial accounting data environments characterized by high missing data, high heterogeneity, and mixed templates, manual rule maintenance becomes costly and it struggles to promptly identify new implicit reconciliation relationships.

[0003] For example, CN112650762B discloses an invention of a method, device, electronic device and storage medium for data quality monitoring. Although the invention involves automatically determining target data rules based on historical data, it still relies on rule templates and preset mathematical relationships. It does not further propose a complete solution for automatically identifying locally consistent samples, shrinking candidate field sets and robustly mining unknown additive balance constraints in highly sparse financial data. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, such as insufficient local co-occurrence and difficulty in automatically, stably, and interpretably discovering financial reconciliation relationships in scenarios with highly sparse and heterogeneous financial data, this invention proposes an automatic discovery method for reconciliation constraints in highly sparse and heterogeneous financial data tables.

[0005] To achieve the above objectives, the present invention adopts the following technical solution, including: An automatic discovery method for reconciliation constraints in highly sparse and heterogeneous financial data tables includes the following steps: S1. Data Reading and Non-Null Modeling: Obtain the financial accounting data table to be analyzed, filter the effective numerical fields and construct a binary matrix representing the field distribution pattern, introduce inverse frequency weights to enhance sparse fields, and generate a weighted non-null state vector for each record. S2. Record Clustering: Perform online clustering on each record based on weighted cosine similarity to obtain record clusters whose weighted cosine similarity meets a preset threshold; S3. Field sub-cluster generation: Construct an undirected field graph within each record cluster, and combine it with common non-empty sample conditions to form sufficient candidate field sub-clusters suitable for modeling; S4. Solving the balance constraint: Solve the balance constraint coefficient vector on the candidate subclusters of sufficient fields through singular value decomposition, mine the financial reconciliation relationship between fields that meet the balance constraint, further filter the inliers that meet the financial reconciliation relationship, and update the balance constraint coefficient vector based on the proportion of inliers. S5. Rule Output: Based on the updated balance constraint coefficient vector and the corresponding sufficient field candidate subclusters, construct a financial reconciliation expression, input actual data to verify the rule confidence of the financial reconciliation expression, and output stable financial reconciliation rules.

[0006] Preferably, in step S1, the non-empty record values ​​of each field in the statistical data table are analyzed, a minimum non-empty rate threshold is preset, and fields with a non-empty record rate not lower than the minimum non-empty rate threshold are selected as fields to be analyzed. Numerical fields from these fields are then extracted as valid fields. Based on all valid fields, a binary matrix representing the distribution pattern of empty / non-empty states of the valid fields is constructed. ,in, N Indicates the total number of records. M This indicates the total number of valid fields; for valid fields... j In the i binary variables in records If the record is not empty, then ;otherwise, .

[0007] Preferably, in step S1, for any valid field Statistical non-empty frequency ; Calculate valid fields The inverse document frequency weight is The set of weight vectors for the effective field set is denoted as . For the first i 1 record, row vector denoted as Weighted state vector .

[0008] Preferably, in step S2, for the first i The weighted cosine similarity is calculated for each record as follows: ; Preset similarity threshold ,in, This is a record cluster used to include records whose mean similarity to this record cluster reaches a threshold. Weighted state vector ; For class Internal weighted non-empty state vector The mean; iterate through all existing classes ,like Then If a class has the highest similarity, it should be assigned to the class with the highest similarity, and the corresponding class center should be updated; otherwise, it should be assigned to the class with the highest similarity. Create a new class for the initial center.

[0009] Preferred, class center The mean of the weighted non-empty state vector is used for online updates, and the update formula is as follows: ; in, The updated classes center; Representation Class The number of elements; This indicates the first one that has not yet been categorized. i The weighted state vector of the records.

[0010] Preferably, in step S3, an undirected graph of the field is constructed: based on record clusters. Calculate any two distinct fields from all the original data. Pearson correlation coefficient between Preset correlation coefficient threshold edge set Record cluster The undirected graph is notated as .

[0011] Preferred method for selecting candidate subclusters with sufficient fields: extracting an undirected graph. G Each connected component is considered as a candidate subcluster for a field. For any one of these candidate subclusters... Calculate the number of common non-empty samples across all fields: Preset minimum threshold for sufficient fields and the minimum threshold for non-empty samples in the candidate field ; like If the number of fields is not less than the set number, then the current field candidate subcluster can be selected as a sufficient field candidate subcluster; like Candidate subclusters of the current field Fields in the array are sorted by non-empty frequency. Sort in descending order, if the number of common non-empty samples of the current field and the other candidate fields is not less than If the candidate field is not specified, then add the candidate field to the preliminary candidate field sub-cluster; after traversing all fields, select the preliminary candidate field sub-cluster with a number of fields not less than the set number as the sufficient candidate field sub-cluster.

[0012] Preferably, in step S4, any one of the sufficient field candidate subclusters is set. Include q Each field and nThe number of common non-empty samples, among which, y Represents candidate subclusters of fields The index of sufficient field candidate subclusters, the sample matrix is For the sample matrix X Perform singular value decomposition: ;in, U Describes a left singular vector matrix. Σ Represents a singular value matrix. V Denotes the right singular vector matrix; takes the singular value matrix. Σ The right singular vector in the right singular vector matrix corresponding to the minimum singular value is used as the summation-type balance constraint coefficient vector c.

[0013] Preferably, the additive balance constraint coefficient vector c Residuals on each sample: ,in, Represents the sample matrix X The i One sample; Represents the vector of summative equilibrium constraint coefficients c In the i The residuals of each sample; The first i Scale of a sample ;in, Indicates the first i In the nth sample, the nth j The original values ​​corresponding to each field; Find the maximum value for all fields; No. i The relative error of each sample is: ;in, To prevent extremely small positive numbers from being divided by zero, ; Preset interior point error threshold The relative error is less than the interior point error threshold. If the samples are considered as interior points, then the set of interior points is denoted as . Correspondingly, the set of interior points Concatenate to obtain the interior point sample matrix Interior point sample matrix contain One sample; Preset interior point ratio threshold ,like For the interior point sample matrix Re-perform the singular value decomposition to obtain the updated additive equilibrium constraint coefficient vector. .

[0014] Preferably, a near-zero threshold is preset. Filter out the updated additive balance constraint coefficient vector The absolute value of the corresponding component in the middle field does not exceed the near-zero threshold. The fields; the components corresponding to the filtered fields are divided into the left set according to their signs, based on the positive or negative signs of the sufficient field candidate sub-clusters. and the set on the right ; When the left set With the set on the right When all records are non-empty and the total number of participating fields is not less than a set number, construct an additive equation reconciliation expression based on the original values ​​of the samples corresponding to the fields: ; in, Representation field j The original value corresponding to the current sample; It is worth noting that both sides of the equation must contain the original value of at least one field. ; Rule confidence The calculation method is as follows: ; in, Represents the interior point sample matrix The i One sample; Indicates the first i In the nth sample j The record values ​​corresponding to each field; (·) is an indicator function; The preset error threshold is used; Preset minimum confidence threshold If the rule confidence level Then output the corresponding cross-validation expression and rule confidence. and the corresponding sufficient field candidate subclusters .

[0015] The advantages of this invention are: (1) By clustering records based on field non-empty patterns and generating field sub-clusters, a locally consistent set of records is first identified, and then the correlation is found within the range of local effective samples, which significantly reduces the noise interference caused by direct modeling of the full sparse data.

[0016] (2) By introducing inverse frequency weights and enhancing sparse fields, fields that only appear in specific business scenarios but play a key role in identifying the reconciliation structure are not easily covered by high-frequency fields, thereby improving the accuracy of local record cluster division.

[0017] (3) Perform common non-empty constraints and splits on sufficient field candidate subclusters to achieve gradual shrinkage of the candidate field set, avoid irrelevant fields from being mixed into the same linear solution space, and improve the concentration and interpretability of numerical correlation discovery.

[0018] (4) Improve the stability, accuracy and interpretability of financial reconciliation rule discovery by solving additive balance constraints and non-empty sample verification mechanism. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 This is a data flow diagram of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1 An automatic discovery method for reconciliation constraints in highly sparse and heterogeneous financial data tables, such as Figure 1 , Figure 2 As shown, it includes the following steps: S1. Data Reading and Non-Null Modeling: Obtain the financial accounting data table to be analyzed, filter the valid numerical fields, construct a binary matrix representing the non-null pattern of the fields, and generate a weighted non-null state vector for each record.

[0022] The data table contains multiple fields, each corresponding to multiple records, and different records have different field missing patterns.

[0023] S11. Data Table Reading and Valid Field Filtering: Analyzes the non-empty records in each field of the data table, setting a minimum non-empty rate threshold. The non-empty record rate should not be lower than the minimum non-empty rate threshold. The fields to be analyzed are selected as the fields to be analyzed, and the numeric fields in the fields to be analyzed are extracted as valid fields.

[0024] Let the total number of records in the data table be... , No. j The number of non-empty records in each field is The non-empty rate is denoted as ; Preferred, .

[0025] S12. Constructing a binary matrix: based on the set of valid fields. F Construct a binary matrix to represent the non-empty pattern of the field.

[0026] The set of valid fields is denoted as F， Valid field set F have M Valid fields.

[0027] No. j There are 1 valid fields N The record will be the first j The first field i Define a binary variable for each record. If the first j The valid field in the first i If a record contains a non-empty record, then ;otherwise, .

[0028] Therefore, a binary matrix is ​​constructed to represent the distribution pattern of the empty / non-empty states of the effective field. .

[0029] S13. For each valid field, count the frequency of its non-empty occurrences: , where valid fields ; Calculate valid fields Inverse document frequency weights: Valid fields j The more non-empty records there are, the smaller the weight. Valid field set F The set of inverse document frequency weight vectors is: .

[0030] In a binary matrix In the middle, for the first i Record number 1 i The row vector of a record is denoted as .

[0031] For the i The weighted non-empty state vector of the records is ; .

[0032] The weighted non-empty state vector It represents the non-empty distribution pattern of the current record on each valid field and the importance of each field to the identification of the local business structure, among which non-empty and low-frequency key fields are given higher weight.

[0033] S2. Record Clustering: Perform online clustering on each record based on weighted cosine similarity to obtain record clusters whose weighted cosine similarity meets a preset threshold.

[0034] Define class collection The first class Initialize to an empty class, that is ; Each record is assigned to a different class (record cluster) using the following online clustering method: ; in, , p For class collection C The index; For class All weighted non-empty state vectors within The mean of the values ​​is the class center; Preset similarity threshold Iterate through all class collections If all records of the class already exist, Then the first i Each record is grouped into the class with the highest similarity. In the middle, and update the corresponding class center. Otherwise, with Create a new class for the initial center; Preferred, .

[0035] Online clustering uses a large similarity threshold Grouping nearly identical rows of records into one category avoids mixing data from different templates together to calculate cross-correlation relationships, effectively preventing cross-contamination.

[0036] Preferred, class center The mean of the weighted non-empty state vector is used for online updates, and the update formula is as follows: ; in, The updated classes center; Representation Class The number of elements; This indicates the first one that has not yet been categorized. i The weighted state vector of the records.

[0037] Collection of classes The initial value is an empty class. By continuously creating new classes and adding them to the class collection C, they become existing classes.

[0038] All classes (record clusters) are counted separately into the class set. middle.

[0039] For class collections Any record cluster in , including the first i The record is called the first. i One sample.

[0040] S3. Field sub-cluster generation: Construct an undirected field graph within each record cluster, and combine it with common non-empty sample conditions to form sufficient candidate field sub-clusters suitable for modeling.

[0041] S31, Based on record clusters Calculate any two distinct fields from all the original data. Pearson correlation coefficient between .

[0042] S32, Preset correlation coefficient threshold Based on Pearson correlation coefficient Constructing a cluster of representation records An undirected graph showing the correlations between fields.

[0043] Edge set Record cluster The undirected graph is notated as .

[0044] Preferred, .

[0045] S33. Extracting undirected graphs G Each connected component is considered as a candidate sub-cluster of a field. The candidate sub-clusters of a field reflect the statistical correlation and potential structural association between fields. Only fields that are highly linearly correlated, including those with strong positive or negative correlations, will be connected.

[0046] S34. For any candidate sub-cluster of any field ,in x To record clusters For the candidate sub-cluster index of fields, calculate the number of common non-null records for all fields: ; Common non-empty sample number Refers to the candidate sub-cluster of the field The number of samples in which all fields are non-empty, representing candidate subclusters of the field. Data completeness.

[0047] Preset minimum threshold for sufficient fields This represents the candidate subcluster for the current field. Minimum number of common non-empty samples; preset minimum threshold for non-empty samples in candidate fields The minimum threshold for non-empty samples in the candidate field Used to constrain the minimum number of common non-empty samples when merging candidate fields into candidate subclusters of preparatory fields.

[0048] Preferred, ; .

[0049] like If the number of fields is not less than the set number, then the current field candidate subcluster can be selected as a sufficient field candidate subcluster; like Candidate subclusters of the current field Fields in the array are sorted by non-empty frequency. Sort the data in descending order, and use each field as an initial seed to iterate through the remaining fields, treating them as candidate fields; if the number of common non-empty samples between the current field and the candidate fields is not less than... If the candidate field is selected, it is added to the candidate sub-cluster of the preliminary field; otherwise, the candidate field is discarded and the next candidate field is selected. After traversing all fields, if the number of fields in the candidate sub-cluster of the preliminary field is not less than the set number, the current candidate sub-cluster of the preliminary field can be selected as the candidate sub-cluster of the sufficient field.

[0050] Preferably, the set quantity is 3.

[0051] This allows for the selection of candidate subclusters of the field. All sufficient field candidate subclusters.

[0052] Through the above filtering and splitting, the system shrinks the "statistically related but insufficient common sample set of large fields" into the "small field set with sufficient common sample set", thereby forming sufficient candidate subclusters of fields that can be solved independently in regular business records and special event records, improving the stable identification capability of mainstream reconciliation rules and special event reconciliation rules.

[0053] S4. Solving balance constraints: Solving balance constraints without constant terms on the candidate subclusters of sufficient fields.

[0054] Preferably, an additive balance constraint is selected.

[0055] The term "no constant" means that the constraint can be expressed as a linear combination of the field values ​​equal to 0, and does not include fixed bias terms that are unrelated to the fields.

[0056] S41. Let any one of the sufficient field candidate subclusters be... Include q There are 3 fields, where y represents the candidate subcluster of the field. Indexes of sufficient candidate subclusters of fields; Sufficient field candidate subclusters All samples constitute the sample matrix ,in, , i= 1, 2, 3, ... n .

[0057] For the sample matrix X Perform singular value decomposition: ; in, U Describes a left singular vector matrix. Σ Represents a singular value matrix. V Represents a right singular vector matrix; Take the singular value matrix Σ The right singular vector in the right singular vector matrix corresponding to the minimum singular value is used as the summation-type equilibrium constraint coefficient vector. c ; S42. To prevent interference from a small number of abnormal records, the system does not directly use the candidate additive balance constraint coefficient vector. c Instead of calculating the candidate additive equilibrium constraint coefficient vector, the final result is obtained. c Residuals on each sample: ; in, Represents the sample matrix X The i One sample; Represents the balance constraint coefficient vector c In the i The residuals of each sample; The first i Scale of a sample ; in, Indicates the first i In the nth sample, the nth j The original values ​​corresponding to each field; This indicates that all fields are iterated over and the maximum value is retrieved.

[0058] The relative error is: ; in, To prevent extremely small positive numbers from being divided by zero, ; relative error Characterizing the candidate additive equilibrium constraint coefficient vector c The degree of deviation on the current sample relative to the normalized deviation of the sample's numerical scale is used to eliminate the influence of different monetary magnitudes on the residual judgment.

[0059] S43, Preset interior point error threshold Preferred Values .

[0060] The relative error is less than the inlier error threshold. If the samples are considered as interior points, then the set of interior points is denoted as . .

[0061] Correspondingly, the set of interior points Concatenate to obtain the interior point sample matrix Interior point sample matrix contain One sample.

[0062] Interior point sample matrix This is the set of effective samples that have a high degree of consistency with the current candidate.

[0063] Preset interior point ratio threshold Preferred The value is 0.7.

[0064] like For the interior point sample matrix Re-perform the singular value decomposition (refer to step S41) to obtain the updated additive equilibrium constraint coefficient vector. And the corresponding sufficient field candidate subclusters Recorded as Otherwise, discard the current sufficient field candidate subcluster. .

[0065] Updated additive balance constraint coefficient vector To create a more stable field coefficient structure that more closely approximates the actual financial reconciliation relationship after removing interference from abnormal samples.

[0066] S5. Rule Output: Convert the updated additive balance constraint into a financial reconciliation expression and calculate the rule confidence based on the full sample.

[0067] Among them, the full applicable sample refers to all samples in the financial accounting data table to be analyzed that participate in the current reconciliation expression after being removed by the non-empty threshold.

[0068] For each pair of updated additive balance constraint coefficient vectors and corresponding sufficient field candidate subclusters Perform the following steps respectively: S51, Preset near-zero detection threshold ; Preferred, ; Based on the updated additive equilibrium constraint coefficient vector Each field corresponds to a component, and the fields in the sufficient field candidate sub-cluster are divided into the left set and the right set to participate in the construction of the reconciliation expression; Specifically, if the absolute value of the corresponding component of the field does not exceed the near-zero threshold... Fields that are not involved in constructing the reconciliation expression do not participate; otherwise, if the corresponding component of a field is negative, the corresponding field is added to the left set. : When the component corresponding to a field is positive, the corresponding field is added to the set on the right. : .

[0069] S52, When the left set With the set on the right When all records are non-empty and the total number of participating fields is no less than 3, construct an additive equation reconciliation expression based on the original values ​​of the samples corresponding to the fields: ; in, Representation field j The original value corresponding to the current sample.

[0070] It is worth noting that both sides of the equation must contain the original value of at least one field. .

[0071] S53. Substitute the actual data from the full range of applicable samples into the verification financial reconciliation expression, statistically analyze the percentage of samples that meet the preset error threshold as the rule confidence level, and output the stable financial reconciliation rule.

[0072] The validity of the financial reconciliation expression generated in step S52 is verified using the data from the financial accounting data table to be analyzed.

[0073] The rule confidence is calculated as follows: ; in, Represents the interior point sample matrix The i Row samples; Indicates the first i The first row of the sample j The record values ​​corresponding to each field; (·) is an indicator function; This is a preset error threshold.

[0074] Preferred, .

[0075] Preset minimum confidence threshold If the rule confidence level Then the interior point sample matrix All samples can be directly used in actual engineering projects, outputting the corresponding logical expressions and rule confidence scores. and the corresponding set of fields.

[0076] Preferred, Minimum confidence threshold This is to avoid rule omissions due to the excessively large scope of the full-scale applicable samples.

[0077] By repeating the above steps with the financial accounting data table to be analyzed as input, the financial reconciliation relationships that meet the stability requirements can be automatically obtained, and the rule confidence of each reconciliation relationship can be calculated, thereby providing directly usable rule results for financial verification, data governance and audit review.

[0078] Example 2 The following section uses specific data to illustrate the above-mentioned method for automatically discovering reconciliation constraints for highly sparse and heterogeneous financial data tables.

[0079] S1. Using the fixed asset thematic detailed data table as the financial accounting data table to be analyzed, fields that are obviously not involved in the discovery of reconciliation relationships are removed. In this embodiment, the core numerical fields mainly include the following groups: Original Book Value field group: Original Book Value_Beginning Balance, Original Book Value_Increase in Current Period, Original Book Value_Purchase, Original Book Value_Increase in Business Combination, Original Book Value_Transfer from Construction in Progress, Original Book Value_Foreign Currency Translation (Increase in Current Period), Original Book Value_Other Increases, Original Book Value_Decrease in Current Period, Original Book Value_Disposal or Scrapping, Original Book Value_Decrease in Business Combination, Original Book Value_Disposal of Subsidiary, Original Book Value_Foreign Currency Translation (Decrease in Current Period), Original Book Value_Other Decreases, Original Book Value_Ending Balance; Accumulated Depreciation Field Group: Accumulated Depreciation_Beginning Balance, Accumulated Depreciation_Increase in Current Period, Accumulated Depreciation_Accumulated, Accumulated Depreciation_Increase in Business Combination, Accumulated Depreciation_Transfer from Construction in Progress, Accumulated Depreciation_Foreign Currency Translation (Increase in Current Period), Accumulated Depreciation_Other Increases, Accumulated Depreciation_Decrease in Current Period, Accumulated Depreciation_Disposal or Scrapping, Accumulated Depreciation_Decrease in Business Combination, Accumulated Depreciation_Disposal of Subsidiary, Accumulated Depreciation_Foreign Currency Translation (Decrease in Current Period), Accumulated Depreciation_Other Decreases, Accumulated Depreciation_Ending Balance; Impairment Provision Field Group: Impairment Provision_Beginning Balance, Impairment Provision_Increase in Current Period, Impairment Provision_Accumulation, Impairment Provision_Increase in Business Combination, Impairment Provision_Transfer from Construction in Progress, Impairment Provision_Foreign Currency Translation (Increase in Current Period), Impairment Provision_Other Increase, Impairment Provision_Decrease in Current Period, Impairment Provision_Disposal or Scrapping, Impairment Provision_Decrease in Business Combination, Impairment Provision_Disposal of Subsidiary, Impairment Provision_Foreign Currency Translation (Decrease in Current Period), Impairment Provision_Other Decrease, Impairment Provision_Ending Balance; Net Book Value field group: Net Book Value_Beginning Balance, Net Book Value_Increase During the Period, Net Book Value_Decrease During the Period, Net Book Value_Ending Balance; Book value field group: Book value_beginning balance, Book value_increase during the period, Book value_decrease during the period, Book value_ending balance.

[0080] After filtering by the minimum non-empty threshold, the remaining valid fields are: those with a disclosed non-empty rate: book value_beginning balance, book value_ending balance, accumulated depreciation_beginning balance, accumulated depreciation_ending balance, impairment provision_ending balance, and book value_ending balance; and those with a high non-empty rate but above the threshold: additions from business combinations, disposals of subsidiaries, and foreign currency translations.

[0081] For fields related to mergers, acquisitions, disposals of subsidiaries, and foreign currency translations that appear only in a few special records, their inverse document frequency weight is increased, resulting in higher recognition weight during clustering.

[0082] In step S33, the candidate subclusters of fields formed based on connected components mainly include: Candidate sub-clusters for the original value total rolling field: Original Book Value_Beginning Balance, Original Book Value_Increase in Current Period, Original Book Value_Decrease in Current Period, Original Book Value_Ending Balance; Original value increase decomposition field candidate sub-clusters: Book value_increase in current period, Book value_purchase, Book value_increase in business combination, Book value_transfer from construction in progress, Book value_foreign currency translation (increase in current period), Book value_other increases; Original value reduction decomposition field candidate sub-clusters: Book value_reduction in current period, Book value_disposal or scrapping, Book value_reduction in business combination, Book value_disposal of subsidiary, Book value_foreign currency translation (reduction in current period), Book value_other reduction; Candidate sub-clusters for the cumulative depreciation total rolling field: Cumulative Depreciation_Beginning Balance, Cumulative Depreciation_Increase in Current Period, Cumulative Depreciation_Decrease in Current Period, Cumulative Depreciation_Ending Balance; Candidate sub-clusters for decomposing accumulated depreciation: Accumulated Depreciation_Increase in Current Period, Accumulated Depreciation_Accumulation, Accumulated Depreciation_Increase in Business Combination, Accumulated Depreciation_Transfer from Construction in Progress, Accumulated Depreciation_Foreign Currency Translation (Increase in Current Period), Accumulated Depreciation_Other Increases; Candidate sub-clusters for decomposing accumulated depreciation reduction: Accumulated Depreciation_Reduction in Current Period, Accumulated Depreciation_Disposal or Scrapping, Accumulated Depreciation_Reduction from Business Combination, Accumulated Depreciation_Disposal of Subsidiary, Accumulated Depreciation_Foreign Currency Translation (Reduction in Current Period), Accumulated Depreciation_Other Reductions; Candidate sub-clusters for the total rolling field of impairment provisions: Impairment Provision_Beginning Balance, Impairment Provision_Increase During the Period, Impairment Provision_Decrease During the Period, Impairment Provision_Ending Balance; The following candidate sub-clusters for decomposition fields are added to the impairment provision: Impairment Provision_Increase in Current Period, Impairment Provision_Accumulation, Impairment Provision_Increase in Business Combination, Impairment Provision_Transfer from Construction in Progress, Impairment Provision_Foreign Currency Translation (Increase in Current Period), Impairment Provision_Other Increases; Impairment provision reduction decomposition field candidate sub-clusters: Impairment provision_Current period reduction, Impairment provision_Disposal or scrapping, Impairment provision_Business combination reduction, Impairment provision_Disposal of subsidiary, Impairment provision_Foreign currency translation (Current period reduction), Impairment provision_Other reduction; Candidate subclusters of book value composition fields: Original Book Value_Ending Balance, Accumulated Depreciation_Ending Balance, Impairment Provision_Ending Balance, Book Value_Ending Balance The aforementioned candidate subclusters reflect the statistical correlation and potential structural associations between fields. In step S34, taking the addition of candidate subclusters of the decomposed field to the original value as an example, the sufficient candidate subclusters of the field obtained after splitting are: Mainstream added field sub-clusters: Book Value_Increase in Current Period, Book Value_Purchase, Book Value_Transfer from Construction in Progress, Book Value_Other Increases; The following sub-clusters of fields have been added for business combination items: Book Value_Increase in Current Period, Book Value_Acquisition, Book Value_Increase in Business Combination, and Book Value_Other Increases; Foreign currency translation items have added the following sub-clusters: Book Value_Increase in Current Period, Book Value_Purchase, Book Value_Foreign Currency Translation (Increase in Current Period), and Book Value_Other.

[0083] Through the above further splitting, the system can shrink the "statistically correlated but insufficient common sample set of large fields" into the "small field set with sufficient common sample".

[0084] In step 52, the main cross-referencing relationships in the final output include: Original book value - ending balance = Original book value - beginning balance + Original book value - increase during the period - Original book value - decrease during the period; Original Book Value - Increase in Current Period = Original Book Value - Acquisitions + Original Book Value - Increase from Business Combinations + Original Book Value - Transfer from Construction in Progress + Original Book Value - Foreign Currency Translation (Increase in Current Period) + Original Book Value - Other Increases; Original Book Value - Decrease in Current Period = Original Book Value - Disposal or Scrapping + Original Book Value - Decrease in Business Combination + Original Book Value - Disposal of Subsidiary + Original Book Value - Foreign Currency Translation (Decrease in Current Period) + Original Book Value - Other Decrease Accumulated Depreciation - Ending Balance = Accumulated Depreciation - Beginning Balance + Accumulated Depreciation - Increase in Current Period - Accumulated Depreciation - Decrease in Current Period; Accumulated Depreciation - Increase in the Current Period = Accumulated Depreciation - Accrued Depreciation + Accumulated Depreciation - Increase from Business Combination + Accumulated Depreciation - Transfer from Construction in Progress + Accumulated Depreciation - Foreign Currency Translation (Increase in the Current Period) + Accumulated Depreciation - Other Increases; Accumulated Depreciation - Decrease in Current Period = Accumulated Depreciation - Disposal or Scrapping + Accumulated Depreciation - Decrease in Business Combination + Accumulated Depreciation - Disposal of Subsidiaries + Accumulated Depreciation - Foreign Currency Translation (Decrease in Current Period) + Accumulated Depreciation - Other Decrease; Impairment provision - ending balance = Impairment provision - beginning balance + Impairment provision - increase during the period - Impairment provision - decrease during the period; Impairment provision - Increase in the current period = Impairment provision - Provision + Impairment provision - Increase in business combination + Impairment provision - Transfer from construction in progress + Impairment provision - Foreign currency translation (increase in the current period) + Impairment provision - Other increases; Impairment provision - Decrease in current period = Impairment provision - Disposal or scrapping + Impairment provision - Decrease in business combination + Impairment provision - Disposal of subsidiary + Impairment provision - Foreign currency translation (decrease in current period) + Impairment provision - Other decreases; Book value - ending balance = Original book value - ending balance - accumulated depreciation - ending balance - impairment provision - ending balance.

[0085] Net book value - ending balance = Original book value - ending balance - Accumulated depreciation - ending balance.

[0086] In step S53, the system can output the following content according to the structured rule result table: The rule number is R001, and the reconciliation expression is "Book value_Ending balance = Book value_Beginning balance + Book value_Current period increase - Book value_Current period decrease". The corresponding field set is {Book value_Beginning balance, Book value_Current period increase, Book value_Current period decrease, Book value_Ending balance}. The rule confidence level is 1.0000, the number of valid samples is 12486, and the source record cluster information is "Fixed asset master record cluster - Original value total rolling field candidate sub-cluster".

[0087] Rule number R002 has the following reconciliation expression: "Book value_Current period increase = Book value_Purchase + Book value_Enterprise combination increase + Book value_Transfer from construction in progress + Book value_Foreign currency translation (Current period increase) + Book value_Other increase". The corresponding field set is {Book value_Current period increase, Book value_Purchase, Book value_Enterprise combination increase, Book value_Transfer from construction in progress, Book value_Foreign currency translation (Current period increase), Book value_Other increase}. The rule confidence level is 0.9867, the number of valid samples is 10214, and the source record cluster information is "Fixed asset master record cluster - Original value increase decomposition field sub-cluster".

[0088] Rule number R003 has the following reconciliation expression: "Book value - Current period reduction = Book value - Disposal or scrapping + Book value - Reduction due to business combination + Book value - Disposal of subsidiary + Book value - Foreign currency translation (Current period reduction) + Book value - Other reduction". The corresponding field set is {Book value - Current period reduction, Book value - Disposal or scrapping, Book value - Reduction due to business combination, Book value - Disposal of subsidiary, Book value - Foreign currency translation (Current period reduction), Book value - Other reduction}. The rule confidence level is 0.9412, the number of valid samples is 4083, and the source record cluster information is "Fixed asset special event record cluster - Original value reduction decomposition field sub-cluster".

[0089] The rule number is R004, and the reconciliation expression is "Accumulated Depreciation_Ending Balance = Accumulated Depreciation_Beginning Balance + Accumulated Depreciation_Current Period Increase - Accumulated Depreciation_Current Period Decrease". The corresponding field set is {Accumulated Depreciation_Beginning Balance, Accumulated Depreciation_Current Period Increase, Accumulated Depreciation_Current Period Decrease, Accumulated Depreciation_Ending Balance}. The rule confidence level is 1.0000, the number of valid samples is 12486, and the source record cluster information is "Fixed Assets Master Record Cluster - Accumulated Depreciation Total Rolling Field Sub-Cluster".

[0090] Rule number R005, reconciliation expression is "Impairment Provision_Ending Balance = Impairment Provision_Beginning Balance + Impairment Provision_Current Period Increase - Impairment Provision_Current Period Decrease", corresponding field set is {Impairment Provision_Beginning Balance, Impairment Provision_Current Period Increase, Impairment Provision_Current Period Decrease, Impairment Provision_Ending Balance}, rule confidence is 0.9735, effective sample size is 3267, source record cluster information is "Fixed Asset Impairment Related Record Cluster - Impairment Provision Total Rolling Field Sub-Cluster".

[0091] The rule number is R006, and the reconciliation expression is "Book Value_Ending Balance = Original Book Value_Ending Balance - Accumulated Depreciation_Ending Balance - Impairment Provision_Ending Balance". The corresponding field set is {Original Book Value_Ending Balance, Accumulated Depreciation_Ending Balance, Impairment Provision_Ending Balance, Book Value_Ending Balance}. The rule confidence level is 1.0000, the number of valid samples is 12486, and the source record cluster information is "Fixed Assets Master Record Cluster - Book Value Component Field Sub-cluster".

[0092] Example 3 This invention has an extended mode, in which the financial accounting data table to be analyzed, which is the input, can be formed by summarizing multiple sub-tables after field alignment.

[0093] For example, in addition to the fixed asset special table, you can also introduce supplementary details such as the bad debt provision change table, the deferred income change table, and the long-term deferred expense change table. After aligning the fields of records from different sources, you can import them into the same financial accounting data table to be analyzed as input.

[0094] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0095] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0096] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A method for automatically discovering reconciliation constraints for highly sparse and heterogeneous financial data tables, characterized in that, Includes the following steps: S1. Data Reading and Non-Null Modeling: Obtain the financial accounting data table to be analyzed, filter the effective numerical fields and construct a binary matrix representing the field distribution pattern, introduce inverse frequency weights to enhance sparse fields, and generate a weighted non-null state vector for each record. S2. Record Clustering: Perform online clustering on each record based on weighted cosine similarity to obtain record clusters whose weighted cosine similarity meets a preset threshold; S3. Field sub-cluster generation: Construct an undirected field graph within each record cluster, and combine it with common non-empty sample conditions to form sufficient candidate field sub-clusters suitable for modeling; S4. Solving the balance constraint: Solve the balance constraint coefficient vector on the candidate subclusters of sufficient fields through singular value decomposition, mine the financial reconciliation relationship between fields that meet the balance constraint, further filter the inliers that meet the financial reconciliation relationship, and update the balance constraint coefficient vector based on the proportion of inliers. S5. Rule Output: Based on the updated balance constraint coefficient vector and the corresponding sufficient field candidate subclusters, construct a financial reconciliation expression, input actual data to verify the rule confidence of the financial reconciliation expression, and output stable financial reconciliation rules.

2. The automatic discovery method for reconciliation constraints of highly sparse and heterogeneous financial data tables as described in claim 1, characterized in that, In step S1, the non-empty record values ​​of each field in the statistical data table are analyzed. A minimum non-empty rate threshold is preset. Fields with a non-empty record rate not lower than the minimum non-empty rate threshold are selected as fields to be analyzed, and the numerical fields in the fields to be analyzed are extracted as valid fields. Based on all valid fields, a binary matrix representing the distribution pattern of empty / non-empty states of the valid fields is constructed. ,in, N Indicates the total number of records. M This indicates the total number of valid fields; for valid fields... j In the i binary variables in records If the record is not empty, then ;otherwise, .

3. The automatic discovery method for reconciliation constraints in highly sparse and heterogeneous financial data tables as described in claim 2, characterized in that, In step S1, for any valid field Statistical non-empty frequency ; Calculate valid fields The inverse document frequency weight is ; The set of weight vectors for the effective field set is denoted as For the first i 1 record, row vector denoted as Weighted state vector .

4. The automatic discovery method for reconciliation constraints of highly sparse and heterogeneous financial data tables as described in claim 3, characterized in that, In step S2, for the first i The weighted cosine similarity is calculated for each record as follows: ; Preset similarity threshold ,in, This is a record cluster used to include records whose mean similarity to this record cluster reaches a threshold. Weighted state vector ; For class Internal weighted non-empty state vector The mean; iterate through all existing classes. ,like Then If a class has the highest similarity, it should be assigned to the class with the highest similarity, and the corresponding class center should be updated; otherwise, it should be assigned to the class with the highest similarity. Create a new class for the initial center.

5. The automatic discovery method for reconciliation constraints in highly sparse and heterogeneous financial data tables as described in claim 4, characterized in that, Class Center The mean of the weighted non-empty state vector is used for online updates, and the update formula is as follows: in, The updated classes center; Representation Class The number of elements; This indicates the first one that has not yet been categorized. i The weighted state vector of the records.

6. The automatic discovery method for reconciliation constraints of highly sparse and heterogeneous financial data tables as described in claim 4, characterized in that, In step S3, an undirected graph of the fields is constructed: based on record clusters. Calculate any two distinct fields from all the original data. Pearson correlation coefficient between ; Preset correlation coefficient threshold edge set Record cluster The undirected graph is notated as .

7. The automatic discovery method for reconciliation constraints of highly sparse and heterogeneous financial data tables as described in claim 6, characterized in that, Methods for filtering candidate subclusters with sufficient fields: Extracting undirected graphs G Each connected component is considered as a candidate subcluster for a field. For any one of these candidate subclusters... Calculate the number of common non-empty samples across all fields: ; Preset minimum threshold for sufficient fields and the minimum threshold for non-empty samples in the candidate field ; like If the number of fields is not less than the set number, then the current field candidate subcluster can be selected as a sufficient field candidate subcluster; like Candidate subclusters of the current field Fields in the array are sorted by non-empty frequency. Sort in descending order, if the number of common non-empty samples of the current field and the other candidate fields is not less than If the candidate field is not specified, then add the candidate field to the preliminary candidate field sub-cluster; after traversing all fields, select the preliminary candidate field sub-cluster with a number of fields not less than the set number as the sufficient candidate field sub-cluster.

8. The automatic discovery method for reconciliation constraints of highly sparse and heterogeneous financial data tables as described in claim 7, characterized in that, In step S4, let any one of the sufficient field candidate subclusters be... Include q Each field and n The number of common non-empty samples, among which, y Indicates candidate subclusters of the field The index of sufficient field candidate subclusters, the sample matrix is For the sample matrix X Perform singular value decomposition: ;in, U Describes a left singular vector matrix. Σ Represents a singular value matrix. V Denotes the right singular vector matrix; takes the singular value matrix. Σ The right singular vector in the right singular vector matrix corresponding to the minimum singular value is used as the summation-type balance constraint coefficient vector c.

9. The automatic discovery method for reconciliation constraints of highly sparse and heterogeneous financial data tables as described in claim 8, characterized in that, Additive equilibrium constraint coefficient vector c Residuals on each sample: ,in, Represents the sample matrix X The i One sample; Represents the vector of summative equilibrium constraint coefficients c In the i The residuals of each sample; The first i Scale of a sample ;in, Indicates the first i In the nth sample, the nth j The original values ​​corresponding to each field; Find the maximum value for all fields; No. i The relative error of each sample is: ;in, To prevent extremely small positive numbers from being divided by zero, ; Preset interior point error threshold The relative error is less than the interior point error threshold. If the samples are considered as interior points, then the set of interior points is denoted as . Correspondingly, the set of interior points Concatenate to obtain the interior point sample matrix Interior point sample matrix contain One sample; Preset interior point ratio threshold ,like For the interior point sample matrix Re-perform the singular value decomposition to obtain the updated additive equilibrium constraint coefficient vector. .

10. The method for automatic discovery of reconciliation constraints for highly sparse and heterogeneous financial data tables as described in claim 9, characterized in that, Preset near-zero threshold Filter out the updated additive balance constraint coefficient vector The absolute value of the corresponding component in the middle field does not exceed the near-zero threshold. The fields; the components corresponding to the filtered fields are divided into the left set according to their signs, based on the positive or negative signs of the sufficient field candidate subclusters. and the set on the right ; When the left set With the set on the right When all records are non-empty and the total number of participating fields is not less than a set number, construct an additive equation reconciliation expression based on the original values ​​of the samples corresponding to the fields: in, Representation field j The original value corresponding to the current sample; It is worth noting that both sides of the equation must contain the original value of at least one field. ; Rule confidence The calculation method is as follows: in, Represents the interior point sample matrix The i One sample; Indicates the first i In the nth sample j The record values ​​corresponding to each field; (·) is an indicator function; The preset error threshold is used; Preset minimum confidence threshold If the rule confidence level Then output the corresponding cross-validation expression and rule confidence. and the corresponding sufficient field candidate subclusters .

Images